rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Log Normalization and special characters

Adiscon SupportFAQ, , ,

When trying to normalize log messages via liblognorm and mmnormalize, you need to create a rulebase first. The rulebase is usually a representation of message formats.

Due to the format of these rules, it is necessary to be cautious. Some messages and rule necessities could possibly cause confusion to the configuration interpreter. This mainly applies to clear text passages in single rules.

For example, if you have a log message from a Cisco ASA, the message looks like this:

2012-11-23T10:47:42+01:00 10.10.10.10 : %ASA-3-313001: ...

The only interesting parts are the IP and the numerical code to identify the message. We are not interested in the timestamp or “%ASA”. But when making the rule, the trouble starts there. The percent character is also used to define variables and their values in a rule. Thus it needs to be escaped. This is done with the ASCII code representation of the percent character. The rule would look like this:

rule=: %date:word% %host:ipv4% : \x25ASA-%char1:char-to:-%-%char2:number%: ...

If you write “%ASA” into the rule, the interpreter will think, that a new variable starts there. This will cause confusion to the rest of the rule and render it not working correctly. This needs to be avoided.

The same applies to “:”. But this time, it needs to be escaped when using it as delimiter vor variables. Example:

%variable:char-to:\x3a%

This will fill “variable” with everything until the next “:” occurs. If you just put a “:” here as a delimiter, the rule will not work anymore.

Changelog for 7.3.4 (v7-devel)

adisconteamChangelog, , , ,

Version 7.3.4 [devel] 2012-11-23

  • further (and rather drastically) improved disk queue performance
    we now save one third of the IO calls
  • imklog: added ParseKernelTimestamp parameter (import from 5.10.2)
    Thanks to Marius Tomaschewski for the patch.
  • imklog: added KeepKernelTimestamp parameter (import from 5.10.2)
    Thanks to Marius Tomaschewski for the patch.
  • bugfix: improper handling of backslash in string-type template()s
  • bugfix: leading quote (“) in string-type template() lead to thight loop
    on startup
  • bugfix: no error msg on invalid field option in legacy/string template
  • bugfix: imklog mistakenly took kernel timestamp subseconds as nanoseconds
    … actually, they are microseconds. So the fractional part of the
    timestamp was not properly formatted. (import from 5.10.2)
    Thanks to Marius Tomaschewski for the bug report and the patch idea.

rsyslog 7.3.4 (v7-devel) released

adisconteamNews, Release Announcement, , , , ,

This release most importantly contains drastic improvements of disk queue performance (including DA queues). The new code saves one third of file IO calls, improving both rsyslogd as well as overall system performance. It also contains some smaller enhancements as well as bug fixes.

ChangeLog:

http://www.rsyslog.com/changelog-for-7-3-4-v7-devel/

Download:

http://www.rsyslog.com/rsyslog-7-3-4-v7-devel/

As always, feedback is appreciated.

Best regards,
Tim Eifler

Changelog for 7.2.3 (v7-stable)

adisconteamChangelog, , , , , ,

Version 7.2.3 [v7-stable] 2012-11-21

  • regression fix: rsyslogd terminated when wild-card $IncludeConfig did not
    find actual include files. For example, if this directive is present:
    $IncludeConfig /etc/rsyslog.d/*.conf
    and there are no *.conf files in /etc/rsyslog.d (but rsyslog.d exists),
    rsyslogd will emit an error message and terminate. Previous (and expected)
    behaviour is that an empty file set is no problem. HOWEVER, if the
    directory itself does not exist, this is flagged as an error and will
    load to termination (no startup).
    Unfortunately, this is often the case by default in many distros, so this
    actually prevents rsyslog startup.

rsyslog 7.2.3 (v7-stable) released

adisconteamNews, Release Announcement, , , ,

This is mainly a re-release of 7.2.2 with an important regression fix. Version 7.2.2 caused premature end of config file (and most often rsyslog termination) if an $IncludeConfig directive was done on a wildcard that matched no files. This is a default situation on my distro plain installs. Users are strongly urged to upgrade to 7.2.3. Note that we did not build RPMs for 7.2.2 as we noticed the problem early enough.

Version 7.2.3 is also available as RPM via the Adiscon repository (see http://www.rsyslog.com/rhelcentos-rpms/ ).

ChangeLog:

http://www.rsyslog.com/changelog-for-7-2-3-v7-stable/

Download:

http://www.rsyslog.com/rsyslog-7-2-3-v7-stable/

As always, feedback is appreciated.

Best regards,
Tim Eifler

Changelog for 7.2.2 (v7-stable)

adisconteamChangelog, , , , ,

Version 7.2.2 [v7-stable] 2012-10-16

  • doc improvements
  • enabled to build without libuuid, at loss of uuid functionality
    this enables smoother builds on older systems that do not support
    libuuid. Loss of functionality should usually not matter too much as
    uuid support has only recently been added and is very seldom used.
  • bugfix: omfwd did not properly support “template” parameter
  • bugfix: potential segfault when re_match() function was used
    Thanks to oxpa for the patch.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=371
  • bugfix: potential abort of imtcp on rsyslogd shutdown
  • bugfix: imzmq3 segfault with PULL subscription
    Thanks to Martin Nilsson for the patch.
  • bugfix: improper handling of backslash in string-type template()s
  • bugfix: leading quote (“) in string-type template() lead to thight loop
    on startup
  • bugfix: no error msg on invalid field option in legacy/string template
  • bugfix: potential segfault due to invalid param handling in comparisons
    This could happen in RainerScript comparisons (like contains); in some
    cases an unitialized variable was accessed, which could lead to an
    invalid free and in turn to a segfault.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=372
    Thanks to Georgi Georgiev for reporting this bug and his great help
    in solving it.
  • bugfix: no error msg on unreadable $IncludeConfig path
  • bugfix: $IncludeConfig did not correctly process directories
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=376
    The testbench was also enhanced to check for these cases.
    Thanks to Georgi Georgiev for the bug report.
  • bugfix: make rsyslog compile on kfreebsd again
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=380
    Thanks to Guillem Jover for the patch
  • bugfix: garbled message if field name was used with jsonf property option
    The length for the field name was invalidly computed, resulting in either
    truncated field names or including extra random data. If the random data
    contained NULs, the rest of the message became unreadable.
  • bugfix: potential segfault at startup with property-based filter
    If the property name was followed by a space before the comma, rsyslogd
    aborted on startup. Note that no segfault could happen if the initial
    startup went well (this was a problem with the config parser).
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=381
  • bugfix: imfile discarded some file parts
    File lines that were incomplete (LF missing) *at the time imfile polled
    the file* were partially discarded. That part of the line that was read
    without the LF was discarded, and the rest of the line was submitted in
    the next polling cycle. This is now changed so that the partial content
    is saved until the complete line is read. Note that the patch affects
    only read mode 0.
    Thanks to Milan Bartos for providing the base idea for the solution.
Scroll to top