rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Search Results for: error 0

rsyslog 6.3.3 config format improvements

adisconteamNews, , , ,

In rsyslog 6.3.3, the config processor has finally changed. The old legacy processor (and with it the early RainerScript implementation) is thrown out and has been replaced by the so-called RainerScript processor (why that crazy name?). This is an extremely important step for rsyslog, as it now has the foundation for a much better and intuitive rsyslog.conf format. However, most of that can not be seen in 6.3.3, as it requires more work, especially in the plugin arena. Still, there are a couple of smaller improvements available.

Most importantly, the performance of script based filters has been considerably enhanced. Preliminary testing shows a three times speedup (we’ll do more benchmarking at a later stage; there is also still lots of room for optimization ;-)).

The ugliness of continuation lines has been removed. They may still be used, and this may make a lot of sense with some actions, but you are usually no longer forced to use continuation lines. Take this config snippet from a leading distro:


if ( \
     /* kernel up to warning except of firewall  */ \
     ($syslogfacility-text == 'kern')      and      \
     ($syslogseverity <= 4 /* warning */ ) and not  \
     ($msg contains 'IN=' and $msg contains 'OUT=') \
 ) or ( \
     /* up to errors except of facility authpriv */ \
     ($syslogseverity <= 3 /* errors  */ ) and not  \
     ($syslogfacility-text == 'authpriv')           \
 ) \
then /dev/tty10
& |/dev/xconsole

This can now be written as follows:


if (
     /* kernel up to warning except of firewall  */
     ($syslogfacility-text == 'kern')      and
     ($syslogseverity <= 4 /* warning */ ) and not
     ($msg contains 'IN=' and $msg contains 'OUT=')
 ) or (
     /* up to errors except of facility authpriv */
     ($syslogseverity <= 3 /* errors  */ ) and not
     ($syslogfacility-text == 'authpriv')
 )
then /dev/tty10

& |/dev/xconsole

Continue reading “rsyslog 6.3.3 config format improvements”

Changelog for 4.7.4 (v4-beta)

adisconteamChangelog, , ,

Version 4.7.4 [v4-beta] (rgerhards), 2011-07-11

  • added support for the “:omusrmsg:” syntax in configuring user messages
  • added support for the “:omfile:” syntax in configuring user messages
  • added $LocalHostName config directive
  • bugfix: PRI was invalid on Solaris for message from local log socket
  • bugfix: local hostname was pulled too-early, so that some config
    directives (namely FQDN settings) did not have any effect

  • bugfix: atomic increment for msg object may not work correct on all
    platforms. Thanks to Chris Metcalf for the patch

  • bugfix: a slightly more informative error message when a TCP
    connections is aborted

Changelog for 6.3.2 (v6-devel)

adisconteamChangelog, , ,

Version 6.3.2 [DEVEL] (rgerhards), 2011-07-06

  • added support for the “:omusrmsg:” syntax in configuring user messages
  • systemd support: set stdout/stderr to null – thx to Lennart for the patch
  • added support for obtaining timestamp for kernel message from message
    If the kernel time-stamps messages, time is now take from that
    timestamp instead of the system time when the message was read. This
    provides much better accuracy. Thanks to Lennart Poettering for
    suggesting this feature and his help during implementation.

  • added support for obtaining timestamp from system for imuxsock
    This permits to read the time a message was submitted to the system
    log socket. Most importantly, this is provided in microsecond resolution.
    So we are able to obtain high precision timestampis even for messages
    that were – as is usual – not formatted with them. This also simplifies
    things in regard to local time calculation in chroot environments.
    Many thanks to Lennart Poettering for suggesting this feature,
    providing some guidance on implementing it and coordinating getting the
    necessary support into the Linux kernel.

  • bugfix: timestamp was incorrectly calculated for timezones with minute
    offset
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=271

  • bugfix: memory leak in imtcp & subsystems under some circumstances
    This leak is tied to error conditions which lead to incorrect cleanup
    of some data structures.

Changelog for 5.9.1 (v5-devel)

adisconteamChangelog, ,

Version 5.9.1 [V5-DEVEL] (rgerhards), 2011-06-30

  • added support for obtaining timestamp for kernel message from message
    If the kernel time-stamps messages, time is now take from that
    timestamp instead of the system time when the message was read. This
    provides much better accuracy. Thanks to Lennart Poettering for
    suggesting this feature and his help during implementation.

  • added support for obtaining timestamp from system for imuxsock
    This permits to read the time a message was submitted to the system
    log socket. Most importantly, this is provided in microsecond resolution.
    So we are able to obtain high precision timestampis even for messages
    that were – as is usual – not formatted with them. This also simplifies
    things in regard to local time calculation in chroot environments.
    Many thanks to Lennart Poettering for suggesting this feature,
    providing some guidance on implementing it and coordinating getting the
    necessary support into the Linux kernel.

  • bugfix: timestamp was incorrectly calculated for timezones with minute
    offset
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=271

  • bugfix: problems in failover action handling
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=270
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=254

  • bugfix: mutex was invalidly left unlocked during action processing
    At least one case where this can occur is during thread shutdown, which
    may be initiated by lower activity. In most cases, this is quite
    unlikely to happen. However, if it does, data structures may be
    corrupted which could lead to fatal failure and segfault. I detected
    this via a testbench test, not a user report. But I assume that some
    users may have had unreproducable aborts that were cause by this bug.

  • bugfix: memory leak in imtcp & subsystems under some circumstances
    This leak is tied to error conditions which lead to incorrect cleanup
    of some data structures. [backport from v6]

  • bugfix/improvement:$WorkDirectory now gracefully handles trailing slashes

Changelog for 4.6.6 (v4-stable)

adisconteamChangelog, , , ,

Version 4.6.6 [v4-stable] (rgerhards), 2011-06-24

  • bugfix: memory leak in imtcp & subsystems under some circumstances
    This leak is tied to error conditions which lead to incorrect cleanup
    of some data structures. [backport from v6, limited testing under v4]
  • bugfix: invalid processing in QUEUE_FULL condition
    If the the multi-submit interface was used and a QUEUE_FULL condition
    occured, the failed message was properly destructed. However, the
    rest of the input batch, if it existed, was not processed. So this
    lead to potential loss of messages and a memory leak. The potential
    loss of messages was IMHO minor, because they would have been dropped
    in most cases due to the queue remaining full, but very few lucky ones
    from the batch may have made it. Anyhow, this has now been changed so
    that the rest of the batch is properly tried to be enqueued and, if
    not possible, destructed.
  • bugfix: invalid storage type for config variables
  • bugfix: stream driver mode was not correctly set on tcp ouput on big
    endian systems.
    thanks varmojfekoj for the patch
  • bugfix: IPv6-address could not be specified in omrelp
    this was due to improper parsing of “:”
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250
  • bugfix: memory and file descriptor leak in stream processing
    Leaks could occur under some circumstances if the file stream handler
    errored out during the open call. Among others, this could cause very
    big memory leaks if there were a problem with unreadable disk queue
    files. In regard to the memory leak, this
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256
  • bugfix: imfile potentially duplicates lines
    This can happen when 0 bytes are read from the input file, and some
    writer appends data to the file BEFORE we check if a rollover happens.
    The check for rollover uses the inode and size as a criterion. So far,
    we checked for equality of sizes, which is not given in this scenario,
    but that does not indicate a rollover. From the source code comments:
    Note that when we check the size, we MUST NOT check for equality.
    The reason is that the file may have been written right after we
    did try to read (so the file size has increased). That is NOT in
    indicator of a rollover (this is an actual bug scenario we
    experienced). So we need to check if the new size is smaller than
    what we already have seen!
    Also, under some circumstances an invalid truncation was detected. This
    code has now been removed, a file change (and thus resent) is only
    detected if the inode number changes.
  • bugfix: a couple of problems that imfile had on some platforms, namely
    Ubuntu (not their fault, but occured there)
  • bugfix: imfile utilizes 32 bit to track offset. Most importantly,
    this problem can not experienced on Fedora 64 bit OS (which has
    64 bit long’s!)
  • bugfix: abort if imfile reads file line of more than 64KiB
    Thanks to Peter Eisentraut for reporting and analysing this problem.
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221
  • bugfix: omlibdbi did not use password from rsyslog.con
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=271
  • some improvements thanks to clang’s static code analyzer
    • overall cleanup (mostly unnecessary writes and otherwise unused stuff)
    • bugfix: fixed a very remote problem in msg.c which could occur when
      running under extremely low memory conditions

Changelog for 5.8.2. (v5-stable)

adisconteamChangelog, , , ,

Version 5.8.2 [V5-stable] (rgerhards), 2011-06-21

  • bugfix: problems in failover action handling
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=270
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=254

  • bugfix: mutex was invalidly left unlocked during action processing
    At least one case where this can occur is during thread shutdown, which
    may be initiated by lower activity. In most cases, this is quite
    unlikely to happen. However, if it does, data structures may be
    corrupted which could lead to fatal failure and segfault. I detected
    this via a testbench test, not a user report. But I assume that some
    users may have had unreproducable aborts that were cause by this bug.

  • bugfix: memory leak in imtcp & subsystems under some circumstances
    This leak is tied to error conditions which lead to incorrect cleanup
    of some data structures. [backport from v6]

  • bugfix/improvement:$WorkDirectory now gracefully handles trailing slashes

Changelog for 5.9.0 (v5-devel)

adisconteamChangelog, , , ,

Version 5.9.0 [V5-DEVEL] (rgerhards), 2011-06-08

  • imfile: added $InputFileMaxLinesAtOnce directive
  • enhanced imfile to support input batching
  • added capability for imtcp and imptcp to activate keep-alive packets
    at the socket layer. This has not been added to imttcp, as the latter is
    only an experimental module, and one which did not prove to be useful.
    reference: http://kb.monitorware.com/post20791.html

  • added support to control KEEPALIVE settings in imptcp
    this has not yet been added to imtcp, but could be done on request.

  • $ActionName is now also used for naming of queues in impstats
    as well as in the debug output

  • bugfix: do not open files with full privileges, if privs will be dropped
    This make the privilege drop code more bulletproof, but breaks Ubuntu’s
    work-around for log files created by external programs with the wrong
    user and/or group. Note that it was long said that this “functionality”
    would break once we go for serious privilege drop code, so hopefully
    nobody still depends on it (and, if so, they lost…).

  • bugfix: pipes not opened in full priv mode when privs are to be dropped
  • this begins a new devel branch for v5
  • better handling of queue i/o errors in disk queues. This is kind of a
    bugfix, but a very intrusive one, this it goes into the devel version
    first. Right now, “file not found” is handled and leads to the new
    emergency mode, in which disk action is stopped and the queue run
    in direct mode. An error message is emited if this happens.

  • added support for user-level PRI provided via systemd
  • added new config directive $InputTCPFlowControl to select if tcp
    received messages shall be flagged as light delayable or not.

  • enhanced omhdfs to support batching mode. This permits to increase
    performance, as we now call the HDFS API with much larger message
    sizes and far more infrequently

  • bugfix: failover did not work correctly if repeated msg reduction was on
    affected directive was: $ActionExecOnlyWhenPreviousIsSuspended on
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=236

Changelog for 5.8.1 (v5-stable)

adisconteamChangelog, , , ,

Version 5.8.1 [V5-stable] (rgerhards), 2011-05-19

  • bugfix: invalid processing in QUEUE_FULL condition
    If the the multi-submit interface was used and a QUEUE_FULL condition
    occured, the failed message was properly destructed. However, the
    rest of the input batch, if it existed, was not processed. So this
    lead to potential loss of messages and a memory leak. The potential
    loss of messages was IMHO minor, because they would have been dropped
    in most cases due to the queue remaining full, but very few lucky ones
    from the batch may have made it. Anyhow, this has now been changed so
    that the rest of the batch is properly tried to be enqueued and, if
    not possible, destructed.

  • new module mmsnmptrapd, a sample message modification module
    This can be useful to reformat snmptrapd messages and also serves as
    a sample for how to write message modification modules using the
    output module interface. Note that we introduced this new
    functionality directly into the stable release, as it does not
    modify the core and as such cannot have any side-effects if it is
    not used (and thus the risk is solely on users requiring that
    functionality).

  • bugfix: rate-limiting inside imuxsock did not work 100% correct
    reason was that a global config variable was invalidly accessed where a
    listener variable should have been used.
    Also performance-improved the case when rate limiting is turned off (this
    is a very unintrusive change, thus done directly to the stable version).

  • bugfix: $myhostname not available in RainerScript (and no error message)
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=233

  • bugfix: memory and file descriptor leak in stream processing
    Leaks could occur under some circumstances if the file stream handler
    errored out during the open call. Among others, this could cause very
    big memory leaks if there were a problem with unreadable disk queue
    files. In regard to the memory leak, this
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256

  • bugfix: doc for impstats had wrong config statements
    also, config statements were named a bit inconsistent, resolved that
    problem by introducing an alias and only documenting the consistent
    statements
    Thanks to Marcin for bringing up this problem.

  • bugfix: IPv6-address could not be specified in omrelp
    this was due to improper parsing of “:”
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250

  • bugfix: TCP connection invalidly aborted when messages needed to be
    discarded (due to QUEUE_FULL or similar problem)

  • bugfix: $LocalHostName was not honored under all circumstances
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=258

  • bugfix(minor): improper template function call in syslogd.c

Changelog for 5.7.10 (v5-beta)

adisconteamChangelog, , ,

Version 5.7.10 [V5-BETA] (rgerhards), 2011-03-29

  • bugfix: ompgsql did not work properly with ANSI SQL strings
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=229
  • bugfix: rsyslog did not build with –disable-regexp configure option
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=243
  • bugfix: PRI was invalid on Solaris for message from local log socket
  • enhance: added $BOM system property to ease writing byte order masks
  • bugfix: RFC5424 parser confused by empty structured data
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=237
  • bugfix: error return from strgen caused abort, now causes action to be
    ignored (just like a failed filter)
  • new sample plugin for a strgen to generate sql statement consumable
    by a database plugin
  • bugfix: strgen could not be used together with database outputs
    because the sql/stdsql option could not be specified. This has been
    solved by permitting the strgen to include the opton inside its name.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=195
Scroll to top