rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Search Results for: queue

Changelog for 5.8.1 (v5-stable)

adisconteamChangelog, , , ,

Version 5.8.1 [V5-stable] (rgerhards), 2011-05-19

  • bugfix: invalid processing in QUEUE_FULL condition
    If the the multi-submit interface was used and a QUEUE_FULL condition
    occured, the failed message was properly destructed. However, the
    rest of the input batch, if it existed, was not processed. So this
    lead to potential loss of messages and a memory leak. The potential
    loss of messages was IMHO minor, because they would have been dropped
    in most cases due to the queue remaining full, but very few lucky ones
    from the batch may have made it. Anyhow, this has now been changed so
    that the rest of the batch is properly tried to be enqueued and, if
    not possible, destructed.

  • new module mmsnmptrapd, a sample message modification module
    This can be useful to reformat snmptrapd messages and also serves as
    a sample for how to write message modification modules using the
    output module interface. Note that we introduced this new
    functionality directly into the stable release, as it does not
    modify the core and as such cannot have any side-effects if it is
    not used (and thus the risk is solely on users requiring that
    functionality).

  • bugfix: rate-limiting inside imuxsock did not work 100% correct
    reason was that a global config variable was invalidly accessed where a
    listener variable should have been used.
    Also performance-improved the case when rate limiting is turned off (this
    is a very unintrusive change, thus done directly to the stable version).

  • bugfix: $myhostname not available in RainerScript (and no error message)
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=233

  • bugfix: memory and file descriptor leak in stream processing
    Leaks could occur under some circumstances if the file stream handler
    errored out during the open call. Among others, this could cause very
    big memory leaks if there were a problem with unreadable disk queue
    files. In regard to the memory leak, this
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256

  • bugfix: doc for impstats had wrong config statements
    also, config statements were named a bit inconsistent, resolved that
    problem by introducing an alias and only documenting the consistent
    statements
    Thanks to Marcin for bringing up this problem.

  • bugfix: IPv6-address could not be specified in omrelp
    this was due to improper parsing of “:”
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250

  • bugfix: TCP connection invalidly aborted when messages needed to be
    discarded (due to QUEUE_FULL or similar problem)

  • bugfix: $LocalHostName was not honored under all circumstances
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=258

  • bugfix(minor): improper template function call in syslogd.c

Changelog for 5.8.0 (v5-stable)

adisconteamChangelog, , ,

Version 5.8.0 [V5-stable] (rgerhards), 2011-04-12

This is the new v5-stable branch, importing all feature from the 5.7.x versions. To see what has changed in regard to the previous v5-stable, check the Changelog for 5.7.x below.

  • bugfix: race condition in deferred name resolution
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=238
    Special thanks to Marcin for his persistence in helping to solve this
    bug.
  • bugfix: DA queue was never shutdown once it was started
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=241

Changelog for 6.1.4 (devel)

adisconteamChangelog, , ,

Version 6.1.4 [DEVEL] (rgerhards), 2011-02-18

  • bugfix/omhdfs: directive $OMHDFSFileName rendered unusable
    due to a search and replace-induced bug ;)
  • bugfix: minor race condition in action.c – considered cosmetic
    This is considered cosmetic as multiple threads tried to write exactly
    the same value into the same memory location without sync. The method
    has been changed so this can no longer happen.
  • added pmsnare parser module (written by David Lang)
  • enhanced imfile to support non-cancel input termination
  • improved systemd socket activation thanks to Marius Tomaschweski
  • improved error reporting for $WorkDirectory
    non-existance and other detectable problems are now reported,
    and the work directory is NOT set in this case
  • bugfix: pmsnare causded abort under some conditions
  • bugfix: abort if imfile reads file line of more than 64KiB
    Thanks to Peter Eisentraut for reporting and analysing this problem.
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221
  • bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
    when in disk-assisted mode. This especially affected imfile, which
    created unnecessarily queue files if a large set of input file data was
    to process.
  • bugfix: very long running actions could prevent shutdown under some
    circumstances. This has now been solved, at least for common
    situations.
  • bugfix: fixed compile problem due to empty structs
    this occured only on some platforms/compilers. thanks to Dražen Kačar
    for the fix

Changelog for 5.7.4 (v5-beta)

adisconteamChangelog, , ,

Version 5.7.4 [V5-BETA] (rgerhards), 2011-02-17

  • added pmsnare parser module (written by David Lang)
  • enhanced imfile to support non-cancel input termination
  • improved systemd socket activation thanks to Marius Tomaschweski
  • improved error reporting for $WorkDirectory
    non-existance and other detectable problems are now reported,
    and the work directory is NOT set in this case
  • bugfix: pmsnare causded abort under some conditions
  • bugfix: abort if imfile reads file line of more than 64KiB
    Thanks to Peter Eisentraut for reporting and analysing this problem.
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221
  • bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
    when in disk-assisted mode. This especially affected imfile, which
    created unnecessarily queue files if a large set of input file data was
    to process.
  • bugfix: very long running actions could prevent shutdown under some
    circumstances. This has now been solved, at least for common
    situations.
  • bugfix: fixed compile problem due to empty structs
    this occured only on some platforms/compilers. thanks to Dražen Kačar
    for the fix

Changelog for 6.1.2 (devel)

Adiscon SupportChangelog, , , ,

Version 6.1.2  [DEVEL] (rgerhards), 2010-12-16

  • added experimental support for log normalizaton (via liblognorm) support for normalizing log messages has been added in the form of mmnormalize. The core engine (property replacer, filter engine) has been enhanced to support properties from normalized events.

Note: this is EXPERIMENTAL code. It is currently known that there are issues if the functionality is used with

  • disk-based queues
  • asynchronous action queues

You can not use the new functionality together with these features.

This limitation will be removed in later releases. However, we preferred to release early, so that one can experiment with the new feature set and accepted the price that this means the full set of functionality is not yet available. If not used together with these features, log normalizing should be pretty stable.

  • enhanced testing tool tcpflood now supports sending via UDP and the capability to run multiple iterations and generate statistics data records
  • bugfix: potential abort when output modules with different parameter passing modes were used in configured output modules

Periodic statistics on rsyslog counters

adisconteamFAQ, , ,

Since rsyslog 5.7.0 (V5-Devel) there is a new module called impstats.

This module provides periodic output of rsyslog internal counters. Note that the whole statistics system is currently under development. So availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version).

The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). Be sure that your configuration records the counter messages (default is syslog.info).

Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be severe (for high-load environments).

Please find below some simple steps how to use that module.

Of course you have to name it additionally in the configure. The configure should look like this (please note that the parameters can be different to your configure, the important thing is ––enable-impstats; this example is for Fedora 13 )

./configure --enable-impstats --sbindir=/sbin --libdir=/lib

The next steps are make and make install

make
make install

Furthermore we have to add that module in the rsyslog config. Below you will find a short excerpt of a sample config.

#### Modules ####
$ModLoad imuxsock
$ModLoad imklog
#$ModLoad immark

$ModLoad impstats
$PStatsInterval 600
$PStatsSeverity 7

syslog.debug  /var/log/rsyslog-stats

$ModLoad impstats – tells rsyslog to load the module impstats

$PStatsInterval 600 – is a configuration directive of impstats

$PStatsSeverity 7 – is also a configuration directive of impstats

Here you will find all information about the impstats module and its configuration directives.

If we have a look at the mentioned output file rsyslog-stats we can see the results of the static module.

The content of that file should look like this

Sep 17 11:43:49 localhost rsyslogd-pstats: umxsock: submitted=16
Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2

At the actual point all objects are shown in the results. Every Object has its own counter like

umxsock: (= object) submitted=16 (=counter; 16 logs received by the object umxsock). The main queue of rsyslog is also shown -> main Q with the parameters size (messages in the queue), enqueued (all received messages), full (how often was the queue full) and maxqsize (the maximal amount of messages in the queue).

Please be sure that the features and options of that module will be develeloped soon.

All information about the periodic statistics module you will find in the documentation. There is also a list of rsyslog impstats counters available on the web site.

Changelog for 3.2.2 (v3-stable)

adisconteamChangelog, , ,

Version 3.22.2 [v3-stable] (rgerhards), 2010-08-05

  • bugfix: comment char (‘#’) in literal terminated script parsing and thus could not be used.
    but tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=119
  • enhance: imrelp now also provides remote peer’s IP address[if librelp != 1.0.0 is used]
  • bugfix: sending syslog messages with zip compression did not work
  • bugfix: potential hang condition on queue shutdown
  • bugfix: segfault on startup when -q or -Q option was given
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=157
    Thanks to Jonas Nogueira for reporting this bug.
  • clarified use of $ActionsSendStreamDriver[AuthMode/PermittedPeers]in doc set (require TLS drivers)
  • bugfix: $CreateDirs variable not properly initialized, default thuswas random (but most often “on”)
  • bugfix: potential segfault when -p command line option was usedthanks to varmojfekoj for pointing me at this bug
  • bugfix: programname filter in ! configuration can not be resetThanks to Kiss Gabor for the patch.

rsyslog 5.5.5 (devel) released

adisconteamNews, Release Announcement, ,

Hi all,

We have just released rsyslog 5.5.5, a member of the devel branch. This is a bug-fixing release which contains a single fix that solves a potential hang condition on system shutdown when infinite action retries are configured for an asynchronous action using a queue in disk-assisted mode and the action was suspended. This is probably not a very common case, but a configuration recommended by our doc samples. If you do not use such a configuration, there is no need to update at this time. See Changelog for more details.

ChangeLog:

http://www.rsyslog.com/article461/

Download:

http://www.rsyslog.com/Downloads-req-viewdownloaddetails-lid-204.phtml

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

Scroll to top