Search Results for: queue

Changelog for 5.9.0 (v5-devel)

Version 5.9.0 [V5-DEVEL] (rgerhards), 2011-06-08

  • imfile: added $InputFileMaxLinesAtOnce directive
  • enhanced imfile to support input batching
  • added capability for imtcp and imptcp to activate keep-alive packets
    at the socket layer. This has not been added to imttcp, as the latter is
    only an experimental module, and one which did not prove to be useful.
    reference: http://kb.monitorware.com/post20791.html

  • added support to control KEEPALIVE settings in imptcp
    this has not yet been added to imtcp, but could be done on request.

  • $ActionName is now also used for naming of queues in impstats
    as well as in the debug output

  • bugfix: do not open files with full privileges, if privs will be dropped
    This make the privilege drop code more bulletproof, but breaks Ubuntu’s
    work-around for log files created by external programs with the wrong
    user and/or group. Note that it was long said that this “functionality”
    would break once we go for serious privilege drop code, so hopefully
    nobody still depends on it (and, if so, they lost…).

  • bugfix: pipes not opened in full priv mode when privs are to be dropped
  • this begins a new devel branch for v5
  • better handling of queue i/o errors in disk queues. This is kind of a
    bugfix, but a very intrusive one, this it goes into the devel version
    first. Right now, “file not found” is handled and leads to the new
    emergency mode, in which disk action is stopped and the queue run
    in direct mode. An error message is emited if this happens.

  • added support for user-level PRI provided via systemd
  • added new config directive $InputTCPFlowControl to select if tcp
    received messages shall be flagged as light delayable or not.

  • enhanced omhdfs to support batching mode. This permits to increase
    performance, as we now call the HDFS API with much larger message
    sizes and far more infrequently

  • bugfix: failover did not work correctly if repeated msg reduction was on
    affected directive was: $ActionExecOnlyWhenPreviousIsSuspended on
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=236

Changelog for 5.8.1 (v5-stable)

Version 5.8.1 [V5-stable] (rgerhards), 2011-05-19

  • bugfix: invalid processing in QUEUE_FULL condition
    If the the multi-submit interface was used and a QUEUE_FULL condition
    occured, the failed message was properly destructed. However, the
    rest of the input batch, if it existed, was not processed. So this
    lead to potential loss of messages and a memory leak. The potential
    loss of messages was IMHO minor, because they would have been dropped
    in most cases due to the queue remaining full, but very few lucky ones
    from the batch may have made it. Anyhow, this has now been changed so
    that the rest of the batch is properly tried to be enqueued and, if
    not possible, destructed.

  • new module mmsnmptrapd, a sample message modification module
    This can be useful to reformat snmptrapd messages and also serves as
    a sample for how to write message modification modules using the
    output module interface. Note that we introduced this new
    functionality directly into the stable release, as it does not
    modify the core and as such cannot have any side-effects if it is
    not used (and thus the risk is solely on users requiring that
    functionality).

  • bugfix: rate-limiting inside imuxsock did not work 100% correct
    reason was that a global config variable was invalidly accessed where a
    listener variable should have been used.
    Also performance-improved the case when rate limiting is turned off (this
    is a very unintrusive change, thus done directly to the stable version).

  • bugfix: $myhostname not available in RainerScript (and no error message)
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=233

  • bugfix: memory and file descriptor leak in stream processing
    Leaks could occur under some circumstances if the file stream handler
    errored out during the open call. Among others, this could cause very
    big memory leaks if there were a problem with unreadable disk queue
    files. In regard to the memory leak, this
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256

  • bugfix: doc for impstats had wrong config statements
    also, config statements were named a bit inconsistent, resolved that
    problem by introducing an alias and only documenting the consistent
    statements
    Thanks to Marcin for bringing up this problem.

  • bugfix: IPv6-address could not be specified in omrelp
    this was due to improper parsing of “:”
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250

  • bugfix: TCP connection invalidly aborted when messages needed to be
    discarded (due to QUEUE_FULL or similar problem)

  • bugfix: $LocalHostName was not honored under all circumstances
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=258

  • bugfix(minor): improper template function call in syslogd.c

Changelog for 5.8.0 (v5-stable)

Version 5.8.0 [V5-stable] (rgerhards), 2011-04-12

This is the new v5-stable branch, importing all feature from the 5.7.x versions. To see what has changed in regard to the previous v5-stable, check the Changelog for 5.7.x below.

  • bugfix: race condition in deferred name resolution
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=238
    Special thanks to Marcin for his persistence in helping to solve this
    bug.
  • bugfix: DA queue was never shutdown once it was started
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=241

Changelog for 6.1.4 (devel)

Version 6.1.4 [DEVEL] (rgerhards), 2011-02-18

  • bugfix/omhdfs: directive $OMHDFSFileName rendered unusable
    due to a search and replace-induced bug ;)
  • bugfix: minor race condition in action.c – considered cosmetic
    This is considered cosmetic as multiple threads tried to write exactly
    the same value into the same memory location without sync. The method
    has been changed so this can no longer happen.
  • added pmsnare parser module (written by David Lang)
  • enhanced imfile to support non-cancel input termination
  • improved systemd socket activation thanks to Marius Tomaschweski
  • improved error reporting for $WorkDirectory
    non-existance and other detectable problems are now reported,
    and the work directory is NOT set in this case
  • bugfix: pmsnare causded abort under some conditions
  • bugfix: abort if imfile reads file line of more than 64KiB
    Thanks to Peter Eisentraut for reporting and analysing this problem.
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221
  • bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
    when in disk-assisted mode. This especially affected imfile, which
    created unnecessarily queue files if a large set of input file data was
    to process.
  • bugfix: very long running actions could prevent shutdown under some
    circumstances. This has now been solved, at least for common
    situations.
  • bugfix: fixed compile problem due to empty structs
    this occured only on some platforms/compilers. thanks to Dražen Kačar
    for the fix

Changelog for 5.7.4 (v5-beta)

Version 5.7.4 [V5-BETA] (rgerhards), 2011-02-17

  • added pmsnare parser module (written by David Lang)
  • enhanced imfile to support non-cancel input termination
  • improved systemd socket activation thanks to Marius Tomaschweski
  • improved error reporting for $WorkDirectory
    non-existance and other detectable problems are now reported,
    and the work directory is NOT set in this case
  • bugfix: pmsnare causded abort under some conditions
  • bugfix: abort if imfile reads file line of more than 64KiB
    Thanks to Peter Eisentraut for reporting and analysing this problem.
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221
  • bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
    when in disk-assisted mode. This especially affected imfile, which
    created unnecessarily queue files if a large set of input file data was
    to process.
  • bugfix: very long running actions could prevent shutdown under some
    circumstances. This has now been solved, at least for common
    situations.
  • bugfix: fixed compile problem due to empty structs
    this occured only on some platforms/compilers. thanks to Dražen Kačar
    for the fix

Changelog for 6.1.2 (devel)

Version 6.1.2  [DEVEL] (rgerhards), 2010-12-16

  • added experimental support for log normalizaton (via liblognorm) support for normalizing log messages has been added in the form of mmnormalize. The core engine (property replacer, filter engine) has been enhanced to support properties from normalized events.

Note: this is EXPERIMENTAL code. It is currently known that there are issues if the functionality is used with

  • disk-based queues
  • asynchronous action queues

You can not use the new functionality together with these features.

This limitation will be removed in later releases. However, we preferred to release early, so that one can experiment with the new feature set and accepted the price that this means the full set of functionality is not yet available. If not used together with these features, log normalizing should be pretty stable.

  • enhanced testing tool tcpflood now supports sending via UDP and the capability to run multiple iterations and generate statistics data records
  • bugfix: potential abort when output modules with different parameter passing modes were used in configured output modules

Periodic statistics on rsyslog counters

Since rsyslog 5.7.0 (V5-Devel) there is a new module called impstats.

This module provides periodic output of rsyslog internal counters. Note that the whole statistics system is currently under development. So availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version).

The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). Be sure that your configuration records the counter messages (default is syslog.info).

Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be severe (for high-load environments).

Please find below some simple steps how to use that module.

Of course you have to name it additionally in the configure. The configure should look like this (please note that the parameters can be different to your configure, the important thing is ––enable-impstats; this example is for Fedora 13 )

./configure --enable-impstats --sbindir=/sbin --libdir=/lib

The next steps are make and make install

make
make install

Furthermore we have to add that module in the rsyslog config. Below you will find a short excerpt of a sample config.

#### Modules ####
$ModLoad imuxsock
$ModLoad imklog
#$ModLoad immark

$ModLoad impstats
$PStatsInterval 600
$PStatsSeverity 7

syslog.debug  /var/log/rsyslog-stats

$ModLoad impstats – tells rsyslog to load the module impstats

$PStatsInterval 600 – is a configuration directive of impstats

$PStatsSeverity 7 – is also a configuration directive of impstats

Here you will find all information about the impstats module and its configuration directives.

If we have a look at the mentioned output file rsyslog-stats we can see the results of the static module.

The content of that file should look like this

Sep 17 11:43:49 localhost rsyslogd-pstats: umxsock: submitted=16
Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2

At the actual point all objects are shown in the results. Every Object has its own counter like

umxsock: (= object) submitted=16 (=counter; 16 logs received by the object umxsock). The main queue of rsyslog is also shown -> main Q with the parameters size (messages in the queue), enqueued (all received messages), full (how often was the queue full) and maxqsize (the maximal amount of messages in the queue).

Please be sure that the features and options of that module will be develeloped soon.

All information about the periodic statistics module you will find in the documentation. There is also a list of rsyslog impstats counters available on the web site.

Changelog for 3.2.2 (v3-stable)

Version 3.22.2 [v3-stable] (rgerhards), 2010-08-05

  • bugfix: comment char (‘#’) in literal terminated script parsing and thus could not be used.
    but tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=119
  • enhance: imrelp now also provides remote peer’s IP address[if librelp != 1.0.0 is used]
  • bugfix: sending syslog messages with zip compression did not work
  • bugfix: potential hang condition on queue shutdown
  • bugfix: segfault on startup when -q or -Q option was given
    bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=157
    Thanks to Jonas Nogueira for reporting this bug.
  • clarified use of $ActionsSendStreamDriver[AuthMode/PermittedPeers]in doc set (require TLS drivers)
  • bugfix: $CreateDirs variable not properly initialized, default thuswas random (but most often “on”)
  • bugfix: potential segfault when -p command line option was usedthanks to varmojfekoj for pointing me at this bug
  • bugfix: programname filter in ! configuration can not be resetThanks to Kiss Gabor for the patch.
Scroll to top