rsyslog 5.8.10 (v5-stable) released
This is primarily a bugfixing release. There were some segfaults in conjunction with $ActionQueueFileName and when the disk-queue was started with a old queue file. Our thanks go to Tomas Heinrich for the Patches. In addition there was a memory leak that has also been fixed.
ChangeLog:
http://www.rsyslog.com/changelog-for-5-8-10-v5-stable/
Download:
http://www.rsyslog.com/rsyslog-5-8-10-v5-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 5.8.10 (v5-stable)
Version 5.8.10 [V5-stable] 2012-04-05
- bugfix: segfault on startup if $actionqueuefilename was missing for disk queue config
Thanks to Tomas Heinrich for the patch. - bugfix: segfault if disk-queue was started up with old queue file
Thanks to Tomas Heinrich for the patch. - bugfix: memory leak in array passing output module mode
Action’s with directives
This snippet will show, how Action directives need to be applied to work properly. We will show it with the RELP output module. RELP should ensure a safe and loss-free transmission between two machines. But if not configured properly, messages may get lost anyway. This is mainly meant for any client side configuration.
First of all you have to enable the RELP module.
To load the module use this:
$ModLoad omrelp
To make sure, messages will not get dropped in the event the receiver is not available, we basically need the following directives. Additionaly, the queued messages should get saved to the harddrive if the client service needs to shut down. It is followed by a forwarding action via RELP to our remote server.
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* :omrelp:192.168.152.2:20514
Attention: The directives are only valid for the next configured action! So you have to set the directives each time you use a new action.
Here is an example with two actions.
#first action
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
:syslogtag, isequal, “app1” :omrelp:192.168.152.2:20514#second action
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
:syslogtag, isequal, “app2” :omrelp:192.168.152.3:20514
As you can see, we have the whole block of directives mulitple times. But this time, we filter the message for the syslogtag and have the diffenrently tagged messages sent to different receivers. Now if the receiver is not available, sending the messages will be retried until it is back up again. If the local rsyslog needs to shut down, all queued messages get written to disk without being lost.
Changelog for 5.8.9 (v5-stable)
Version 5.8.9 [V5-stable] 2012-03-15
- added tool to recover disk queue if .qi file is missing (recover_qi.pl)
Thanks to Kaiwang Chen for contributing this tool - bugfix: stopped DA queue was never processed after a restart due to a
regression from statistics module - added better doc for statsobj interface
Thanks to Kaiwang Chen for his suggestions and analysis in regard to the
stats subsystem.
rsyslog 5.8.9 (v5-stable) released
This is primarily a maintenance release. It also provides a new tool that permits the recovery of disk queues, when the queue information file (.qi) has been lost. Thanks to Kaiwang Chen for contributing this tool.
There was an important flaw in 5.8.8 that caused disk queue content to be lost after a shutdown and restart of rsyslogd. So if you run this version, an upgrade to 5.8.9 is highly suggested.
ChangeLog:
http://www.rsyslog.com/changelog-for-5-8-9-v5-stable/
Download:
http://www.rsyslog.com/rsyslog-5-8-9-v5-stable/
As always, feedback is appreciated.
Best regards,
Tim Eifler
rsyslog statistics counter
Rsyslog supports statistic counters via the impstats module. It is important to know that impstats and friends only provides an infrastructure where core components and plugins can register statistics counter. This FAQ entry tries to describe all counters available, but please keep in mind that there may exist that we do not know about.
When interpreting rsyslog statistics, please keep in mind that statistics records are processed as regular syslog messages. As such, the statistics messages themselves increment counters when they are emitted via the regular syslog stream, which is the default (and so counters keep slowly increasing even if there is absolutely no other traffic). Also keep in mind that a busy rsyslog system is very dynamic. Most importantly, this means that the counters may not be 100% consistent, but some slight differences may exist. Avoiding such inconsistencies would be possible only at the price of a very tight locking discipline, which would cause serious performance bottlenecks. Thus, this is not done. Finally, though extremely unlikely, some counters may experience an overflow and restart at 0 for that reasons. However, most counters are 64-bit, so this is extremely unlikely. Those which are not 64 bit are typically taken from some internal data structure that uses lower bits for performance reasons and guards against overflow.
The listing starts with the core component or plugin that creates the counters and than specifies various counters that exist for the sub-entities. The listing below is extended as new counters are added. Some counters probably do not exist in older releases of rsyslog.
Below you can find all available core components and plugins. Please note that every core component or plugin are linked to a information site.
Queue
Actions
PLUGINS
Changelog for 5.8.6 (v5-stable)
Version 5.8.6 [V5-stable] 2011-10-21
- bugfix: missing whitespace after property-based filter was not detected
- bugfix: $OMFileFlushInterval period was doubled – now using correct value
- bugfix: ActionQueue could malfunction due to index error
Thanks to Vlad Grigorescu for the patch - bugfix: $ActionExecOnlyOnce interval did not work properly
Thanks to Tomas Heinrich for the patch - bugfix: race condition when extracting program name, APPNAME, structured data and PROCID (RFC5424 fields) could lead to invalid characters e.g. in dynamic file names or during forwarding (general malfunction of these fields in templates, mostly under heavy load)
- bugfix: imuxsock did no longer ignore message-provided timestamp, if so configured (the *default*). Lead to no longer sub-second timestamps.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=281 - bugfix: omfile returns fatal error code for things that go really wrong previously, RS_RET_RESUME was returned, which lead to a loop inside the rule engine as omfile could not really recover.
- bugfix: imfile did invalid system call under some circumstances when a file that was to be monitored did not exist BUT the state file actually existed. Mostly a cosmetic issue. Root cause was incomplete error checking in stream.c; so patch may affect other code areas.
- bugfix: rsyslogd -v always said 64 atomics were not present
Thanks to mono_matsuko for the patch
rsyslog 6.3.3 config format improvements
In rsyslog 6.3.3, the config processor has finally changed. The old legacy processor (and with it the early RainerScript implementation) is thrown out and has been replaced by the so-called RainerScript processor (why that crazy name?). This is an extremely important step for rsyslog, as it now has the foundation for a much better and intuitive rsyslog.conf format. However, most of that can not be seen in 6.3.3, as it requires more work, especially in the plugin arena. Still, there are a couple of smaller improvements available.
Most importantly, the performance of script based filters has been considerably enhanced. Preliminary testing shows a three times speedup (we’ll do more benchmarking at a later stage; there is also still lots of room for optimization ;-)).
The ugliness of continuation lines has been removed. They may still be used, and this may make a lot of sense with some actions, but you are usually no longer forced to use continuation lines. Take this config snippet from a leading distro:
if ( \ /* kernel up to warning except of firewall */ \ ($syslogfacility-text == 'kern') and \ ($syslogseverity <= 4 /* warning */ ) and not \ ($msg contains 'IN=' and $msg contains 'OUT=') \ ) or ( \ /* up to errors except of facility authpriv */ \ ($syslogseverity <= 3 /* errors */ ) and not \ ($syslogfacility-text == 'authpriv') \ ) \ then /dev/tty10 & |/dev/xconsole
This can now be written as follows:
if ( /* kernel up to warning except of firewall */ ($syslogfacility-text == 'kern') and ($syslogseverity <= 4 /* warning */ ) and not ($msg contains 'IN=' and $msg contains 'OUT=') ) or ( /* up to errors except of facility authpriv */ ($syslogseverity <= 3 /* errors */ ) and not ($syslogfacility-text == 'authpriv') ) then /dev/tty10 & |/dev/xconsole
Changelog for 4.6.6 (v4-stable)
Version 4.6.6 [v4-stable] (rgerhards), 2011-06-24
- bugfix: memory leak in imtcp & subsystems under some circumstances
This leak is tied to error conditions which lead to incorrect cleanup
of some data structures. [backport from v6, limited testing under v4] - bugfix: invalid processing in QUEUE_FULL condition
If the the multi-submit interface was used and a QUEUE_FULL condition
occured, the failed message was properly destructed. However, the
rest of the input batch, if it existed, was not processed. So this
lead to potential loss of messages and a memory leak. The potential
loss of messages was IMHO minor, because they would have been dropped
in most cases due to the queue remaining full, but very few lucky ones
from the batch may have made it. Anyhow, this has now been changed so
that the rest of the batch is properly tried to be enqueued and, if
not possible, destructed. - bugfix: invalid storage type for config variables
- bugfix: stream driver mode was not correctly set on tcp ouput on big
endian systems.
thanks varmojfekoj for the patch - bugfix: IPv6-address could not be specified in omrelp
this was due to improper parsing of “:”
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250 - bugfix: memory and file descriptor leak in stream processing
Leaks could occur under some circumstances if the file stream handler
errored out during the open call. Among others, this could cause very
big memory leaks if there were a problem with unreadable disk queue
files. In regard to the memory leak, this
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256 - bugfix: imfile potentially duplicates lines
This can happen when 0 bytes are read from the input file, and some
writer appends data to the file BEFORE we check if a rollover happens.
The check for rollover uses the inode and size as a criterion. So far,
we checked for equality of sizes, which is not given in this scenario,
but that does not indicate a rollover. From the source code comments:
Note that when we check the size, we MUST NOT check for equality.
The reason is that the file may have been written right after we
did try to read (so the file size has increased). That is NOT in
indicator of a rollover (this is an actual bug scenario we
experienced). So we need to check if the new size is smaller than
what we already have seen!
Also, under some circumstances an invalid truncation was detected. This
code has now been removed, a file change (and thus resent) is only
detected if the inode number changes. - bugfix: a couple of problems that imfile had on some platforms, namely
Ubuntu (not their fault, but occured there) - bugfix: imfile utilizes 32 bit to track offset. Most importantly,
this problem can not experienced on Fedora 64 bit OS (which has
64 bit long’s!) - bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: omlibdbi did not use password from rsyslog.con
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=271 - some improvements thanks to clang’s static code analyzer
- overall cleanup (mostly unnecessary writes and otherwise unused stuff)
- bugfix: fixed a very remote problem in msg.c which could occur when
running under extremely low memory conditions
Changelog for 5.9.0 (v5-devel)
Version 5.9.0 [V5-DEVEL] (rgerhards), 2011-06-08
- imfile: added $InputFileMaxLinesAtOnce directive
- enhanced imfile to support input batching
- added capability for imtcp and imptcp to activate keep-alive packets
at the socket layer. This has not been added to imttcp, as the latter is
only an experimental module, and one which did not prove to be useful.
reference: http://kb.monitorware.com/post20791.html - added support to control KEEPALIVE settings in imptcp
this has not yet been added to imtcp, but could be done on request. - $ActionName is now also used for naming of queues in impstats
as well as in the debug output - bugfix: do not open files with full privileges, if privs will be dropped
This make the privilege drop code more bulletproof, but breaks Ubuntu’s
work-around for log files created by external programs with the wrong
user and/or group. Note that it was long said that this “functionality”
would break once we go for serious privilege drop code, so hopefully
nobody still depends on it (and, if so, they lost…). - bugfix: pipes not opened in full priv mode when privs are to be dropped
- this begins a new devel branch for v5
- better handling of queue i/o errors in disk queues. This is kind of a
bugfix, but a very intrusive one, this it goes into the devel version
first. Right now, “file not found” is handled and leads to the new
emergency mode, in which disk action is stopped and the queue run
in direct mode. An error message is emited if this happens. - added support for user-level PRI provided via systemd
- added new config directive $InputTCPFlowControl to select if tcp
received messages shall be flagged as light delayable or not. - enhanced omhdfs to support batching mode. This permits to increase
performance, as we now call the HDFS API with much larger message
sizes and far more infrequently - bugfix: failover did not work correctly if repeated msg reduction was on
affected directive was: $ActionExecOnlyWhenPreviousIsSuspended on
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=236