Changelog for 5.9.0 (v5-devel)
Version 5.9.0 [V5-DEVEL] (rgerhards), 2011-06-08
- imfile: added $InputFileMaxLinesAtOnce directive
- enhanced imfile to support input batching
- added capability for imtcp and imptcp to activate keep-alive packets
at the socket layer. This has not been added to imttcp, as the latter is
only an experimental module, and one which did not prove to be useful.
reference: http://kb.monitorware.com/post20791.html - added support to control KEEPALIVE settings in imptcp
this has not yet been added to imtcp, but could be done on request. - $ActionName is now also used for naming of queues in impstats
as well as in the debug output - bugfix: do not open files with full privileges, if privs will be dropped
This make the privilege drop code more bulletproof, but breaks Ubuntu’s
work-around for log files created by external programs with the wrong
user and/or group. Note that it was long said that this “functionality”
would break once we go for serious privilege drop code, so hopefully
nobody still depends on it (and, if so, they lost…). - bugfix: pipes not opened in full priv mode when privs are to be dropped
- this begins a new devel branch for v5
- better handling of queue i/o errors in disk queues. This is kind of a
bugfix, but a very intrusive one, this it goes into the devel version
first. Right now, “file not found” is handled and leads to the new
emergency mode, in which disk action is stopped and the queue run
in direct mode. An error message is emited if this happens. - added support for user-level PRI provided via systemd
- added new config directive $InputTCPFlowControl to select if tcp
received messages shall be flagged as light delayable or not. - enhanced omhdfs to support batching mode. This permits to increase
performance, as we now call the HDFS API with much larger message
sizes and far more infrequently - bugfix: failover did not work correctly if repeated msg reduction was on
affected directive was: $ActionExecOnlyWhenPreviousIsSuspended on
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=236
Changelog for 5.8.1 (v5-stable)
Version 5.8.1 [V5-stable] (rgerhards), 2011-05-19
- bugfix: invalid processing in QUEUE_FULL condition
If the the multi-submit interface was used and a QUEUE_FULL condition
occured, the failed message was properly destructed. However, the
rest of the input batch, if it existed, was not processed. So this
lead to potential loss of messages and a memory leak. The potential
loss of messages was IMHO minor, because they would have been dropped
in most cases due to the queue remaining full, but very few lucky ones
from the batch may have made it. Anyhow, this has now been changed so
that the rest of the batch is properly tried to be enqueued and, if
not possible, destructed. - new module mmsnmptrapd, a sample message modification module
This can be useful to reformat snmptrapd messages and also serves as
a sample for how to write message modification modules using the
output module interface. Note that we introduced this new
functionality directly into the stable release, as it does not
modify the core and as such cannot have any side-effects if it is
not used (and thus the risk is solely on users requiring that
functionality). - bugfix: rate-limiting inside imuxsock did not work 100% correct
reason was that a global config variable was invalidly accessed where a
listener variable should have been used.
Also performance-improved the case when rate limiting is turned off (this
is a very unintrusive change, thus done directly to the stable version). - bugfix: $myhostname not available in RainerScript (and no error message)
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=233 - bugfix: memory and file descriptor leak in stream processing
Leaks could occur under some circumstances if the file stream handler
errored out during the open call. Among others, this could cause very
big memory leaks if there were a problem with unreadable disk queue
files. In regard to the memory leak, this
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256 - bugfix: doc for impstats had wrong config statements
also, config statements were named a bit inconsistent, resolved that
problem by introducing an alias and only documenting the consistent
statements
Thanks to Marcin for bringing up this problem. - bugfix: IPv6-address could not be specified in omrelp
this was due to improper parsing of “:”
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250 - bugfix: TCP connection invalidly aborted when messages needed to be
discarded (due to QUEUE_FULL or similar problem) - bugfix: $LocalHostName was not honored under all circumstances
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=258 - bugfix(minor): improper template function call in syslogd.c
Changelog for 5.8.0 (v5-stable)
Version 5.8.0 [V5-stable] (rgerhards), 2011-04-12
This is the new v5-stable branch, importing all feature from the 5.7.x versions. To see what has changed in regard to the previous v5-stable, check the Changelog for 5.7.x below.
- bugfix: race condition in deferred name resolution
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=238
Special thanks to Marcin for his persistence in helping to solve this
bug. - bugfix: DA queue was never shutdown once it was started
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=241
Changelog for 6.1.4 (devel)
Version 6.1.4 [DEVEL] (rgerhards), 2011-02-18
- bugfix/omhdfs: directive $OMHDFSFileName rendered unusable
due to a search and replace-induced bug ;) - bugfix: minor race condition in action.c – considered cosmetic
This is considered cosmetic as multiple threads tried to write exactly
the same value into the same memory location without sync. The method
has been changed so this can no longer happen. - added pmsnare parser module (written by David Lang)
- enhanced imfile to support non-cancel input termination
- improved systemd socket activation thanks to Marius Tomaschweski
- improved error reporting for $WorkDirectory
non-existance and other detectable problems are now reported,
and the work directory is NOT set in this case - bugfix: pmsnare causded abort under some conditions
- bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
when in disk-assisted mode. This especially affected imfile, which
created unnecessarily queue files if a large set of input file data was
to process. - bugfix: very long running actions could prevent shutdown under some
circumstances. This has now been solved, at least for common
situations. - bugfix: fixed compile problem due to empty structs
this occured only on some platforms/compilers. thanks to Dražen Kačar
for the fix
Changelog for 5.7.4 (v5-beta)
Version 5.7.4 [V5-BETA] (rgerhards), 2011-02-17
- added pmsnare parser module (written by David Lang)
- enhanced imfile to support non-cancel input termination
- improved systemd socket activation thanks to Marius Tomaschweski
- improved error reporting for $WorkDirectory
non-existance and other detectable problems are now reported,
and the work directory is NOT set in this case - bugfix: pmsnare causded abort under some conditions
- bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
when in disk-assisted mode. This especially affected imfile, which
created unnecessarily queue files if a large set of input file data was
to process. - bugfix: very long running actions could prevent shutdown under some
circumstances. This has now been solved, at least for common
situations. - bugfix: fixed compile problem due to empty structs
this occured only on some platforms/compilers. thanks to Dražen Kačar
for the fix
rsyslog 6.1.2 (devel) released
Hi all,
We have just released rsyslog 6.1.2, the new v6-devel.
This release has as first release the experimental support for log Continue reading “rsyslog 6.1.2 (devel) released”
Changelog for 6.1.2 (devel)
Version 6.1.2 [DEVEL] (rgerhards), 2010-12-16
- added experimental support for log normalizaton (via liblognorm) support for normalizing log messages has been added in the form of mmnormalize. The core engine (property replacer, filter engine) has been enhanced to support properties from normalized events.
Note: this is EXPERIMENTAL code. It is currently known that there are issues if the functionality is used with
- disk-based queues
- asynchronous action queues
You can not use the new functionality together with these features.
This limitation will be removed in later releases. However, we preferred to release early, so that one can experiment with the new feature set and accepted the price that this means the full set of functionality is not yet available. If not used together with these features, log normalizing should be pretty stable.
- enhanced testing tool tcpflood now supports sending via UDP and the capability to run multiple iterations and generate statistics data records
- bugfix: potential abort when output modules with different parameter passing modes were used in configured output modules
Features
This page represents the features available in rsyslog. The list will show in which version the features have been implemented. The list is not yet complete and therefore a first draft.
| Feature | Available since Version |
| $LocalHostName [name] – this directive permits to overwrite the system hostname with the one specified in the directive. If the directive is given multiple times, all but the last one will be ignored. Please note that startup error messages may be issued with the real hostname. This is by design and not a bug (but one may argue if the design should be changed ;)). | 4.7.4+, 5.7.3+, 6.1.3+. |
| support for Hadoop’s HDFS added (via omhdfs) | 5.7.1 |
| module impstat to emit periodic statistics on rsyslog counters | 5.7.0 |
| imptcp, a simplified, Linux-specific and potentielly fastsyslog plain tcp input plugin (NOT supporting TLS!) | 6.1.0 |
| parser module: # pmrfc3164sd (contributed), supports RFC5424 structured data in RFC3164 messages [untested] | 5.5.6 |
| parser module: # pmlastmsg, which supports the notoriously malformed “last message repeated n times” messages from some syslogd’s (namely sysklogd) | 5.5.6 |
| new module type “string generator”, used to speed up output processing. Expected speedup for (typical) rsyslog processing is roughly 5 to 6 percent compared to using string-based templates. | 5.5.6 |
| Support for OS X | 5.5.2 |
| omruleset output module, which provides great flexibility in action processing. THIS IS A VERY IMPORTANT ADDITION, see its doc for why. | 5.3.4 |
| ability to create custom message parsers | 5.3.4 |
| multi-ruleset support to imudp | 5.3.2 |
| added omuxsock, which permits to write message to local Unix sockets this is the counterpart to imuxsock, enabling fast local forwarding | 4.7.3 |
| added imptcp, a simplified, Linux-specific and potentielly fast syslog plain tcp input plugin (NOT supporting TLS!) | 4.7.3 |
| Support for Solaris | 4.7.1 |
| new feature: “.” action type added to support writing files to relative pathes (this is primarily meant as a debug aid) | 4.6.2 |
| so-called “On Demand Debug” mode, in which debug output can be generated only after the process has started, but not right from the beginning. This is assumed to be useful for hard-to-find bugs. Also improved the doc on the debug system. | 4.5.7 |
| ability for the TCP output action to “rebind” its send socket after sending n messages (actually, it re-opens the connection, the name is used because this is a concept very similiar to $ActionUDPRebindInterval). New config directive actionSendTCPRebindInterval added for the purpose. By default, rebinding is disabled. This is considered useful for load balancers. | 4.5.1 |
| capability to fsync() queue disk files for enhanced reliability (also add’s speed, because you do no longer need to run the whole file system in sync mode) | 4.5.0 |
| ability for the UDP output action to rebind its send socket after sending n messages. New config directive $ActionSendUDPRebindInterval added for the purpose. By default, rebinding is disabled. This is considered useful for load balancers. | 4.3.2 |
| new transactional output module interface which provides superior performance (for databases potentially far superior performance) | 5.1.0 |
| generic network stream server (in addition to rather specific syslog tcp server) | 4.3.2 |
| capability to run multiple tcp listeners (on different ports) | 4.3.1 |
| new output plugin omprog, which permits to start program and feed it (via its stdin) with syslog messages. If the program terminates, it is restarted. | 4.3.0 |
| parser testing suite (still needs to be extended, but a good start) | 4.1.6 |
| function support in RainerScript. That means the engine parses and compile functions, as well as executes a few build-in ones. Dynamic loading and registration of functions is not yet supported – but we now have a good foundation to do that later on. | 4.1.6 |
| support for comma-seperated-values (CSV) output generation (via the “csv” property replace option). The CSV format supported is that from RFC 4180. | 4.1.6 |
| testbed for common config errors | 3.21.1 |
| world’s first implementation of syslog-transport-tls | 3.19.0 |
| support for selectively processing messages only during specific timeframes and spooling them to disk otherwise | 3.17.0 |
| native support for sending mail messages | 3.17.0 |
| support for arbitrary complex boolean, string and arithmetic expressions in message filters | 3.12.0 |
| direct support for Firebird/Interbase, OpenTDS (MS SQL, Sybase), SQLLite, Ingres, Oracle, and mSQL via libdbi, a database abstraction layer (almost as good as native) | 3.11.2 |
| ability to monitor text files and convert their contents into syslog messages (one per line) | 3.11.2 |
| ability to send SNMP trap messages | 3.11.1 |
| easy-to-write to plugin interface | 3.11.1 |
| support for on-demand on-disk spooling of messages that can not be processed fast enough (a great feature for writing massive amounts of syslog messages to a database) | |
| modular design for inputs and outputs – easily extensible via custom plugins | 3.10.0 |
| the sysklogd’s klogd functionality is implemented as the imklog input plug-in. So rsyslog is a full replacement for the sysklogd package | 3.10.0 |
| MySQL and Postgres SQL functionality as a dynamically loadable plug-in | 1.19.0 |
| supports multiple actions per selector/filter condition | 1.18.0 |
| ability to configure backup syslog/database servers – if the primary fails, control is switched to a prioritized list of backups | 1.18.0 |
| ability to use regular expressions in filters | 1.17.0 |
| ability to control repeated line reduction (“last message repeated n times”) on a per selector-line basis | 1.15.1 |
| ability to generate file names and directories (log targets) dynamically, based on many different properties | 1.15.0 |
| support for IPv6 | 1.14.0 |
| ability to limit the allowed network senders | 1.13.5 |
| support for sending and receiving compressed syslog messages | 1.13.0 |
| good timestamp format control; at a minimum, ISO 8601/RFC 3339 second-resolution UTC zone | 1.12.2 |
| very experimental and volatile support for syslog-protocol compliant messages (it is volatile because standardization is currently underway and this is a proof-of-concept implementation to aid this effort) | 1.12.2 |
| massively multi-threaded with dynamic work thread pools that start up and shut themselves down on an as-needed basis (great for high log volume on multicore machines) | 1.12.0 |
| native support for writing to Postgres databases | 1.12.0 |
| powerful BSD-style hostname and program name blocks for easy multi-host support | 1.11.1 |
| support for receiving messages via reliable RFC 3195 delivery (a bit clumpsy to build right now…) | 1.11.0 |
| ability to execute shell scripts on received messages | 1.10.1 |
| support for discarding messages based on filters | 1.10.0 |
| ability to filter on any part of the message, not just facility and severity | 1.10.0 |
| native support for writing to MySQL databases | 0.9.6 |
| support for running multiple rsyslogd instances on a single machine | 0.9.5 |
| support for (plain) tcp based syslog | 0.9.2 |
| support for log files larger than 2gb | 0.9.0 |
| ability to filter out messages based on sequence of arrival | |
| support for TLS-protected syslog (both natively and via stunnel) | |
| support for file size limitation and automatic rollover command execution | |
| ability to reformat message contents and work with substrings | |
| control of log output format, including ability to present channel and priority as visible log data | |
| supports sub-configuration files, which can be automatically read from directories. Includes are specified in the main configuration file | |
| ability to preserve the original hostname in NAT environments and relay chains | |
| control of whether the local hostname or the hostname of the origin of the data is shown as the hostname in the output |
rsyslog 5.7.0 (v5-devel) released
Hi all,
We have just released rsyslog 5.7.0, the new v5-devel. This is a feature-enhancement release. It now supports a module to periodically output statistics about Continue reading “rsyslog 5.7.0 (v5-devel) released”
Periodic statistics on rsyslog counters
Since rsyslog 5.7.0 (V5-Devel) there is a new module called impstats.
This module provides periodic output of rsyslog internal counters. Note that the whole statistics system is currently under development. So availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version).
The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). Be sure that your configuration records the counter messages (default is syslog.info).
Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be severe (for high-load environments).
Please find below some simple steps how to use that module.
Of course you have to name it additionally in the configure. The configure should look like this (please note that the parameters can be different to your configure, the important thing is ––enable-impstats; this example is for Fedora 13 )
./configure --enable-impstats --sbindir=/sbin --libdir=/lib
The next steps are make and make install
make
make install
Furthermore we have to add that module in the rsyslog config. Below you will find a short excerpt of a sample config.
#### Modules #### $ModLoad imuxsock $ModLoad imklog #$ModLoad immark $ModLoad impstats $PStatsInterval 600 $PStatsSeverity 7 syslog.debug /var/log/rsyslog-stats
$ModLoad impstats – tells rsyslog to load the module impstats
$PStatsInterval 600 – is a configuration directive of impstats
$PStatsSeverity 7 – is also a configuration directive of impstats
Here you will find all information about the impstats module and its configuration directives.
If we have a look at the mentioned output file rsyslog-stats we can see the results of the static module.
The content of that file should look like this
Sep 17 11:43:49 localhost rsyslogd-pstats: umxsock: submitted=16 Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2
At the actual point all objects are shown in the results. Every Object has its own counter like
umxsock: (= object) submitted=16 (=counter; 16 logs received by the object umxsock). The main queue of rsyslog is also shown -> main Q with the parameters size (messages in the queue), enqueued (all received messages), full (how often was the queue full) and maxqsize (the maximal amount of messages in the queue).
Please be sure that the features and options of that module will be develeloped soon.
All information about the periodic statistics module you will find in the documentation. There is also a list of rsyslog impstats counters available on the web site.