rsyslog message flow (video tutorial)
| This is a video tutorial on how messages flow through the rsyslog engine. Explains types of modules, queues and the “big picture”. This is actually the first video tutorial, and I guess you’ll notice that. Anyhow, I think it is useful for understanding the system. Continue reading “rsyslog message flow (video tutorial)” |
ChangeLog for 4.0.1 (devel)
Version 4.1.0 [DEVEL] (rgerhards), 2008-11-18
********************************* WARNING ********************************* This version has a slightly different on-disk format for message entries. As a consequence, old queue files being read by this version may have an invalid output timestamp, which could result to some malfunction inside the output driver. It is recommended to drain queues with the previous version before switching to this one. ********************************* WARNING *********************************
- greatly enhanced performance when compared to v3.
- added configuration directive “HUPisRestart” which enables to configure
HUP to be either a full restart or “just” a leightweight way to
close open files. - enhanced legacy syslog parser to detect year if part of the timestamp
the format is based on what Cisco devices seem to emit. - added a setting “$OptimizeForUniprocessor” to enable users to turn off
pthread_yield calls which are counter-productive on multiprocessor
machines (but have been shown to be useful on uniprocessors) - reordered imudp processing. Message parsing is now done as part of main
message queue worker processing (was part of the input thread)
This should also improve performance, as potentially more work is
done in parallel. - bugfix: compressed syslog messages could be slightly mis-uncompressed
if the last byte of the compressed record was a NUL - added $UDPServerTimeRequery option which enables to work with
less acurate timestamps in favor of performance. This enables querying
of the time only every n-th time if imudp is running in the tight
receive loop (aka receiving messsages at a high rate) - doc bugfix: queue doc had wrong parameter name for setting controlling
worker thread shutdown period - restructured rsyslog.conf documentation
- bugfix: memory leak in ompgsql
Thanks to Ken for providing the patch
ChangeLog for 3.18.5 (v3-stable)
Version 3.18.5 (rgerhards), 2008-10-09
- bugfix: imudp input module could cause segfault on HUP
It did not properly de-init a variable acting as a linked list head.
That resulted in trying to access freed memory blocks after the HUP. - bugfix: rsyslogd could hang on HUP
because getnameinfo() is not cancel-safe, but was not guarded against
being cancelled. pthread_cancel() is routinely being called during
HUP processing. - bugfix[minor]: if queue size reached light_delay mark, enqueuing
could potentially be blocked for a longer period of time, which
was not the behaviour desired. - doc bugfix: $ActionExecOnlyWhenPreviousIsSuspended was still misspelled
as $…OnlyIfPrev… in some parts of the documentation. Thanks to
Lorenzo M. Catucci for reporting this bug. - added doc on malformed messages, cause and how to work-around, to the
doc set - added doc on how to build from source repository
ChangeLog for 3.21.5 (devel)
Version 3.21.5 [DEVEL] (rgerhards), 2008-09-30
- performance optimization: unnecessary time() calls during message
parsing removed – thanks to David Lang for his excellent performance
analysis - added new capability to property replacer: multiple immediately
successive field delimiters are treated as a single one.
Thanks to Zhuang Yuyao for the patch. - added message property “inputname”, which contains the name of the
input (module) that generated it. Presence is depending on suport in
each input module (else it is blank). - added system property “$myhostname”, which contains the name of the
local host as it knows itself. - imported a number of fixes and enhancements from the stable and
devel branches, including a fix to a potential segfault on HUP
when using UDP listners - re-enabled gcc builtin atomic operations and added a proper
./configure check - bugfix: potential race condition when adding messages to queue
There was a wrong order of mutex lock operations. It is hard to
believe that really caused problems, but in theory it could and with
threading we often see that theory becomes practice if something is only
used long enough on a fast enough machine with enough CPUs ;) - cleaned up internal debug system code and made it behave better
in regard to multi-threading
ChangeLog for 3.18.4 (v3-stable)
Version 3.18.4 (rgerhards), 2008-09-18
- bugfix: order-of magnitude issue with base-10 size definitions
in config file parser. Could lead to invalid sizes, constraints
etc for e.g. queue files and any other object whose size was specified
in base-10 entities. Did not apply to binary entities. Thanks to
RB for finding this bug and providing a patch. - bugfix: action was not called when system time was set backwards
(until the previous time was reached again). There are still some
side-effects when time is rolled back (A time rollback is really a bad
thing to do, ideally the OS should issue pseudo time (like NetWare did)
when the user tries to roll back time). Thanks to varmojfekoj for this
patch. - doc bugfix: rsyslog.conf man page improved and minor nit fixed
thanks to Lukas Kuklinek for the patch. - bugfix: error code -2025 was used for two different errors. queue full
is now -2074 and -2025 is unique again. (did cause no real problem
except for troubleshooting) - bugfix: default discard severity was incorrectly set to 4, which lead
to discard-on-queue-full to be enabled by default. That could cause
message loss where non was expected. The default has now been changed
to the correct value of 8, which disables the functionality. This
problem applied both to the main message queue and the action queues.
Thanks to Raoul Bhatia for pointing out this problem. - bugfix: option value for legacy -a option could not be specified,
resulting in strange operations. Thanks to Marius Tomaschewski
for the patch. - bugfix: colon after date should be ignored, but was not. This has
now been corrected. Required change to the internal ParseTIMESTAMP3164()
interface.
Changelog for 3.21.4 (devel)
Version 3.21.4 [DEVEL] (rgerhards), 2008-09-04
- removed compile time fixed message size limit (was 2K), limit can now
be set via $MaxMessageSize global config directive (finally gotten rid
of MAXLINE ;)) - enhanced doc for $ActionExecOnlyEveryNthTimeTimeout
- integrated a number of patches from 3.18.4, namely
- bugfix: order-of magnitude issue with base-10 size definitions
in config file parser. Could lead to invalid sizes, constraints
etc for e.g. queue files and any other object whose size was specified
in base-10 entities. Did not apply to binary entities. Thanks to
RB for finding this bug and providing a patch. - bugfix: action was not called when system time was set backwards
(until the previous time was reached again). There are still some
side-effects when time is rolled back (A time rollback is really a bad
thing to do, ideally the OS should issue pseudo time (like NetWare did)
when the user tries to roll back time). Thanks to varmojfekoj for this
patch. - doc bugfix: rsyslog.conf man page improved and minor nit fixed
thanks to Lukas Kuklinek for the patch.
- bugfix: order-of magnitude issue with base-10 size definitions
ChangeLog for 3.19.11
Version 3.19.11 [BETA] (rgerhards), 2008-08-25
- This is a refresh of the beta. No beta-specific fixes have been added.
- included fixes from v3-stable (most importantly 3.18.3). Among
others, these are:- bugfix: imfile could cause a segfault upon rsyslogd HUP and termination
Thanks to lperr for an excellent bug report that helped detect this
problem. - enhanced ommysql to support custom port to connect to server
Port can be set via new $ActionOmmysqlServerPort config directive
Note: this was a very minor change and thus deemed appropriate to be
done in the stable release. - bugfix: misspelled config directive, previously was
$MainMsgQueueWorkeTimeoutrThreadShutdown, is now
$MainMsgQueueWorkerTimeoutThreadShutdown. Note that the misspelled
directive is not preserved – if the misspelled directive was used
(which I consider highly unlikely), the config file must be changed.
Thanks to lperr for reporting the bug. - disabled flow control for imuxsock, as it could cause system hangs
under some circumstances. The devel (3.21.3 and above) will
re-enable it and provide enhanced configurability to overcome the
problems if they occur.
- bugfix: imfile could cause a segfault upon rsyslogd HUP and termination
ChangeLog for 3.18.3
Version 3.18.3 (rgerhards), 2008-08-18
- bugfix: imfile could cause a segfault upon rsyslogd HUP and termination
Thanks to lperr for an excellent bug report that helped detect this
problem. - enhanced ommysql to support custom port to connect to server
Port can be set via new $ActionOmmysqlServerPort config directive
Note: this was a very minor change and thus deemed appropriate to be
done in the stable release. - bugfix: misspelled config directive, previously was
$MainMsgQueueWorkeTimeoutrThreadShutdown, is now
$MainMsgQueueWorkerTimeoutThreadShutdown. Note that the misspelled
directive is not preserved – if the misspelled directive was used
(which I consider highly unlikely), the config file must be changed.
Thanks to lperr for reporting the bug. - disabled flow control for imuxsock, as it could cause system hangs
under some circumstances. The devel (3.21.3 and above) will
re-enable it and provide enhanced configurability to overcome the
problems if they occur.
ChangeLog for 3.21.3 (devel)
Version 3.21.3 [DEVEL] (rgerhards), 2008-08-13
- added ability to specify flow control mode for imuxsock
- added ability to execute actions only after the n-th call of the action
This also lead to the addition of two new config directives:
$ActionExecOnlyEveryNthTime and $ActionExecOnlyEveryNthTimeTimeout
This feature is useful, for example, for alerting: it permits you to
send an alert only after at least n occurences of a specific message
have been seen by rsyslogd. This protectes against false positives
due to waiting for additional confirmation. - bugfix: IPv6 addresses could not be specified in forwarding actions
New syntax @[addr]:port introduced to enable that. Root problem was IPv6
addresses contain colons. - somewhat enhanced debugging messages
- imported from 3.18.3:
- enhanced ommysql to support custom port to connect to server
Port can be set via new $ActionOmmysqlServerPort config directive
Note: this was a very minor change and thus deemed appropriate to be
done in the stable release. - bugfix: misspelled config directive, previously was
$MainMsgQueueWorkeTimeoutrThreadShutdown, is now
$MainMsgQueueWorkerTimeoutThreadShutdown. Note that the misspelled
directive is not preserved – if the misspelled directive was used
(which I consider highly unlikely), the config file must be changed.
Thanks to lperr for reporting the bug.
- enhanced ommysql to support custom port to connect to server
