Changelog for 5.9.1 (v5-devel)
Version 5.9.1 [V5-DEVEL] (rgerhards), 2011-06-30
- added support for obtaining timestamp for kernel message from message
If the kernel time-stamps messages, time is now take from that
timestamp instead of the system time when the message was read. This
provides much better accuracy. Thanks to Lennart Poettering for
suggesting this feature and his help during implementation. - added support for obtaining timestamp from system for imuxsock
This permits to read the time a message was submitted to the system
log socket. Most importantly, this is provided in microsecond resolution.
So we are able to obtain high precision timestampis even for messages
that were – as is usual – not formatted with them. This also simplifies
things in regard to local time calculation in chroot environments.
Many thanks to Lennart Poettering for suggesting this feature,
providing some guidance on implementing it and coordinating getting the
necessary support into the Linux kernel. - bugfix: timestamp was incorrectly calculated for timezones with minute
offset
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=271 - bugfix: problems in failover action handling
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=270
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=254 - bugfix: mutex was invalidly left unlocked during action processing
At least one case where this can occur is during thread shutdown, which
may be initiated by lower activity. In most cases, this is quite
unlikely to happen. However, if it does, data structures may be
corrupted which could lead to fatal failure and segfault. I detected
this via a testbench test, not a user report. But I assume that some
users may have had unreproducable aborts that were cause by this bug. - bugfix: memory leak in imtcp & subsystems under some circumstances
This leak is tied to error conditions which lead to incorrect cleanup
of some data structures. [backport from v6] - bugfix/improvement:$WorkDirectory now gracefully handles trailing slashes
Changelog for 4.6.6 (v4-stable)
Version 4.6.6 [v4-stable] (rgerhards), 2011-06-24
- bugfix: memory leak in imtcp & subsystems under some circumstances
This leak is tied to error conditions which lead to incorrect cleanup
of some data structures. [backport from v6, limited testing under v4] - bugfix: invalid processing in QUEUE_FULL condition
If the the multi-submit interface was used and a QUEUE_FULL condition
occured, the failed message was properly destructed. However, the
rest of the input batch, if it existed, was not processed. So this
lead to potential loss of messages and a memory leak. The potential
loss of messages was IMHO minor, because they would have been dropped
in most cases due to the queue remaining full, but very few lucky ones
from the batch may have made it. Anyhow, this has now been changed so
that the rest of the batch is properly tried to be enqueued and, if
not possible, destructed. - bugfix: invalid storage type for config variables
- bugfix: stream driver mode was not correctly set on tcp ouput on big
endian systems.
thanks varmojfekoj for the patch - bugfix: IPv6-address could not be specified in omrelp
this was due to improper parsing of “:”
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250 - bugfix: memory and file descriptor leak in stream processing
Leaks could occur under some circumstances if the file stream handler
errored out during the open call. Among others, this could cause very
big memory leaks if there were a problem with unreadable disk queue
files. In regard to the memory leak, this
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256 - bugfix: imfile potentially duplicates lines
This can happen when 0 bytes are read from the input file, and some
writer appends data to the file BEFORE we check if a rollover happens.
The check for rollover uses the inode and size as a criterion. So far,
we checked for equality of sizes, which is not given in this scenario,
but that does not indicate a rollover. From the source code comments:
Note that when we check the size, we MUST NOT check for equality.
The reason is that the file may have been written right after we
did try to read (so the file size has increased). That is NOT in
indicator of a rollover (this is an actual bug scenario we
experienced). So we need to check if the new size is smaller than
what we already have seen!
Also, under some circumstances an invalid truncation was detected. This
code has now been removed, a file change (and thus resent) is only
detected if the inode number changes. - bugfix: a couple of problems that imfile had on some platforms, namely
Ubuntu (not their fault, but occured there) - bugfix: imfile utilizes 32 bit to track offset. Most importantly,
this problem can not experienced on Fedora 64 bit OS (which has
64 bit long’s!) - bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: omlibdbi did not use password from rsyslog.con
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=271 - some improvements thanks to clang’s static code analyzer
- overall cleanup (mostly unnecessary writes and otherwise unused stuff)
- bugfix: fixed a very remote problem in msg.c which could occur when
running under extremely low memory conditions
Changelog for 5.8.2. (v5-stable)
Version 5.8.2 [V5-stable] (rgerhards), 2011-06-21
- bugfix: problems in failover action handling
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=270
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=254 - bugfix: mutex was invalidly left unlocked during action processing
At least one case where this can occur is during thread shutdown, which
may be initiated by lower activity. In most cases, this is quite
unlikely to happen. However, if it does, data structures may be
corrupted which could lead to fatal failure and segfault. I detected
this via a testbench test, not a user report. But I assume that some
users may have had unreproducable aborts that were cause by this bug. - bugfix: memory leak in imtcp & subsystems under some circumstances
This leak is tied to error conditions which lead to incorrect cleanup
of some data structures. [backport from v6] - bugfix/improvement:$WorkDirectory now gracefully handles trailing slashes
Changelog for 6.1.9
Version 6.1.9 [BETA] (rgerhards), 2011-06-14
- bugfix: memory leak in imtcp & subsystems under some circumstances
This leak is tied to error conditions which lead to incorrect cleanup
of some data structures. [backport from v6.3] - bugfix: $ActionFileDefaultTemplate did not work
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=262
Changelog for 5.9.0 (v5-devel)
Version 5.9.0 [V5-DEVEL] (rgerhards), 2011-06-08
- imfile: added $InputFileMaxLinesAtOnce directive
- enhanced imfile to support input batching
- added capability for imtcp and imptcp to activate keep-alive packets
at the socket layer. This has not been added to imttcp, as the latter is
only an experimental module, and one which did not prove to be useful.
reference: http://kb.monitorware.com/post20791.html - added support to control KEEPALIVE settings in imptcp
this has not yet been added to imtcp, but could be done on request. - $ActionName is now also used for naming of queues in impstats
as well as in the debug output - bugfix: do not open files with full privileges, if privs will be dropped
This make the privilege drop code more bulletproof, but breaks Ubuntu’s
work-around for log files created by external programs with the wrong
user and/or group. Note that it was long said that this “functionality”
would break once we go for serious privilege drop code, so hopefully
nobody still depends on it (and, if so, they lost…). - bugfix: pipes not opened in full priv mode when privs are to be dropped
- this begins a new devel branch for v5
- better handling of queue i/o errors in disk queues. This is kind of a
bugfix, but a very intrusive one, this it goes into the devel version
first. Right now, “file not found” is handled and leads to the new
emergency mode, in which disk action is stopped and the queue run
in direct mode. An error message is emited if this happens. - added support for user-level PRI provided via systemd
- added new config directive $InputTCPFlowControl to select if tcp
received messages shall be flagged as light delayable or not. - enhanced omhdfs to support batching mode. This permits to increase
performance, as we now call the HDFS API with much larger message
sizes and far more infrequently - bugfix: failover did not work correctly if repeated msg reduction was on
affected directive was: $ActionExecOnlyWhenPreviousIsSuspended on
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=236
Changelog for 5.8.1 (v5-stable)
Version 5.8.1 [V5-stable] (rgerhards), 2011-05-19
- bugfix: invalid processing in QUEUE_FULL condition
If the the multi-submit interface was used and a QUEUE_FULL condition
occured, the failed message was properly destructed. However, the
rest of the input batch, if it existed, was not processed. So this
lead to potential loss of messages and a memory leak. The potential
loss of messages was IMHO minor, because they would have been dropped
in most cases due to the queue remaining full, but very few lucky ones
from the batch may have made it. Anyhow, this has now been changed so
that the rest of the batch is properly tried to be enqueued and, if
not possible, destructed. - new module mmsnmptrapd, a sample message modification module
This can be useful to reformat snmptrapd messages and also serves as
a sample for how to write message modification modules using the
output module interface. Note that we introduced this new
functionality directly into the stable release, as it does not
modify the core and as such cannot have any side-effects if it is
not used (and thus the risk is solely on users requiring that
functionality). - bugfix: rate-limiting inside imuxsock did not work 100% correct
reason was that a global config variable was invalidly accessed where a
listener variable should have been used.
Also performance-improved the case when rate limiting is turned off (this
is a very unintrusive change, thus done directly to the stable version). - bugfix: $myhostname not available in RainerScript (and no error message)
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=233 - bugfix: memory and file descriptor leak in stream processing
Leaks could occur under some circumstances if the file stream handler
errored out during the open call. Among others, this could cause very
big memory leaks if there were a problem with unreadable disk queue
files. In regard to the memory leak, this
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256 - bugfix: doc for impstats had wrong config statements
also, config statements were named a bit inconsistent, resolved that
problem by introducing an alias and only documenting the consistent
statements
Thanks to Marcin for bringing up this problem. - bugfix: IPv6-address could not be specified in omrelp
this was due to improper parsing of “:”
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=250 - bugfix: TCP connection invalidly aborted when messages needed to be
discarded (due to QUEUE_FULL or similar problem) - bugfix: $LocalHostName was not honored under all circumstances
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=258 - bugfix(minor): improper template function call in syslogd.c
Changelog for 5.7.10 (v5-beta)
Version 5.7.10 [V5-BETA] (rgerhards), 2011-03-29
- bugfix: ompgsql did not work properly with ANSI SQL strings
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=229 - bugfix: rsyslog did not build with –disable-regexp configure option
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=243 - bugfix: PRI was invalid on Solaris for message from local log socket
- enhance: added $BOM system property to ease writing byte order masks
- bugfix: RFC5424 parser confused by empty structured data
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=237 - bugfix: error return from strgen caused abort, now causes action to be
ignored (just like a failed filter) - new sample plugin for a strgen to generate sql statement consumable
by a database plugin - bugfix: strgen could not be used together with database outputs
because the sql/stdsql option could not be specified. This has been
solved by permitting the strgen to include the opton inside its name.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=195
Using the Text File Input Module
Log files should be processed by rsyslog. Here is some information on how the file monitor works. This will only describe setting up the Text File Input Module. Further configuration like processing rules or output methods will not be described.
Things to think about
The configuration given here should be placed on top of the rsyslog.conf file.
Config Statements
module(load="imfile" PollingInterval="10") # needs to be done just once. PollingInterval is a module directive and is only set once when loading the module
# File 1 input(type="imfile" File="/path/to/file1" Tag="tag1" StateFile="/var/spool/rsyslog/statefile1" Severity="error" Facility="local7") # File 2 input(type="imfile" File="/path/to/file2" Tag="tag2" StateFile="/var/spool/rsyslog/statefile2") # ... and so on ... #
How it works
The configuration for using the Text File Input Module is very extensive. At the beginning of your rsyslog configuration file, you always load the modules. There you need to load the module for Text File Input as well. Like all other modules, this has to be made just once. Please note that the directive PollingInterval is a module directive which needs to be set when loading the module.
module(load="imfile" PollingInterval="10")
Next up comes the input and its parameters. We configure a input of a certain type and then set the parameters to be used by this input. This is basically the same principle for all inputs:
# File 1 input(type="imfile" File="/path/to/file1" Tag="tag1" StateFile="/var/spool/rsyslog/statefile1" Severity="error" Facility="local7")
File specifies, the path and name of the text file that should be monitored. The file name must be absolute.
Tag will set a tag in front of each message pulled from the file. If you want a colon after the tag you must set it as well, it will not be added automatically.
StateFile will create a file where rsyslog keeps track of the position it currently is in a file. You only need to set the filename. This file always is created in the rsyslog working directory (configurable via $WorkDirectory). This file is important so rsyslog will not pull messages from the beginning of the file when being restarted.
Severity will give all log messages of a file the same severity. This is optional. By default all mesages will be set to “notice”.
Facility gives alle log messages of a file the same facility. Again, this is optional. By default all messages will be set to “local0”.
These statements are needed for monitoring a file. There are other statements described in the doc, which you might want to use. If you want to monitor another file the statements must be repeated.
Since the files cannot be monitored in genuine real time (which generates too much processing effort) you need to set a polling interval:
PollingInterval 10
This is a module setting and it defines the interval in which the log files will be polled. By default this value is set to 10 seconds. If you want this to get more near realtime, you can decrease the value, though this is not suggested due to increasing processing load. Setting this to 0 is supported, but not suggested. Rsyslog will continue reading the file as long as there are unprocessed messages in it. The interval only takes effect once rsyslog reaches the end of the file.
Important
The StateFile needs to be unique for every file that is monitored. If not, strange things could happen.
Changelog for 6.1.5 (v6 devel)
Version 6.1.5 [DEVEL] (rgerhards), 2011-03-04
- improved testbench
- enhanced imtcp to use a pool of worker threads to process incoming
messages. This enables higher processing rates, especially in the TLS
case (where more CPU is needed for the crypto functions) - added support for TLS (in anon mode) to tcpflood
- improved TLS error reporting
- improved TLS startup (Diffie-Hellman bits do not need to be generated,
as we do not support full anon key exchange — we always need certs) - bugfix: fixed a memory leak and potential abort condition
this could happen if multiple rulesets were used and some output batches
contained messages belonging to more than one ruleset.
fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=226
fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=218 - bugfix: memory leak when $RepeatedMsgReduction on was used
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=225 - bugfix: potential abort condition when $RepeatedMsgReduction set to on
as well as potentially in a number of other places where MsgDup() was
used. This only happened when the imudp input module was used and it
depended on name resolution not yet had taken place. In other words,
this was a strange problem that could lead to hard to diagnose
instability. So if you experience instability, chances are good that
this fix will help.
Changelog for 6.1.4 (devel)
Version 6.1.4 [DEVEL] (rgerhards), 2011-02-18
- bugfix/omhdfs: directive $OMHDFSFileName rendered unusable
due to a search and replace-induced bug ;) - bugfix: minor race condition in action.c – considered cosmetic
This is considered cosmetic as multiple threads tried to write exactly
the same value into the same memory location without sync. The method
has been changed so this can no longer happen. - added pmsnare parser module (written by David Lang)
- enhanced imfile to support non-cancel input termination
- improved systemd socket activation thanks to Marius Tomaschweski
- improved error reporting for $WorkDirectory
non-existance and other detectable problems are now reported,
and the work directory is NOT set in this case - bugfix: pmsnare causded abort under some conditions
- bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
when in disk-assisted mode. This especially affected imfile, which
created unnecessarily queue files if a large set of input file data was
to process. - bugfix: very long running actions could prevent shutdown under some
circumstances. This has now been solved, at least for common
situations. - bugfix: fixed compile problem due to empty structs
this occured only on some platforms/compilers. thanks to Dražen Kačar
for the fix