Using the Text File Input Module
Log files should be processed by rsyslog. Here is some information on how the file monitor works. This will only describe setting up the Text File Input Module. Further configuration like processing rules or output methods will not be described.
Things to think about
The configuration given here should be placed on top of the rsyslog.conf file.
Config Statements
module(load="imfile" PollingInterval="10") # needs to be done just once. PollingInterval is a module directive and is only set once when loading the module
# File 1 input(type="imfile" File="/path/to/file1" Tag="tag1" StateFile="/var/spool/rsyslog/statefile1" Severity="error" Facility="local7") # File 2 input(type="imfile" File="/path/to/file2" Tag="tag2" StateFile="/var/spool/rsyslog/statefile2") # ... and so on ... #
How it works
The configuration for using the Text File Input Module is very extensive. At the beginning of your rsyslog configuration file, you always load the modules. There you need to load the module for Text File Input as well. Like all other modules, this has to be made just once. Please note that the directive PollingInterval is a module directive which needs to be set when loading the module.
module(load="imfile" PollingInterval="10")
Next up comes the input and its parameters. We configure a input of a certain type and then set the parameters to be used by this input. This is basically the same principle for all inputs:
# File 1 input(type="imfile" File="/path/to/file1" Tag="tag1" StateFile="/var/spool/rsyslog/statefile1" Severity="error" Facility="local7")
File specifies, the path and name of the text file that should be monitored. The file name must be absolute.
Tag will set a tag in front of each message pulled from the file. If you want a colon after the tag you must set it as well, it will not be added automatically.
StateFile will create a file where rsyslog keeps track of the position it currently is in a file. You only need to set the filename. This file always is created in the rsyslog working directory (configurable via $WorkDirectory). This file is important so rsyslog will not pull messages from the beginning of the file when being restarted.
Severity will give all log messages of a file the same severity. This is optional. By default all mesages will be set to “notice”.
Facility gives alle log messages of a file the same facility. Again, this is optional. By default all messages will be set to “local0”.
These statements are needed for monitoring a file. There are other statements described in the doc, which you might want to use. If you want to monitor another file the statements must be repeated.
Since the files cannot be monitored in genuine real time (which generates too much processing effort) you need to set a polling interval:
PollingInterval 10
This is a module setting and it defines the interval in which the log files will be polled. By default this value is set to 10 seconds. If you want this to get more near realtime, you can decrease the value, though this is not suggested due to increasing processing load. Setting this to 0 is supported, but not suggested. Rsyslog will continue reading the file as long as there are unprocessed messages in it. The interval only takes effect once rsyslog reaches the end of the file.
Important
The StateFile needs to be unique for every file that is monitored. If not, strange things could happen.
Changelog for 6.1.5 (v6 devel)
Version 6.1.5 [DEVEL] (rgerhards), 2011-03-04
- improved testbench
- enhanced imtcp to use a pool of worker threads to process incoming
messages. This enables higher processing rates, especially in the TLS
case (where more CPU is needed for the crypto functions) - added support for TLS (in anon mode) to tcpflood
- improved TLS error reporting
- improved TLS startup (Diffie-Hellman bits do not need to be generated,
as we do not support full anon key exchange — we always need certs) - bugfix: fixed a memory leak and potential abort condition
this could happen if multiple rulesets were used and some output batches
contained messages belonging to more than one ruleset.
fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=226
fixes: http://bugzilla.adiscon.com/show_bug.cgi?id=218 - bugfix: memory leak when $RepeatedMsgReduction on was used
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=225 - bugfix: potential abort condition when $RepeatedMsgReduction set to on
as well as potentially in a number of other places where MsgDup() was
used. This only happened when the imudp input module was used and it
depended on name resolution not yet had taken place. In other words,
this was a strange problem that could lead to hard to diagnose
instability. So if you experience instability, chances are good that
this fix will help.
Changelog for 6.1.4 (devel)
Version 6.1.4 [DEVEL] (rgerhards), 2011-02-18
- bugfix/omhdfs: directive $OMHDFSFileName rendered unusable
due to a search and replace-induced bug ;) - bugfix: minor race condition in action.c – considered cosmetic
This is considered cosmetic as multiple threads tried to write exactly
the same value into the same memory location without sync. The method
has been changed so this can no longer happen. - added pmsnare parser module (written by David Lang)
- enhanced imfile to support non-cancel input termination
- improved systemd socket activation thanks to Marius Tomaschweski
- improved error reporting for $WorkDirectory
non-existance and other detectable problems are now reported,
and the work directory is NOT set in this case - bugfix: pmsnare causded abort under some conditions
- bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
when in disk-assisted mode. This especially affected imfile, which
created unnecessarily queue files if a large set of input file data was
to process. - bugfix: very long running actions could prevent shutdown under some
circumstances. This has now been solved, at least for common
situations. - bugfix: fixed compile problem due to empty structs
this occured only on some platforms/compilers. thanks to Dražen Kačar
for the fix
Changelog for 5.7.4 (v5-beta)
Version 5.7.4 [V5-BETA] (rgerhards), 2011-02-17
- added pmsnare parser module (written by David Lang)
- enhanced imfile to support non-cancel input termination
- improved systemd socket activation thanks to Marius Tomaschweski
- improved error reporting for $WorkDirectory
non-existance and other detectable problems are now reported,
and the work directory is NOT set in this case - bugfix: pmsnare causded abort under some conditions
- bugfix: abort if imfile reads file line of more than 64KiB
Thanks to Peter Eisentraut for reporting and analysing this problem.
bug tracker: http://bugzilla.adiscon.com/show_bug.cgi?id=221 - bugfix: queue engine did not properly slow down inputs in FULL_DELAY mode
when in disk-assisted mode. This especially affected imfile, which
created unnecessarily queue files if a large set of input file data was
to process. - bugfix: very long running actions could prevent shutdown under some
circumstances. This has now been solved, at least for common
situations. - bugfix: fixed compile problem due to empty structs
this occured only on some platforms/compilers. thanks to Dražen Kačar
for the fix
Changelog for 5.6.3 (v5-stable)
Version 5.6.3 [V5-STABLE] (rgerhards), 2011-01-26
- bugfix: action processor released memory too early, resulting inpotential issue in retry cases (but very unlikely due to anotherbug, which I also fixed — only after the fix this problem here became actually visible).
- bugfix: batch processing flagged invalid message as “bad” under somecircumstances
- bugfix: unitialized variable could cause issues under extreme conditionsplus some minor nits. This was found after a clang static code analyzeranalysis (great tool, and special thanks to Marcin for telling me about it!)
- bugfix: batches which had actions in error were not properly retried inall cases
- bugfix: imfile did duplicate messages under some circumstances
Installing RSyslog 5 on RHEL 4 / 5
To have rsyslog working correctly on RHEL 4 or 5, some conditions have to be met. The method described has been tested with rsyslog 5.7.1.
First of all compile and install the dependencies.
- gnutls-2.8.6.tar.bz2
- libgcrypt-1.4.6.tar.gz
- libgpg-error-1.9.tar.gz
- libtasn1-2.2.tar.gz
After that, you can install rsyslog using the following commands:
./configure PKG_CONFIG_PATH=/usr/local/lib/pkgconfig --enable-gnutls make make install
It could happen, that the install might complain about gnutls.pc. Simply comment out the URL found near the start of the file /usr/local/lib/pkgconfig/gnutls.pc.
Credit for this find goes to Forum member Johann Reinhard (johannreinhard).
Changelog for 5.5.7 (v5-beta)
Version 5.5.7 [V5-BETA] (rgerhards), 2010-08-09
- changed omudpspoof default spoof address to simplify typical use casethanks to David Lang for suggesting this
- doc bugfix: pmlastmsg doc samples had errors
- bugfix[minor]: pmrfc3164sd had invalid name (resided in rsyslog namespace, what should not be the case for a contributed module)
- added omuxsock, which permits to write message to local Unix socketsthis is the counterpart to imuxsock, enabling fast local forwarding
Changelog for 4.6.3 (v4-stable)
Version 4.6.3 [v4-stable] (rgerhards), 2010-07-07
- improvded testbench
- added test with truly random data received via syslog to test robustness
- added new configure option that permits to disable and enable an extended testbench
- bugfix: segfault on HUP when “HUPIsRestart” was set to “on”
thanks varmojfekoj for the patch - bugfix: default for $OMFileFlushOnTXEnd was wrong (“off”).
This, in default mode, caused buffered writing to be used, what means that it looked like no output were written or partial lines. Thanks to Michael Biebl for pointing out this bug. - bugfix: testbench failed when not executed in UTC+1 timezone accidently, the time zone information was kept inside some to-be-checked-for responses
- temporary bugfix replaced by permanent one for message-induced off-by-one error (potential segfault) (see 4.6.2)
The analysis has been completed and a better fix been crafted and integrated. - some doc fixes; incorrect config samples could cause confusion
thanks to Anthony Edwards for pointing the problems out
Regex
This is a regular expression checker especially programmed for rsyslog. Rsyslog uses POSIX ERE (and optionally BRE) expressions. This checker works with the php POSIX ERE
functions. So in theory, there can be a difference between what the engine included in rsyslog (clib) and this web app does. However, in practice the results should be fairly the
same.
Note: you need at least rsyslog version 3.19.5 (for v3) or 4.1.5 (for v4 and above) to use ERE expressions.
To use this tool, paste a sample of the field in question into “Sample Log Line”, write your regular expression and press the button ;) You will then see what the regular expression engine extracts. Also, the result fields contain a property replacer
field definition in theory suitable for copying and pasting into your rsyslog.conf.
‘;
writeTopicBoxTail();
if(0 == 1) echo ‘
‘;
// now check if we got any data to process. If so, do it
if(strlen($regex) == 0 && strlen($sampleData) == 0)
; // no data, nothing to do ;)
else {
writeTopicBoxHdr(‘Regular Expression Result’);
echo ‘
| ‘; if(strlen($regex) == 0 && strlen($sampleData) > 0) echo ‘error, regex must be specified ‘; error, sample log line must be specified ‘; No matches found or invalid search expression. ‘; | ‘; writeSupportAd(); echo ‘ |
‘;
writeTopicBoxTail();
}
}
//echo ‘
'; //print_r($_REQUEST); //echo '
‘;
Changelog for 4.6.2 (v4-stable)
Version 4.6.2 [v4-stable] (rgerhards), 2010-03-26
- new feature: “.” action type added to support writing files to relative pathes (this is primarily meant as a debug aid)
- new feature: $OMFileAsyncWriting directive added it permits to specifiy if asynchronous writing should be done or not
- bugfix(temporary): message-induced off-by-one error (potential segfault)
Some types of malformed messages could trigger an off-by-one error (for example, \0 or \n as the last character, and generally control character escaption is questionable). This is due to not strictly following a the \0 or string counted string paradigm (during the last optimization on the cstring class). As a temporary fix, we have introduced a proper recalculation of the size. However, a final patch is expected in the future. See bug tracker for further details and when the final patch will be available: http://bugzilla.adiscon.com/show_bug.cgi?id=184 Note that the current patch is considered sufficient to solve the situation, but it requires a bit more runtime than desirable. - bugfix: potential segfault in dynafile cache
This bug was triggered by an open failure. The the cache was full and a new entry needed to be placed inside it, a victim for eviction was selected. That victim was freed, then the open of the new file tried. If the open failed, the victim entry was still freed, and the function exited. However, on next invocation and cache search, the victim entry was used as if it were populated, most probably resulting in a segfault. - bugfix: race condition during directory creation
If multiple files try to create a directory at (almost) the same time, some of them may fail. This is a data race and also exists with other processes that may create the same directory. We do now check for this condition and gracefully handle it. - bugfix: potential re-use of free()ed file stream object in omfile
when dynaCache is enabled, the cache is full, a new entry needs to be allocated, thus the LRU discarded, then a new entry is opend and that fails. In that case, it looks like the discarded stream may be reused improperly (based on code analysis, test case and confirmation pending) - added new property replacer option “date-rfc3164-buggyday” primarily to ease migration from syslog-ng. See property replacer doc for details. [backport from 5.5.3 because urgently needed by some]
- improved testbench
- bugfix: invalid buffer write in (file) stream class
currently being accessed buffer could be overwritten with new data. While this probably did not cause access violations, it could case loss and/or duplication of some data (definitely a race with no deterministic outcome) - bugfix: potential hang condition during filestream close predicate was not properly checked when waiting for the background file writer
- bugfix: improper synchronization when “$OMFileFlushOnTXEnd on” was used
Internal data structures were not properly protected due to missing mutex calls. - bugfix: potential data loss during file stream shutdown
- bugfix: potential problems during file stream shutdown
The shutdown/close sequence was not clean, what potentially (but unlikely) could lead to some issues. We have not been able to describe any fatal cases, but there was some bug potential. Sequence has now been straighted out. - bugfix: potential problem (loop, abort) when file write error occured
When a write error occured in stream.c, variable iWritten had the error code but this was handled as if it were the actual number of bytes written. That was used in pointer arithmetic later on, and thus could lead to all sorts of problems. However, this could only happen if the error was EINTR or the file in question was a tty. All other cases were handled properly. Now, iWritten is reset to zero in such cases, resulting in proper retries. - bugfix: $omfileFlushOnTXEnd was turned on when set to off and vice versa due to an invalid check
- bugfix: recent patch to fix small memory leak could cause invalid free.
This could only happen during config file parsing. - bugfix(minor): handling of extremely large strings in dbgprintf() fixed
Previously, it could lead to garbagge output and, in extreme cases, also to segfaults. Note: this was a problem only when debug output was actually enabled, so it caused no problem in production use. - bugfix(minor): BSD_SO_COMPAT query function had some global vars not properly initialized. However, in practice the loader initializes them with zero, the desired value, so there were no actual issue in almost all cases.