Log files should be processed by rsyslog. Here is some information on how the file monitor works. This will only describe setting up the Text File Input Module. Further configuration like processing rules or output methods will not be described.
Things to think about
The configuration given here should be placed on top of the rsyslog.conf file.
Config Statements
$ModLoad imfile # needs to be done just once
# File 1 $InputFileName /path/to/file1 $InputFileTag tag1: $InputFileStateFile stat-file1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor # File 2 $InputFileName /path/to/file2 $InputFileTag tag2: $InputFileStateFile stat-file2 $InputRunFileMonitor # ... and so on ... # # check for new lines every 10 seconds $InputFilePollingInterval 10
How it works
The configuration for using the Text File Input Module is very extensive. At the beginning of your rsyslog configuration file, you always load the modules. There you need to load the module for Text File Input as well. Like all other modules, this has to be made just once:
$ModLoad imfile
Next up comes a sequence of several parameters. Each one after another belong to one file only:
# File 1 $InputFileName /path/to/file1 $InputFileTag tag1: $InputFileStateFile stat-file1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor
InputFileName specifies, the path and name of the text file that should be monitored. The file name must be absolute.
InputFileTag will set a tag in front of each message pulled from the file. If you want a colon after the tag you must set it as well, it will not be added automatically.
InputFileStatFile will create a file where rsyslog keeps track of the position it currently is in a file. You only need to set the filename. This file always is created in the rsyslog working directory (configurable via $WorkDirectory). This file is important so rsyslog will not pull messages from the beginning of the file when being restarted.
InputFileSeverity will give all log messages of a file the same severity. This is optional. By default all mesages will be set to “notice”.
InputFileFacility gives alle log messages of a file the same facility. Again, this is optional. By default all messages will be set to “local0″.
InputRunFileMonitor will execute the statements above. No parameters are required, but the statement itself is mandatory.
These statements are needed for monitoring a file. If you want to monitor another file the statements must be repeated.
Since the files cannot be monitored in genuine real time (which generates too much processing effort) you need to set a polling interval:
$InputFilePollingInterval 10
This is a global setting and it defines the interval in which the log files will be polled. By default this value is set to 10 seconds. If you want this to get more near realtime, you can decrease the value, though this is not suggested due to increasing processing load. Setting this to 0 is supported, but not suggested. Rsyslog will continue reading the file as long as there are unprocessed messages in it. The interval only takes effect once rsyslog reaches the end of the file.
Important
Sequence of configuration statements is very important. Invalid sequence of otherwise perfectly legal configuration statements can lead to totally wrong results. Further, the InputFileStatFile needs to be unique for every file that is monitored. If not, strange things could happen. Additionaly the directive InputRunFilemonitor is absolutely necessary. If you forget this, no monitoring will take place. It has to be used for every file that should be monitored.
Tags: config snippet, Config Snippets, imfile, rsyslog, text file input module
How hard would it be to make rsyslog monitor a named pipe? If this were to happen, then applications could be configured to write log file output to the named pipe, which just looks like any other file to the application. If rsyslog were to monitor the pipe, the thread would basically sleep until a message arrives, process the message, then go back to sleep. There is no need to have a polling interval and the processing is immediate. I have had some success with this by implementing the following simple script:
mkfifo /path/to/named_pipe
while true; do
logger -p local0.notice /path/to/named_pipe
Another possible useful feature would be to allow arbitrary facility names instead of the usual hard coded list. We have many applications that need to send messages to syslog so that we can centralize them on one server, but for various reasons we do not what their messages combined into common files. Being able to specify arbitrary facilities would allow us to direct messages from different applications back into separate files on the central log server.
Thanks,
-Mark
I solved the “arbitrary facility” suggestion by just reading further into the documentation (I am new to rsyslog, so I am still finding out about it capabilities). But the “named pipe” functionality could still be a useful addition.
-Mark