Storing and forwarding remote messages
In this scenario, we want to store remote sent messages into a specific local file and forward the received messages to another syslog server. Local messages should still be locally stored.
Things to think about
How should this work out? Basically, we need a syslog listener for TCP and one for UDP, the local logging service and two rulesets, one for the local logging and one for the remote logging.
TCP recpetion is not a build-in capability. You need to load the imtcp plugin in order to enable it. This needs to be done only once in rsyslog.conf. Do it right at the top.
Note that the server port address specified in $InputTCPServerRun must match the port address that the clients send messages to.
Config Statements
# Modules
$ModLoad imtcp $ModLoad imudp $ModLoad imuxsock $ModLoad imklog
# Templates
# log every host in its own directory $template RemoteHost,"/var/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"
### Rulesets
# Local Logging $RuleSet local kern.* /var/log/messages *.info;mail.none;authpriv.none;cron.none /var/log/messages authpriv.* /var/log/secure mail.* -/var/log/maillog cron.* /var/log/cron *.emerg * uucp,news.crit /var/log/spooler local7.* /var/log/boot.log # use the local RuleSet as default if not specified otherwise $DefaultRuleset local # Remote Logging $RuleSet remote *.* ?RemoteHost # Send messages we receive to Gremlin *.* @@W.X.Y.Z:514
### Listeners
# bind ruleset to tcp listener $InputTCPServerBindRuleset remote # and activate it: $InputTCPServerRun 10514 $InputUDPServerBindRuleset remote $UDPServerRun 514
How it works
The configuration basically works in 4 parts. First, we load all the modules (imtcp, imudp, imuxsock, imklog). Then we specify the templates for creating files. The we create the rulesets which we can use for the different receivers. And last we set the listeners.
The rulesets are somewhat interesting to look at. The ruleset “local” will be set as the default ruleset. That means, that it will be used by any listener if it is not specified otherwise. Further, this ruleset uses the default log paths vor various facilities and severities.
The ruleset “remote” on the other hand takes care of the local logging and forwarding of all log messages that are received either via UDP or TCP. First, all the messages will be stored in a local file. The filename will be generated with the help of the template at the beginning of our configuration (in our example a rather complex folder structure will be used). After logging into the file, all the messages will be forwarded to another syslog server via TCP.
In the last part of the configuration we set the syslog listeners. We first bind the listener to the ruleset “remote”, then we give it the directive to run the listener with the port to use. In our case we use 10514 for TCP and 514 for UDP.
Important
There are some tricks in this configuration. Since we are actively using the rulesets, we must specify those rulesets before being able to bind them to a listener. That means, the order in the configuration is somewhat different than usual. Usually we would put the listener commands on top of the configuration right after the modules. Now we need to specify the rulesets first, then set the listeners (including the bind command). This is due to the current configuration design of rsyslog. To bind a listener to a ruleset, the ruleset object must at least be present before the listener is created. And that is why we need this kind of order for our configuration.
Using the syslog receiver module
We want to use rsyslog in its general purpose. We want to receive syslog. In rsyslog, we have two possibilities to achieve that.
Things to think about
First of all, we will determine, which way of syslog reception we want to use. We can receive syslog via UDP or TCP. The config statements are each a bit different in both cases.
In most cases, UDP syslog should be fully sufficient and performing well. But, you should be aware, that on large message bursts messages can be dropped. That is not the case with TCP syslog, since the sender and receiver communicate about the arrival of network packets. That makes TCP syslog more suitable for environments where log messages may not be lost or that must ensure PCI compliance (like banks).
Config Statements
module(load="imudp") # needs to be done just once input(type="imudp" port="514")
and
module(load="imtcp") # needs to be done just once input(type="imtcp" port="514")
How it works
The configuration part for the syslog server is pretty simple basically. Though, there are some parameters that can be set for both modules. But this is not necessary in most cases.
In general, first the module needs to be loaded. This is done via the directive module().
module(load="imudp / imtcp")
The module must be loaded, because the directives and functions rely on it. Just by using the right commands, rsyslog will not know where to get the functionality code from.
And next is the command that runs the syslog server itself is called input().
input(type="imudp" port="514") input(type="imtcp" port="514")
Basically, the command says to run a syslog server on a specific port. Depending on the command, you can easily determine the UDP and the TCP server.
You can of course use both types of syslog server at the same time, too. You just need to load both modules for that and configure the server command to listen to specific ports. Then you can receive both UDP and TCP syslog.
Important
In general, we suggest to use TCP syslog. It is way more reliable than UDP syslog and still pretty fast. The main reason is, that UDP might suffer of message loss. This happens when the syslog server must receive large bursts of messages. If the system buffer for UDP is full, all other messages will be dropped. With TCP, this will not happen. But sometimes it might be good to have a UDP server configured as well. That is, because some devices (like routers) are not able to send TCP syslog by design. In that case, you would need both syslog server types to have everything covered. If you need both syslog server types configured, please make sure they run on proper ports. By default UDP syslog is received on port 514. TCP syslog needs a different port because often the RPC service is using this port as well.
Using the Text File Input Module
Log files should be processed by rsyslog. Here is some information on how the file monitor works. This will only describe setting up the Text File Input Module. Further configuration like processing rules or output methods will not be described.
Things to think about
The configuration given here should be placed on top of the rsyslog.conf file.
Config Statements
module(load="imfile" PollingInterval="10") # needs to be done just once. PollingInterval is a module directive and is only set once when loading the module
# File 1 input(type="imfile" File="/path/to/file1" Tag="tag1" StateFile="/var/spool/rsyslog/statefile1" Severity="error" Facility="local7") # File 2 input(type="imfile" File="/path/to/file2" Tag="tag2" StateFile="/var/spool/rsyslog/statefile2") # ... and so on ... #
How it works
The configuration for using the Text File Input Module is very extensive. At the beginning of your rsyslog configuration file, you always load the modules. There you need to load the module for Text File Input as well. Like all other modules, this has to be made just once. Please note that the directive PollingInterval is a module directive which needs to be set when loading the module.
module(load="imfile" PollingInterval="10")
Next up comes the input and its parameters. We configure a input of a certain type and then set the parameters to be used by this input. This is basically the same principle for all inputs:
# File 1 input(type="imfile" File="/path/to/file1" Tag="tag1" StateFile="/var/spool/rsyslog/statefile1" Severity="error" Facility="local7")
File specifies, the path and name of the text file that should be monitored. The file name must be absolute.
Tag will set a tag in front of each message pulled from the file. If you want a colon after the tag you must set it as well, it will not be added automatically.
StateFile will create a file where rsyslog keeps track of the position it currently is in a file. You only need to set the filename. This file always is created in the rsyslog working directory (configurable via $WorkDirectory). This file is important so rsyslog will not pull messages from the beginning of the file when being restarted.
Severity will give all log messages of a file the same severity. This is optional. By default all mesages will be set to “notice”.
Facility gives alle log messages of a file the same facility. Again, this is optional. By default all messages will be set to “local0”.
These statements are needed for monitoring a file. There are other statements described in the doc, which you might want to use. If you want to monitor another file the statements must be repeated.
Since the files cannot be monitored in genuine real time (which generates too much processing effort) you need to set a polling interval:
PollingInterval 10
This is a module setting and it defines the interval in which the log files will be polled. By default this value is set to 10 seconds. If you want this to get more near realtime, you can decrease the value, though this is not suggested due to increasing processing load. Setting this to 0 is supported, but not suggested. Rsyslog will continue reading the file as long as there are unprocessed messages in it. The interval only takes effect once rsyslog reaches the end of the file.
Important
The StateFile needs to be unique for every file that is monitored. If not, strange things could happen.
How to write to a local socket?
One member of the rsyslog comunity wrote:
I’d like to forward via a local UNIX domain socket, instead. I think I understand how to configure the ‘imuxsock’ module so my unprivileged instance reads from a non-standard socket location. But I can’t figure out how to tell my root instance to forward via a local domain socket.
I didn’t figure out a completely RSyslog-native method, but another poster’s message pointed me toward ‘socat’ and ‘omprog’, which I have working, now. (It would be really nice if RSyslog could support this natively, though.)
In case anyone else wants to set this up, maybe this will save you some effort. I’m also interested in any comments/criticisms about this method, I’d love to hear suggestions for better ways to make this work.
Also, I rolled it all up into a Fedora/EL RPM spec, and I’ll send it on to anyone who’s interested–just ask.
Setup steps:
- Install the ‘socat’ utility.
- Build RSyslog with the ‘–enable-omprog’ ./configure flag.
- Create two separate RSyslog config files, one for the ‘root’ instance (writes to the socket) and a second for the ‘unprivileged’ instance (reads from the socket).
- Rewrite your RSyslog init script to start two separate daemon instances, one using each config file (and separate PID files, too).
- Create the user ‘rsyslogd’ and the group ‘rsyslogd’.
- Set permissions/ownerships as needed to allow the user ‘rsyslogd’ to write to the file ‘/var/log/rsyslog.log’
- Create an executable script called
'/usr/libexec/rsyslogd/omprog_socat' that contains the lines:
#!/bin/bash
/usr/bin/socat -t0 -T0 -lydaemon -d - UNIX-SENDTO:/dev/log
- The ‘root’ instance config file should contain (modifying the output actions to taste):
$ModLoad imklog
$ModLoad omprog
$Template FwdViaUNIXSocket,"<%pri%>%syslogtag%%msg%"
$ActionOMProgBinary /usr/libexec/rsyslogd/omprog_socat
*.* :omprog:;FwdViaUNIXSocket - The ‘unprivileged’ instance config file should contain (modifying the output actions to taste):
$ModLoad imuxsock
$PrivDropToUser rsyslogd
$PrivDropToGroup rsyslogd
*.* /var/log/rsyslog.logThe ‘root’ daemon can only accept input from the kernel message buffer, and nothing else (especially not the syslog socket (/dev/log) or any network sockets). The unprivileged user will handle all of local and network log messages. To merge the kernel logs into the same data channel as everything else, here’s what happens:
[During the RSyslog daemons’ startup]
A) At startup, the ‘root’ daemon’s ‘imklog’ module starts listening for kernel messages (via ‘/prog/kmsg’), and its ‘omprog’ module starts an instance of ‘socat’ (called via the ‘omprog_socat’ wrapper), establishing a persistent one-way IO connection where ‘omprog’ pipes its output to the STDIN of ‘socat’.
- (Note that this same ‘socat’ instance remains running throughout the life of the RSyslog daemon, handling everything ‘omprog’ outputs. Contrast this, efficiency-wise, against the built-in ‘subshell’ module [the ‘^/path/to/program’ action], which runs a separate instance instance of the child program for each message.)
B) At startup, the ‘unprivileged’ daemon’s ‘imuxsock’ module opens the system logging socket (‘/dev/log’) and starts listening for incoming log messages from other programs.
- [During normal operation]1) The kernel buffer produces a message string on ‘/proc/kmsg’.2) The ‘root’ RSyslog daemon reads the message from ‘/proc/kmsg’, assigning it the priority number of ‘kern.info’ and the string tag ‘kernel’.3) The ‘root’ daemon prepends the priority number and tag as a header to the message string, and then passes it to the ‘omprog’ module for output (via persistent pipe) to the running ‘socat’ instance.4) The ‘socat’ instance receives the header-framed message and sends it to the system logging socket (‘/dev/log’).
5) The ‘unprivileged’ RSyslog daemon reads the message from ‘/dev/log’, assigning it the priority and tag given in the message header, plus all of the other properties (timestamp, hostname, etc.) a message object should have.
6) The ‘unprivileged’ daemon formats the message and writes it to the output file.
The only real difference I can see in the forwarded messages is that the ‘source’ property is set to ‘imuxsock’ instead of ‘imklog’. I don’t think that’s a real problem, though, since the priority and tag are still distinct.
Writing specific messages to a file and discarding them
Messages with the text “error” inside the text part of the message shall be written to a specific file. They shall not be written to any other file or be processed in any other way.
Things to think about
The configuration given here should be placed on top of the rsyslog.conf file.
Config Statements
:msg, contains, "error" /var/log/error.log
& ~
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
How it works
The configuration uses a property-based filter to see if the string “error is contained” inside the MSG part of the syslog message. If so, the message is written to /var/log/error.log. The next line then discards all messages that have been written. Thus, no additional rules will be applied to the message. As such, it will not be written to /var/log/other.log.
Note the difference to this invalid sequence:
*.* /var/log/other.log
:msg, contains, "error" /var/log/error.log
& ~
Here everything is first written to /var/log/other.log and only then the message content is checked. In the later case, the message with “error” in them will be written to both files.
Important
Sequence of configuration statements is very important. Invalid sequence of otherwise perfectly legal configuration statements can lead to totally wrong results.
Sending Messages to a Remote Syslog Server
In this recipe, we forward messages from one system to another one. Typical use cases are:
- the local system does not store any messages (e.g. has not sufficient space to do so)
- there is a (e.g. legal) requirement to consolidate all logs on a single system
- the server may run some advanced alerting rules, and needs to have a full picture or network activity to work well
- you want to get the logs to a different system in a different security domain (to prevent attackers from hiding their tracks)
- and many more …
In our case, we forward all messages to the remote system. Note that by applying different filters, you may only forward select entries to the remote system. Also note that you can include as many forwarding actions as you like. For example, if you need to have a backup central server, you can simply forward to both of them, using two different forwarding actions.
To learn how to configure the remote server, see recipe Receiving Messages from a Remote System.
Config Statements
# this is the simplest forwarding action:
*.* action(type="omfwd" target="192.0.2.1" port="10514" protocol="tcp")
# it is equivalent to the following obsolete legacy format line:
*.* @@192.0.2.1:10514 # do NOT use this any longer!
# Note: if the remote system is unreachable, processing will
# block here and discard messages after a while
# so a better use is
*.* action(type="omfwd" target="192.0.2.2" port="10514" protocol="tcp"
action.resumeRetryCount="100"
queue.type="linkedList" queue.size="10000")
# this will de-couple the sending from the other logging actions,
# and prevent delays when the remote system is not reachable. Also,
# it will try to connect 100 times before it discards messages as
# undeliverable.
# the rest below is more or less a plain vanilla rsyslog.conf as
# many distros ship it - it's more for your reference...
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.logThings to think about
You need to select the protocol best suitable for your use case. If in doubt, TCP is a decent choice. This recipe uses TCP for that reason.
TCP forwarding is a build-in capability and always present. As such, no plugin needs to be loaded. The target can be specified by DNS name or IP address. Use IP addresses for most robust operations. If you use a DNS name and name resolution fails, forwarding may be disabled for some time. DNS resolution typically fails on the DNS server itself during system startup.
In this example, we forward to port 10514. We could as well remove the port=”…” parameter from the configuration, which would result in the default port being used. However, you need to specify the port address on the server in any case. So it is strongly advised to use an explicit port number to make sure that client and server configuration match each other (if they used different ports, the message transfer would not work.
Receiving Messages from a Remote System
This is a log-consolidation scenario. There exist at least two systems, a server and at least one client. The server is meant to gather log data from all the clients. Clients may (or may not) process and store messages locally. If they do doesn’t matter here. See recipe Sending Messages to a Remote Syslog Server for how to configure the clients.
Note that in this scenario, we just receive messages from remote machines but do not process them in any special way. Thus, messages from both the local and all remote systems show up in all log files that are written (as well, of course, in all other actions). While the log files contain the source, messages from all systems are intermixed. If you would like to record messages from remote systems to files different from the local system, please see recipe Storing Messages from a Remote System into a specific File for a potential solution.
This scenario provides samples for both UDP and TCP reception. There exist other choices (like RELP), but these are less frequently used. If in question what to use, check the rsyslog module reference and protocol documentation. Note that most devices send UDP messages by default. UDP is an unreliable transmission protocol, thus messages may get lost. TCP supports much more reliability, so if you can not accept message loss, you need to use TCP. Not all devices support TCP-based transports.
Things to think about
TCP and UDP recpetion are not build-in capabilities. You need to load the imtcp and/or imudp plugin in order to enable it. This needs to be done only once in rsyslog.conf. Do it right at the top. Also note that some distributions may package imtcp and/or imudp in separate packages. If so, you need to install them first.
Rsyslog versions prior to v3 had a command-line switch (-r/-t) to activate remote listening. This switch is still available by default and loads the required plugins and configures them with default parameters. However, that still requires the plugins are present on the system. It is recommended not to rely on compatibility mode but rather use proper configuration.
Note that the server port address specified in $InputTCPServerRun must match the port address that the clients send messages to.
Config Statements
# for TCP use: module(load="imtcp") # needs to be done just once input(type="imtcp" port="514") # for UDP use: module(load="imudp") # needs to be done just once input(type="imudp" port="514") # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log
How it works
Note that loading the plugins is not sufficient. You also need to activate the listeners. Note the subtle difference between the two startup commands. If you need to have listeners for multiple ports, you can define the startup commands more than once. If you need only TCP or only UDP, you can comment out the other part.
Using a different log Format for all Files
Rsyslog comes with a limited set of log file formats. These resemble the default format that people (and log analyzers) usually expect. However, for some reason or another, it may be required to change the log format. In this recipe, we define a new format and use it as the default format for all log files.
Config Statements
$template myFormat,"%rawmsg%\n"
$ActionFileDefaultTemplate myFormat
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
Things to think about
The template and ActionFileDefaultTemplate statements must be made at the top of the configuration file, before any of the files are specified.
How it works
The Template-statement defines the new format. It consist of fields to be written, potential modifications as well as literal text. In the sample config statement, “rawmsg” ist a property that contains the syslog message as it was received by rsyslogd (“received” from any source, for example a remote system or the local log socket). The string “\n” is a line feed (ASCII LF), a constant being added to the string. Usually, log line templates need to end with “\n”, because without that, all log records would be written into a single line. Note that there are many fields and options for these fields that you can specify. The system is very flexible. But getting into the detail of all of that is beyond the scope of this cookbook-style book. Please see the “property replacer” official documentation for more details.
The $ActionFileDefaultTemplate then makes the newly defined template the default for all file actions. This saves you from specifying it with any single action line. But otherwise, it is equivalent to
$template myFormat,"%rawmsg%\n"# The authpriv file has restricted access.
authpriv.* /var/log/secure;myFormat# Log all the mail messages in one place.
mail.* /var/log/maillog;myFormat# Log cron stuff
cron.* /var/log/cron;myFormat# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler;myFormat# Save boot messages also to boot.log
local7.* /var/log/boot.log;myFormatDiscarding unwanted messages
Often, there are some messages that you know you will never store in any log file. Even worse, these messages are sometimes very frequently emitted. There are various ways to get rid of those unwanted messages.
First of all, you need to identify them. Then look carfully and see what is special with these messages. A common case may be that they contain a specific text inside the message itself. If so, you can filter on that text and discard anything that matches. You need to be careful, though: if there are other messages matching this text, these other messages will also be discarded. So it is vital to make sure the text you use is actually unique.
In the sample below, let’s assume that you want to discard messages that contain either the text “user nagios” or “module-alsa-sink.c: ALSA woke us up to write new data to the device, but there was actually nothing to write”. The later is an actual sample from pulseaudio, which is known to spam syslog with an enourmous volume of these messages.
Config Statements
:msg, contains, "user nagios" ~ :msg, contains, "module-alsa-sink.c: ALSA woke us up to write new data to the device, but there was actually nothing to write" ~ # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log
Note that these are just two lines. The second to forth line are just broken for printing purposes. These two must be on a single line in an actual rsyslog.conf.
How it works
Note that the statements are placed on top of rsyslog.conf. This makes them being executed before any other action statement. So each message received will be checked against the two string and be discarded, if a match is found. Note that you can move the discard action to another place inside rsyslog.conf if you would like to write the messages to some files, but not to others. For example, this configuration:
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# do not log the following to other files
:msg, contains, "user nagios" ~
:msg, contains, "module-alsa-sink.c: ALSA woke us up to
write new data to the device, but there was actually
nothing to write" ~# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
logs all messages to /var/log/messages, even those that then shall be discarded.
Storing Messages from a Remote System into a specific File
This is a log-consolidation scenario. There exist at least two systems, a server and at least one client. The server is meant to gather log data from all the clients. Clients may (or may not) process and store messages locally. If they do, doesn’t matter here. See recipe Sending Messages to a Remote Syslog Server for how to configure the clients.
Messages from remote hosts in the 192.0.1.x network shall be written to one file and messages from remote hosts in the 192.0.2.x network shallbe written to another file.
Things to think about
TCP recpetion is not a build-in capability. You need to load the imtcp plugin in order to enable it. This needs to be done only once in rsyslog.conf. Do it right at the top.
Note that the server port address specified in $InputTCPServerRun must match the port address that the clients send messages to.
Config Statements
$ModLoad imtcp
$InputTCPServerRun 10514
# do this in FRONT of the local/regular rules
if $fromhost-ip startswith '192.0.1.' then /var/log/network1.log
& ~
if $fromhost-ip startswith '192.0.2.' then /var/log/network2.log
& ~
# local/regular rules, like
*.* /var/log/syslog.log
How it works
It is important that the rules processing the remote messages come before any rules to process local messages. The if’s above check if a message originates on the network in question and, if so, writes them to the appropriate log. The next line (“& ~”) is important: it tells rsyslog to stop processing the message after it was written to the log. As such, these messages will not reach the local part. Without that “& ~”, messages would also be written to the local files.
Also note that in the filter there is a dot after the last number in the IP address. This is important to get reliable filters. For example, both of the addresses “192.0.1.1” and “192.0.10.1” start with “192.0.1” but only one actually starts with “192.0.1.”!