The rocket-fast system for log processing

Storing and forwarding remote messages

In this scenario, we want to store remote sent messages into a specific local file and forward the received messages to another syslog server. Local messages should still be locally stored.

Things to think about

How should this work out? Basically, we need a syslog listener for TCP and one for UDP, the local logging service and two rulesets, one for the local logging and one for the remote logging.

TCP recpetion is not a build-in capability. You need to load the imtcp plugin in order to enable it. This needs to be done only once in rsyslog.conf. Do it right at the top.

Note that the server port address specified in $InputTCPServerRun must match the port address that the clients send messages to.

Config Statements

# Modules
$ModLoad imtcp
$ModLoad imudp
$ModLoad imuxsock
$ModLoad imklog
# Templates
# log every host in its own directory
$template RemoteHost,"/var/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"
### Rulesets
# Local Logging
$RuleSet local
kern.*                                                 /var/log/messages
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
# use the local RuleSet as default if not specified otherwise
$DefaultRuleset local

# Remote Logging
$RuleSet remote
*.* ?RemoteHost
# Send messages we receive to Gremlin
*.* @@W.X.Y.Z:514
### Listeners
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 10514

$InputUDPServerBindRuleset remote
$UDPServerRun 514

How it works

The configuration basically works in 4 parts. First, we load all the modules (imtcp, imudp, imuxsock, imklog). Then we specify the templates for creating files. The we create the rulesets which we can use for the different receivers. And last we set the listeners.

The rulesets are somewhat interesting to look at. The ruleset "local" will be set as the default ruleset. That means, that it will be used by any listener if it is not specified otherwise. Further, this ruleset uses the default log paths vor various facilities and severities.

The ruleset "remote" on the other hand takes care of the local logging and forwarding of all log messages that are received either via UDP or TCP. First, all the messages will be stored in a local file. The filename will be generated with the help of the template at the beginning of our configuration (in our example a rather complex folder structure will be used). After logging into the file, all the messages will be forwarded to another syslog server via TCP.

In the last part of the configuration we set the syslog listeners. We first bind the listener to the ruleset "remote", then we give it the directive to run the listener with the port to use. In our case we use 10514 for TCP and 514 for UDP.


There are some tricks in this configuration. Since we are actively using the rulesets, we must specify those rulesets before being able to bind them to a listener. That means, the order in the configuration is somewhat different than usual. Usually we would put the listener commands on top of the configuration right after the modules. Now we need to specify the rulesets first, then set the listeners (including the bind command). This is due to the current configuration design of rsyslog. To bind a listener to a ruleset, the ruleset object must at least be present before the listener is created. And that is why we need this kind of order for our configuration.

10 thoughts on “Storing and forwarding remote messages

  1. thank you all for your help in advance, my problem is as follows, once I configure my rsyslog nothing goes in my /var/logs/ any more nothing in message, cron, secure …etc…

    below is my rsyslog.conf.

    # Modules
    $ModLoad imtcp
    $ModLoad imudp
    $ModLoad imuxsock
    $ModLoad imklog
    # Templates

    $template FILENAME,"/logs/syslogs/%HOSTNAME%/logs-%$YEAR%-%$MONTH%-%$DAY%-%$HOUR%.log"
    if ($hostname != ‘MyServerName’) then -?FILENAME
    & ~

    # Where to place auxiliary files
    $WorkDirectory /var/lib/rsyslog

    # Use default timestamp format
    $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

    # File syncing capability is disabled by default. This feature is usually not required,
    # not useful and an extreme performance hit
    #$ActionFileEnableSync on

    # Include all config files in /etc/rsyslog.d/
    $IncludeConfig /etc/rsyslog.d/*.conf

    # Turn off message reception via local log socket;
    # local messages are retrieved through imjournal now.
    #$OmitLocalLogging on

    # File to store the position in the journal
    $IMJournalStateFile imjournal.state

    $RuleSet local

    #### RULES ####
    #kern.* /dev/console
    *.info;mail.none;authpriv.none;cron.none /var/log/messages
    authpriv.* /var/log/secure
    mail.* -/var/log/maillog
    cron.* /var/log/cron
    *.emerg :omusrmsg:*
    local7.* /var/log/boot.log

    any ideas what am I missing?
    thank you again.

  2. hi all,
    i’m a newbie with Rsyslog and i need help.
    I receive on my server, some logs with $programme == ‘radiusN2′. I would like to redirect logs who go in "/var/log/syslog" and with $programme = ‘radiusN2′ and concerning user, toward "/var/log/radiusN2/radiusN2.log"..

    I tested this code :
    if $programme == ‘radiusN2’ then {

    user.* -/var/log/radiusN2/radiusN2.log


    & ~

    but he didn’t work, his errors appears : Oct 8 11:23:17 dorad rsyslogd: the last error occured in /etc/rsyslog.d/radiusN2.conf, line 1:"if $programme == ‘radiusN2′ then {"
    Oct 8 11:23:17 dorad rsyslogd: warning: selector line without actions will be discarded
    Oct 8 11:23:17 dorad rsyslogd-3000: unknown priority name "" [try ]
    Oct 8 11:23:17 dorad rsyslogd: the last error occured in /etc/rsyslog.d/radiusN2.conf, line 3:"}"

    Thank in advance

  3. This is currently what I have in my rsyslog.conf

    local2.* @@localhost:5140

    It is forwarding the messages over tcp fine, and the consumer service is able to collect the log from the above host and port configuration.

    But I can’t figure out how to provide the configuration so that when the consumer service dies, the logs are written to disk and then sent again when it comes alive. Currently when I shut down the consumer service and then switch it back on, it loses all the messages that arrived while the consumer was down.

    I see the below configuration at the end of rsyslog.conf but can’t figure how to map this to my configuration above

    #$WorkDirectory /var/lib/rsyslog # where to place spool files
    #$ActionQueueFileName fwdRule1 # unique name prefix for spool files
    #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
    #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
    #$ActionQueueType LinkedList # run asynchronously
    #$ActionResumeRetryCount -1 # infinite retries if host is down
    # remote host is: name/ip:port, e.g., port optional
    #*.* @@remote-host:514

    Any help would be much appreciated.

  4. Is it possible to use a $FileCreateMode directive to influence the mask of the templated file name? It seems like when I do this (v5.8.6) it does not work.

  5. While using the below configuration:

    $template RemoteHost,”/var/log/remote_host_logs/%HOSTNAME%.log”
    $RuleSet remote
    if $fromhost-ip != ’′ then -?RemoteHost
    & ~

    $InputTCPServerBindRuleset remote

    I am running rsyslog both in TCP and UDP on different ports.
    I can get logs from the remote hosts/devices to their separate log-files, but somehow the log messages from the local system stopped getting generated. After adding the above lines I can not see any log generated for the local system in /var/log/syslog. Can any one please help?

  6. Hi,
    How can I get rsyslog messages containing year along with the timestamp in UTC, evenif my machine’s TZ is NOT UTC ???
    How should I define the template for this format?

    Thanks in advance for your help.

  7. Hi

    I have a question and could not find it in docu.
    What does the "-" sign mean before the file name?
    For example:

    Thx for help.

Comments are closed.