The rocket-fast system for log processing

Sending Messages to a Remote Syslog Server

In this recipe, we forward messages from one system to another one. Typical use cases are:

  • the local system does not store any messages (e.g. has not sufficient space to do so)
  • there is a (e.g. legal) requirement to consolidate all logs on a single system
  • the server may run some advanced alerting rules, and needs to have a full picture or network activity to work well
  • you want to get the logs to a different system in a different security domain (to prevent attackers from hiding their tracks)
  • and many more …

In our case, we forward all messages to the remote system. Note that by applying different filters, you may only forward select entries to the remote system. Also note that you can include as many forwarding actions as you like. For example, if you need to have a backup central server, you can simply forward to both of them, using two different forwarding actions.

To learn how to configure the remote server, see recipe Receiving Messages from a Remote System.

Config Statements

# this is the simplest forwarding action:
*.* action(type="omfwd" target="" port="10514" protocol="tcp")
# it is equivalent to the following obsolete legacy format line:
*.* @@ # do NOT use this any longer!
# Note: if the remote system is unreachable, processing will
# block here and discard messages after a while

# so a better use is
*.*  action(type="omfwd" target="" port="10514" protocol="tcp"
            queue.type="linkedList" queue.size="10000")
# this will de-couple the sending from the other logging actions,
# and prevent delays when the remote system is not reachable. Also,
# it will try to connect 100 times before it discards messages as
# undeliverable.
# the rest below is more or less a plain vanilla rsyslog.conf as 
# many distros ship it - it's more for your reference...
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none      /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                    /var/log/secure
# Log all the mail messages in one place.
mail.*                                        /var/log/maillog
# Log cron stuff
cron.*                                        /var/log/cron
# Everybody gets emergency messages
*.emerg                                       :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                /var/log/spooler
# Save boot messages also to boot.log
local7.*                                      /var/log/boot.log

Things to think about

You need to select the protocol best suitable for your use case. If in doubt, TCP is a decent choice. This recipe uses TCP for that reason.

TCP forwarding is a build-in capability and always present. As such, no plugin needs to be loaded. The target can be specified by DNS name or IP address. Use IP addresses for most robust operations. If you use a DNS name and name resolution fails, forwarding may be disabled for some time. DNS resolution typically fails on the DNS server itself during system startup.

In this example, we forward to port 10514. We could as well remove the port="…" parameter from the configuration, which would result in the default port being used. However, you need to specify the port address on the server in any case. So it is strongly advised to use an explicit port number to make sure that client and server configuration match each other (if they used different ports, the message transfer would not work.

24 thoughts on “Sending Messages to a Remote Syslog Server

  1. I put something like following in my config to strip hostname and timegenerated before sending to remote host but failed:

    $template myFTemplate, "%msg%\n"

    if $msg contains ‘mod=http request’ then
    local6.* @@;myFTemplate

    It works in 5.x. Did i miss anything? Thanks.

  2. What if @IP is used instead of @IP:Port? Will the system use the default syslog port, normally 514, or will it not work until it gets updated to @IP:Port? Thanks.

  3. Hi.
    i need to send my syslogs to remote server udp5179 port but where i try

    *.* @remote_ip_adress:5179

    checkpoint firewall is stops posting.

    How can i fix it?

  4. I want to send only certain logs from /var/log/messages on client server to central rsys log server. So can I define any conditions to send only message like "UserAllowed" from client server to central rsys log server?.
    Is there any parameter like "grep" I can define in configuration file?

    • You need to apply filters to only send certain messages. Use imfile to poll /var/log/messages, then before forwarding to another server use a filter like this:

      if $msg contains "UserAllowed" then { action }

  5. Pingback: Настройка централизованного логирования с LogAnalyzer и Rsyslog « Домик Миа

  6. Pingback: Настройка централизованного логирования с LogAnalyzer и Rsyslog » CreativLabs

  7. Okay. Nevermind.
    I was completely off base. I wasn’t even in the same ball park. ;-D

    I finally got this working.
    It was my primitive monkey brain that was the problem.
    I removed the local?.none statements from the /var/log/messages instruction.
    I removed the CustomLog statements from httpd.conf, and changed my if statements in rsyslog.conf to:
    if $syslogfacility-text == ‘local5′ then @@10.6.x.x:5140
    & ~
    if $syslogfacility-text == ‘local6′ then @@10.6.x.x:5140
    & ~

    I now forward to central logs host without any httpd access_log or error_log entries going to /var/adm/messages
    Monkey chow has done what Starbucks could not: awakening some semblance of intellect within the savage beast.

  8. First, my thanks for this wonderful blog.
    The information you have presented on rsyslog facilities and capabilities is invaluable.

    Second, I recently configured our web servers to forward apache2 logs to a central logs host using rsyslog "local" facilities, and Apache2 "CustomLog" configuration entries using /usr/bin/logger statements to local5 (access_log), and local6 (error_log). FYI I am forwarding to Logstash with Redis and Elasticsearch, not to rsyslog on the remote machine.

    Everything was great, except that the apache access and error log entries were going to /var/log/messages as well. I did not want this and my search for a way to prevent this led me to this blog site.

    After some study of your recommendations for centralized logs hosting, I implemented the following rsyslog configuration changes:

    #### MODULES ####

    $ModLoad imfile # load the imfile input module
    $ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
    $ModLoad imklog # provides kernel logging support (previously done by rklogd)

    # Configurations for central logs hosting for LAMP servers
    # Watch /var/log/apache2/access_log
    $InputFileName /usr/local/apache2/logs/access_log
    $InputFileTag apache-access:
    $InputFileStateFile state-apache-access

    # Watch /var/log/apache2/error_log
    $InputFileName /usr/local/apache2/logs/error_log
    $InputFileTag apache-error:
    $InputFileStateFile state-apache-error

    #### RULES ####
    # Do not write apache logs to any file
    if $syslogfacility-text == ‘local5’ and $programname == ‘httpd’ then @@10.6.x.x:5140
    & ~
    if $syslogfacility-text == ‘local6’ and $programname == ‘httpd’ then @@10.6.x.x:5140
    & ~

    This did not stop the apache logs from being written to /var/log/messages.

    On a lark, I decided to add "httpd.none" to the directive for sending to /var/log/messages thusly:

    *.info;mail.none;authpriv.none;cron.none;httpd.none /var/log/messages

    This fixed it, and now rsyslog is forwarding the apache logs to my central logs host without locally storing the apache logs in /var/log/messages. Apache is writing to /usr/local/apache2/logs directory, where they belong locally, but that is as I wish, so it is all good now.

    Is there some reason why the configuration I have under "Do not write apache logs to any file" was not preventing rsyslog from writing them to messages?



  9. How to send dynamic files from apache logs to remote server? I have modsec logs in /opt/apache/logs/modsec/modsecnnnn.log
    where nnnn is random number generated via application

    I want to send those files to remote server, Can you help me with client & server config example.

  10. Pingback: vCenter Log Insight: configuration des sources de données - VMnerds blog

  11. Hi,

    I have created syslog server,Now i want to forward all the logs from syslog server to other SIEM tool,whenever syslog server gets the logs from clients.I don’t want to store logs locally in syslog server,Just i want to bypass it to the SIEM tool.

    Can you please help me out on this.

  12. If some one needs expert view on the topic of running a blog after that
    i propose him/her to pay a quick visit this website, Keep up the nice work.

  13. Hi All,

    How to send logs to a remote host to store logs in different files.The log files should be different not only for the host but also for the type of log .in short different logs for different types for a specific host .

    how do we send logs for apache and java based applications which are related to log4j.

    please let me know ,thanks in advance.

  14. Pingback: Linux » Blog Archive » RHCSA

  15. FYI, the "This recipe uses TCP for that reason." is the double @ in "*.* @@″

    To use UDP use only one single @. For eample: "*.* @″

  16. great blog.thanks for tell us how to configure the remote server.the Config Statements is really useful and TCP is TCP is a decent choice.
    thanks for sharing the article,through the article i learned a lot.

  17. the blog is very useful.I agree with the Method of configuring a remote server wha you said in the article.TCP really is a good agreement.TCP forwarding is a build-in capablity and always present.It is easy to use.thanks for sharing the article.

  18. Pingback: Storing Messages from a Remote System into a specific File rsyslog

  19. Pingback: Receiving Messages from a Remote System rsyslog

Comments are closed.