rsyslog

The rocket-fast system for log processing

Storing Messages from a Remote System into a specific File

This is a log-consolidation scenario. There exist at least two systems, a server and at least one client. The server is meant to gather log data from all the clients. Clients may (or may not) process and store messages locally. If they do, doesn’t matter here. See recipe Sending Messages to a Remote Syslog Server for how to configure the clients.

Messages from remote hosts in the 192.0.1.x network shall be written to one file and messages from remote hosts in the 192.0.2.x network shallbe written to another file.

Things to think about

TCP recpetion is not a build-in capability. You need to load the imtcp plugin in order to enable it. This needs to be done only once in rsyslog.conf. Do it right at the top.

Note that the server port address specified in $InputTCPServerRun must match the port address that the clients send messages to.

Config Statements

$ModLoad imtcp
$InputTCPServerRun 10514
# do this in FRONT of the local/regular rules
if $fromhost-ip startswith '192.0.1.' then /var/log/network1.log
& ~
if $fromhost-ip startswith '192.0.2.' then /var/log/network2.log
& ~
# local/regular rules, like
*.* /var/log/syslog.log

How it works

It is important that the rules processing the remote messages come before any rules to process local messages. The if’s above check if a message originates on the network in question and, if so, writes them to the appropriate log. The next line ("& ~") is important: it tells rsyslog to stop processing the message after it was written to the log. As such, these messages will not reach the local part. Without that "& ~", messages would also be written to the local files.

Also note that in the filter there is a dot after the last number in the IP address. This is important to get reliable filters. For example, both of the addresses "192.0.1.1″ and "192.0.10.1″ start with "192.0.1″ but only one actually starts with "192.0.1."!

30 thoughts on “Storing Messages from a Remote System into a specific File

  1. Pingback: Receiving Messages from a Remote System rsyslog

  2. Hi,

    How can one have the receiving rsyslog server store the incoming message into a file based on the incoming message’s IP address, without having to create a new "if $fromhost-ip startswith ’10.20.4.5.’ then /var/log/10.20.4.5.log" each time.

    If it were automatic, then the individual files would be created automatically. When a new system is added or deleted one does not have to modify the rsyslog.conf file (because everyone forgets). With about 750 servers, this starts to be a little tedious :(

    Cheers.s

  3. Hi sdds,

    You can achieve per host log file by using ‘template’ as follows:

    $template PerHostLog,"/var/log/%HOSTNAME%.log"
    if $fromhost-ip startswith ’192.0.1.’ then -?PerHostLog
    & ~

    Cheers.

  4. http://rsyslog.com/config-snippets/the-recipies/more-complex-scenarios/ shows a different method of routing remote syslog message to a separate file by binding the TCP and UDP ports to a ruleset. Which method is preferred? In my specific scenario, I want to log messages from a piece of test equipment to a separate file to keep them from contaminating my workstations legitimate log files. My workstation did not previously have TCP or UDP on, and won’t receive any "real" syslog message over the network.

  5. This syntax does not work with rsyslog 5.8.6 that ships with the widely used Debian Squeeze distrobution.

    # provides TCP syslog reception
    $ModLoad imtcp
    $InputTCPServerRun 514

    $template HostBasedLog,"/var/log/%HOSTNAME%.log"
    if $fromhost-ip isequal ’192.168.22.1′ then -?HostBasedLog
    & ~
    if $fromhost-ip isequal ’192.168.22.2′ then -?HostBasedLog
    & ~

    Causes syntax errors on those two lines "syntax error in expression"

    the last error occured in /etc/rsyslog.conf, line 26:"if $fromhost-ip isequal ’192.168.22.2′ then -?HostBasedLog"

  6. With Debian Squeeze (currently at 6.0.4) you get this rsyslog:
    ii rsyslog 4.6.4-2 enhanced multi-threaded syslogd

    Can that be the cause?

  7. Hi, is there any ways to strip the starting part of every line in the log file from a remote system.
    I configure it in my system, but every line in the logs has a heading part that goes: "Mar 7 10:22:20 hostname (service): " and that’s killing my log analyzers scripts. Does anyone knows how to remove that ?

    Thxsx in advance

  8. Pingback: Small syslog server « Simon Josefsson's blog

  9. This seems to be working the best for me

    $template PerHostLog,"/var/log/cisco/%HOSTNAME%.log"
    if $fromhost-ip != ’127.0.0.1′ then -?PerHostLog
    & ~

    I also tried the following, but

    $template PerHostLog,"/var/log/cisco/%HOSTNAME%.log"
    if $fromhost-ip != ’127.0.0.1′ then -?PerHostLog
    & ~

  10. While using the below configuration:

    $template RemoteHost,"/var/log/remote_host_logs/%HOSTNAME%.log"
    $RuleSet remote
    if $fromhost-ip != ’127.0.0.1′ then -?RemoteHost
    & ~

    $InputTCPServerBindRuleset remote

    I am running rsyslog both in TCP and UDP on different ports.
    I can get logs from the remote hosts/devices to their separate log-files, but somehow the log messages from the local system stopped getting generated. After adding the above lines I can not see any log generated for the local system in /var/log/syslog. Can any one please help?

  11. There is another syntax which avoids the use of "if" entirely:

    :HOSTNAME, isequal, "REMOTE_HOSTNAME" /var/log/REMOTE_HOSTNAME.log
    & ~

    Replace REMOTE_HOSTNAME with your remote hosts name…

    You can use any rsyslog property name, e.g. HOSTNAME, FROMHOST, etc.

    I think you also can combine this with the template approach.

  12. but if i want to redir like this ?

    :FROMHOST-IP, isequal, “10.0.0.10” | @10.251.115.20

    this works :)

    if $hostname startswith ‘hostname’ and $syslogseverity <= 3 then | @10.251.115.20

  13. Hi..I have a query about sorting logs, please help me:
    i have one server (192.168.1.10), 2 clients (192.168.1.20, 192.168.1.30).
    clients send their logs to server.
    supposing 192.168.1.20 belongs to groupA and 192.168.1.30 belongs to groupB, i would like to sort logs at server as:
    /var/log/groupA/192.168.1.20/*.*
    /var/log/groupA/192.168.1.30/*.*
    /var/log/local/192.168.1.10/*.*

    Now if i have a 3rd client, 192.168.1.40, belonging to groupA, the new logs would be stored as:
    /var/log/groupA/192.168.1.40/*.*

    how can i automate this process using templates? i tried it using this:
    $template groupa,"/var/log/groupA/%fromhost-ip%/%programname%.log"
    $template groupb,"/var/log/groupB/%fromhost-ip%/%programname%.log"
    $template local,"/var/log/local/%fromhost-ip%/%programname%.log"
    if ($fromhost-ip == ’192.168.1.20′ or $fromhost-ip == ’192.168.1.40′) then {
    *.* ?groupa}
    if $fromhost-ip == ’192.168.1.30′ then {
    *.* ?groupb
    }
    else {
    *.* ?local
    }

    But the problem is, I have ALL logs in ALL folders as copies, groupA’s logs are also in groupB and local folder and groupB’s in groupA’s and local and local in groupA and groupB folder.
    How can i keep it unique using if-else rule? what is the mistake here? kindly help please…
    Thanks in advance.

  14. Any respond to my query? I’d really appreciate some help!!

    My post was:
    [zayed says:
    July 29, 2013 at 5:14 pm

    Hi..I have a query about sorting logs, please help me:
    i have one server (192.168.1.10), 2 clients (192.168.1.20, 192.168.1.30).
    clients send their logs to server.
    supposing 192.168.1.20 belongs to groupA and 192.168.1.30 belongs to groupB, i would like to sort logs at server as:
    /var/log/groupA/192.168.1.20/*.*
    /var/log/groupA/192.168.1.30/*.*
    /var/log/local/192.168.1.10/*.*

    Now if i have a 3rd client, 192.168.1.40, belonging to groupA, the new logs would be stored as:
    /var/log/groupA/192.168.1.40/*.*

    how can i automate this process using templates? i tried it using this:
    $template groupa,”/var/log/groupA/%fromhost-ip%/%programname%.log”
    $template groupb,”/var/log/groupB/%fromhost-ip%/%programname%.log”
    $template local,”/var/log/local/%fromhost-ip%/%programname%.log”
    if ($fromhost-ip == ’192.168.1.20′ or $fromhost-ip == ’192.168.1.40′) then {
    *.* ?groupa}
    if $fromhost-ip == ’192.168.1.30′ then {
    *.* ?groupb
    }
    else {
    *.* ?local
    }

    But the problem is, I have ALL logs in ALL folders as copies, groupA’s logs are also in groupB and local folder and groupB’s in groupA’s and local and local in groupA and groupB folder.
    How can i keep it unique using if-else rule? what is the mistake here? kindly help please…
    Thanks in advance.]

  15. I have one rsyslog server and 3 client server. i can send client log to remote server. But all the client logs are located in a single folder. So how can i create 3 client directory in remote server and how to send their own logs to their own file. Am using ubuntu 12.04 server.

  16. I have setup rsyslog to create a separate file for each remote host and service on that host as per the following config.
    $template RemoteHostUser,"/var/log/%HOSTNAME%.%programname%.log"
    :fromhost-ip, !isequal, "127.0.0.1" ?RemoteHostUser

    This works well (using v4.6.4 with Debian Squeeze), but I am missing fromhost-ip: in the file contents.
    Whats the best way to add fromhost-ip: to the output using this template?

  17. Ok, I’ve solved my own problem, and will record here to help others.
    1) Add a custom template somewhere before the RemoteHostUser template
    # Create Custom Template
    $template MarksRemoteTemplate,"%syslogfacility-text%,%syslogseverity-text%,%timereported% %fromhost-ip%%msg%\n"
    and then add the name of this template to the end of the conditional rule statement above, like you would if you were using ;RSYSLOG_DebugFormat
    $template RemoteHostUser,"/var/log/%HOSTNAME%.%programname%.log"
    :fromhost-ip, !isequal, "127.0.0.1" ?RemoteHostUser;MarksRemoteTemplate

  18. $template MarksRemoteTemplate,"%syslogfacility-text%,%syslogseverity-text%,%timereported% %fromhost-ip%%msg%\n"
    $template RemoteHostUser,"/var/log/%HOSTNAME%.%programname%.log"
    :fromhost-ip, !isequal, "127.0.0.1″ ?RemoteHostUser;MarksRemoteTemplate

  19. hi Mark,

    I was using your suggested code lines.
    but it was giving "syntax error" in first line
    $template MarksRemoteTemplate,"%syslogfacility-text%,%syslogseverity-text%,%timereported% %fromhost-ip%%msg%\n"

    Can you please tell me what i am missing.
    I am using Ubuntu 10.04 LTS – the Lucid Lynx on a x86 machine and running syslog-ng.

  20. Hi,

    is it possible to filter on a source network which is not a /24 network?

    As an example: We have a small network 131.234.131.192/255.255.255.192. I want all messages from these hosts in one file. But I cannot filter with "$fromhost-ip startswith 131.234.131″ because that would be 131.234.131.192/255.255.255.0 which is obviously too large.

    Any ideas?

    Thanks,

    Christopher

  21. Hello,

    I have a cisco network with almost 50 devices, is there any chance to configure rsyslog con put each host in one specific file?

    Thanks

  22. Sorry to bother again I have this config

    if $fromhost-ip startswith ’10.16.200.14′ then /var/log/cisco/r02.log

    !
    if $fromhost-ip startswith ’10.16.200.3′ then /var/log/cisco/r03.log

    but when i go to the cisco file I only see the r02.log

    any ideas?
    thanks

Leave a comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>