template

LogAnalyzer: Facility and Severity is missing

Question: I use a logfile of rsyslog as source type in LogAnalyzer, everything is good but the facility and severity information tabs of the messages are missing, like in this screenshot.

Answer: The solution is rather simple, your current file template does not contain syslog priority or facility. Kindly switch to RSYSLOG_SyslogProtocol23Format which is RFC5424 format which contains the required information. You can use the template for a single action or you can use it as the default template. Below you can find a example for both cases:
Please note that this example only apply to a single logfile:

mail.* /var/log/maillog;RSYSLOG_SyslogProtocol23Format

This is the example for a default template:

$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format

Please note that you need to change the logfile type to RSyslog Format23 in your Loganalyzer logstream sources as well. You can do that by editing the “config.php” of LogAnalyzer.
Open the “config.php” with your favourite editor and add the following line to the correct source:

$CFG[‘Sources’][‘Source1’][‘LogLineType’] = “Syslog23”;

Afterwards the configuration should look like this.
Don’t forget to save the changes. Now you can refresh the Webpanel of LogAnalyzer and then you should see the facility and severity missing information tabs, like in this screenshot.

rsyslog 8.3.2 (v8-devel) released

We have just released 8.3.2 of the v8-devel branch.

This is primarily a bug-fixing release, but it also adds the ability to extract parts of a timestamp via the property replacer and adds support for per-listener imrelp rulesets (thanks to bobthesecurityguy for the patch!).

ChangeLog:

http://www.rsyslog.com/changelog-for-8-3-2-v8-devel/

Download:

http://www.rsyslog.com/download-v8-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Filter optimization with arrays

If you are using a lot of filters and templates in rsyslog, this can not only be affecting the performance drastically, but it is also a hassle to set up all the different actions and templates. It is always worthy to check, if there isn’t a shortcut somewhere, which might not only save you time for creating the configuration, but also make it much simpler in the end to keep track of all the actions.

In our example, we have several programnames. The log messages should be sorted by programname and then be stored in a specific file and be sorted by host. After storing the log messages, the message should be discarded, so it won’t be processed by the following filters, thus saving otherwise wasted processing time. This example is applicable to rsyslog v7.2.x and above.

Here are some sample config lines.

## apache_access
template(name="DailyPerHost_apache_access" type="string" string="/syslog/%FROMHOST%/apache_access.log")
:programname, isequal, "apache_access" { action(type="omfile" DynaFile="DailyPerHost_apache_access") stop }

## apache_error
template(name="DailyPerHost_apache_error" type="string" string="/syslog/%FROMHOST%/apache_error.log")
:programname, isequal, "apache_error" { action(type="omfile" DynaFile="DailyPerHost_apache_error") stop }
## mysql
template(name="DailyPerHost_mysql" type="string" string="/syslog/%FROMHOST%/mysql.log")
:programname, isequal, "mysql" { action(type="omfile" DynaFile="DailyPerHost_mysql") stop }

## php
template(name="DailyPerHost_php" type="string" string="/syslog/%FROMHOST%/php.log")
:programname, isequal, "php" { action(type="omfile" DynaFile="DailyPerHost_php") stop }

These are some basic services, which are often run together. Please note, that these are just a few examples. As you can see here, the template is created first. It is given a name, type and format. Templates of type string are usually used for file names. Here the log messages get stored in the folder /syslog a subfolder for the host where the message occured and then a filename which reflects the type of message that occured.

The second line holds the actions. First you see the property based filter (programname) and the condition. After that the actions get chained with the curly braces. The log messages where the filter evaluates to true get stored in a file. The filename and path is generated dynamically with the DynaFile parameter. Through this, the above written template will be used to generate the path and filename. The second action is represented by stop. Please note that this is case sensitive. Basically, stop means to stop the message processing. No further processing of the message will take place.

If we look closely at the sample config lines, we see, that the filter condition is basically always the same. It will always filter the programname property for a certain value. This is a predestinated case for using an array for simplification. We can use the property programname in the file template as well and filter an array of values. This will greatly save the overhead for all the seperate filter, not only in the configuration, but also in processing the messages.

template(name="DailyPerHost_app" type="string" string="/syslog/%FROMHOST%/%programname%.log")
if $programname == ["apache_access",
                    "apache_error",
                    "mysql",
                    "php"]
                then {
                action(type="omfile" DynaFile="DailyPerHost_app")
                stop
                }

Again, we first create the template. Please note the difference in the filename where the hardcoded text has been replaced by the property programname. In the next lines, we see the filter and the array of values. This is just to reflect the example. Virtually, the array can have near-infinite values. The filter is also a common if/then construct. After the then we see our chain of commands. First the action which writes the log messages into a file where the filename is created by the above template and then a stop as second action.

This case is applicable in many forms. It is also most useful if you are filtering and the discarding a lot of messages with very common filter settings. You could use it to filter for an array of property values and even chain comparison operations.

rsyslog 6.5.0 (v6-devel) released

This is an important new release of the rsyslog v6 devel branch. This release contains bugfixes, as well as new features. The most important one might be the possibility to use the config format for templates.. Please note that part of the feature set is still experimental and might be unstable. ;)

ChangeLog:

http://www.rsyslog.com/changelog-for-6-5-0-v6-devel/

Download:

http://www.rsyslog.com/rsyslog-6-5-0-v6-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Changelog for 6.5.0 (v6-devel)

Version 6.5.0  [devel] 2012-08-28

  • imrelp now supports non-cancel thread termination
    (but now requires at least librelp 1.0.1)
  • implemented freeCnf() module interface
    This was actually not present in older versions, even though some modules already used it. The implementation was now done, and not in 6.3/6.4 because the resulting memory leak was ultra-slim and the new interface handling has some potential to seriously break things. Not the kind of thing you want to add in late beta state, if avoidable.
  • added –enable-debugless configure option for very high demanding envs
    This actually at compile time disables a lot of debug code, resulting in some speedup (but serious loss of debugging capabilities)
  • added new 0mq plugins (via czmq lib)
    Thanks to David Kelly for contributing these modules
  • bugfix: omhdfs did no longer compile
  • bugfix: SystemLogSocketAnnotate did not work correctly
    Thanks to Miloslav Trmač for the patch
  • $SystemLogParseTrusted config file option
    Thanks to Milan Bartos for the patch
  • added template config directive
  • added new uuid message property
    Thanks to Jérôme Renard for the idea and patches.
    Note: patches were released under ASL 2.0, see
    http://bugzilla.adiscon.com/show_bug.cgi?id=353

How to bind a template

This little FAQ describe how to bind a template.
First with the new template format “list” and then with the old “legacy” format.

First off all you have to define a template for example for specify output.

Here is an example template in the list format:

template(name=”FileFormat” type=”list”) {
property(name=”timestamp” dateFormat=”rfc3339″)
constant(value=” “)
property(name=”hostname”)
constant(value=” “)
property(name=”syslogtag”)
constant(value=” “)
property(name=”msg” spifno1stsp=”on” )
property(name=”msg” droplastlf=”on” )
constant(value=”\n”)
}

Then you have to bind the template to an action. You do this via the “template” parameter.

action(… template=”name-of-template”)

Here is an example action with a example-template:

*.* action(type=”omfile” file=”/var/log/all-messages.log” template=”Name-of-your-template”)

In the configuration it should looks like this:

template(name=”FileFormat” type=”list”) {
property(name=”timestamp” dateFormat=”rfc3339″)
constant(value=” “)
property(name=”hostname”)
constant(value=” “)
property(name=”syslogtag”)
constant(value=” “)
property(name=”msg” spifno1stsp=”on” )
property(name=”msg” droplastlf=”on” )
constant(value=”\n”)
}
action(type=”omfile” file=”/var/log/all-msgs.log” template=”FileFormat”)

Here is an example for obsolete legacy format

Here is an example template in the legacy format:

$template ExampleFormat,”%timereported:::date-rfc3339% %HOSTNAME% %msg%”

Here is an example action with a example-template:

*.* /var/log/all-messages.log;Your-Template-Name

In the Configuration it looks like this:

$template ExampleFormat,”%timereported:::date-rfc3339% %HOSTNAME% %msg%”
*.* /var/log/all-messages.log;ExampleFormat

Note that it is advised NOT to use obsolete legacy format for anything complex. Assigning templates is at the border of being not recommended.

Storing and forwarding remote messages

In this scenario, we want to store remote sent messages into a specific local file and forward the received messages to another syslog server. Local messages should still be locally stored.

Things to think about

How should this work out? Basically, we need a syslog listener for TCP and one for UDP, the local logging service and two rulesets, one for the local logging and one for the remote logging.

TCP recpetion is not a build-in capability. You need to load the imtcp plugin in order to enable it. This needs to be done only once in rsyslog.conf. Do it right at the top.

Note that the server port address specified in $InputTCPServerRun must match the port address that the clients send messages to.

Config Statements

# Modules
$ModLoad imtcp
$ModLoad imudp
$ModLoad imuxsock
$ModLoad imklog
# Templates
# log every host in its own directory
$template RemoteHost,"/var/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"
### Rulesets
# Local Logging
$RuleSet local
kern.*                                                 /var/log/messages
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
# use the local RuleSet as default if not specified otherwise
$DefaultRuleset local

# Remote Logging
$RuleSet remote
*.* ?RemoteHost
# Send messages we receive to Gremlin
*.* @@W.X.Y.Z:514
### Listeners
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 10514

$InputUDPServerBindRuleset remote
$UDPServerRun 514

How it works

The configuration basically works in 4 parts. First, we load all the modules (imtcp, imudp, imuxsock, imklog). Then we specify the templates for creating files. The we create the rulesets which we can use for the different receivers. And last we set the listeners.

The rulesets are somewhat interesting to look at. The ruleset “local” will be set as the default ruleset. That means, that it will be used by any listener if it is not specified otherwise. Further, this ruleset uses the default log paths vor various facilities and severities.

The ruleset “remote” on the other hand takes care of the local logging and forwarding of all log messages that are received either via UDP or TCP. First, all the messages will be stored in a local file. The filename will be generated with the help of the template at the beginning of our configuration (in our example a rather complex folder structure will be used). After logging into the file, all the messages will be forwarded to another syslog server via TCP.

In the last part of the configuration we set the syslog listeners. We first bind the listener to the ruleset “remote”, then we give it the directive to run the listener with the port to use. In our case we use 10514 for TCP and 514 for UDP.

Important

There are some tricks in this configuration. Since we are actively using the rulesets, we must specify those rulesets before being able to bind them to a listener. That means, the order in the configuration is somewhat different than usual. Usually we would put the listener commands on top of the configuration right after the modules. Now we need to specify the rulesets first, then set the listeners (including the bind command). This is due to the current configuration design of rsyslog. To bind a listener to a ruleset, the ruleset object must at least be present before the listener is created. And that is why we need this kind of order for our configuration.

Scroll to top