Basic Configuration

Newbie guide to rsyslog

By Adiscon SupportPosted on May 24, 2013Posted in Basic Configuration, Guides for rsyslogTagged basic configuration, if filter, installation, libee, libestr, packages, rsyslog, ruleset

Written by Jan Gerhards

Here are some different guides for people, who never worked with rsyslog. First I’ll explain, how to install rsyslog. Besides, I will explain how to install some packages, which you will need to install rsyslog. There are two important ones and some minor ones. After that, I will show how to do some easy configurations. Questions like how can I configure a module? Or How can I configure the input? Will be answered. In the example I will configure rsyslog to receive messages, filter them for one specific IP and store only the messages in a file. In the end I’ll explain something about easy rulesets. In my example I will try the same like I did with the configuration, but I will work with rulesets.

Before you begin

Please note, there are many ways to distribute rsyslog. But, make sure that the platform you build rsyslog on is the same platform as where it should be used. You cannot build rsyslog on CentOS and use it on Ubuntu. The differences between the platforms are just to big. The same applies to different versions of the same platform. When building on a older platform and using it on a newer version, this may work, but with restrictions. Whereas building on a newer version and using it on a older version will probably not work at all. So, if you build rsyslog from Source and want to use it on another machine, make sure the platform is the same.

Preliminary actions

There are generally two very often needed packages, that you have to install before rsyslog: libestr and libee. These can be downloaded here:

Libestr: http://libestr.adiscon.com/download/

Libee: http://www.libee.org/download/

Both are being installed the same, so here is only one step-by-step instruction.

Download the file
Just download libestr or libee and “cd” into the folder, where you want them.
“tar xzf” the file
after you “cd” to the folder, where the file is, type “tar xzf -name of file”. Your command line should look like this:
```
 tar xzf -name of file-
```
“Cd” into the new folder
“cd” into the made directory. It will have the same name like the downloaded file. Your command line should look like this:
```
 cd -name of file-
```
Run “./configure configure –libdir=/usr/lib –includedir=/usr/include”
After you “cd” into that file, just run ” ./configure –libdir=/usr/lib –includedir=/usr/include”. Your command line should look like this:
```
 ./configure --libdir=/usr/lib --includedir=/usr/include
```
Type “sudo make”
Just type this and let the Computer work. Your command line should look like this:
```
 sudo make
```
Type “sudo make install”
This is the last step. Like with “sudo make”, just let the computer work. Your command line should look like this
```
 sudo make install
```
The package should work now
Congratulation! You have installed the package! If it doesn’t work properly, you should check if you followed every step right and if you have, contact the support.

There might also be some other packages like libjson0-dev, uuid-dev, bison, flex, libz-dev or python-docutils. These you’ll have to install before installing rsyslog, too. This guide was made on Ubuntu, so if you use any other system, they might be named differently. You can install them with the command

sudo apt-get install -package name-

After you got all the packages, you can install rsyslog. Here is a How to for installing rsyslog.

How to install rsyslog

In this guide is explained how to install rsyslog with the download from the homepage.

Download rsyslog
You can download rsyslog from the rsyslog webpage at http://www.rsyslog.com/download/
“tar xzf” the file
Open a terminal. Then you “cd” to where you want the file and then type “tar xzf -name of downloaded file”. Your command line should look like this:
```
tar xzf -name of downloaded file-
```
“cd” into the new folder
Then “cd” into the made directory. The new directory will have the same name like the downloaded file. Your command line should look like this:
```
 cd -name of file-
```
Type “./configure –prefix=/usr”
You just need to run “./configure –prefix=/usr”. There might be an Error that you need to install some packages to precede with installing. For two specific packages I’ve explained how to install them, because you will need them often. Your command line should look like this:
```
./configure --prefix=/usr
```
Run “sudo make”
it’s easy, just run “sudo make” and let the computer work. Your command line should look like this:
```
 sudo make
```
Run “sudo make install”
Just like “sudo make”. Type it, press enter and let the Computer work. Again, there might be some missing packages. Just install them. Your command line should look like this:
```
 sudo make install
```
Rsyslog should now be installed
Congratulation! You have installed rsyslog! You can check the version of rsyslog by typing: “rsyslogd -v”.
If it doesn’t work properly, you should check if you followed every step right and if you have, contact the support.

Now after you installed rsyslog, you have to configure it. How to do this is explained here.

Configure rsyslog

In this part I’ll explain some basic configuration steps for rsyslog. We configure rsyslog to recive UDP messages, to filter them depending on the IP of the host and to store them in a file.

How to configure the module
The module has to be configured first. The general line for this configuration is: “module (load=”im-type of protocol-”). So in our example, where we want UDP, it will look like this:
```
Module (load=”imudp”)
```
How to configure the input for rsyslog
For the input, you have to give two different information to rsyslog. The first information needed is the protocol type of the input; in my example again UDP. Like in the first line there is an “im-” in front of the protocol-type. The other information is to configure a port for rsyslog, in my example 514. These two information are together in only one line. The line is: “Input (type=”-protocol of input-“port=”-number of port-“). This means for my example, the line has to be
```
Input (type=”imudp” port=”514”)
```
How to configure a filter for fromhost-IPs and store them in a file
A filter always has, like a normal conditional sentence, an “if…then” part. If you want to configure it to do something with all notes from a specific IP, between “if” and “then” will be the property “$fromhost-ip ==”-IP, you want to filter-”. After this stays a “then” and after the “then” follows an action in brackets, which I will explain later. In my example I want only the notes from the host with the IP 172.19.1.135. So the line will be
```
If $fromhost-ip == “172.19.1.135” then {
```
After this we have to tell the computer, what to do if that case is given. In this example we want him to store these messages in the file “/var/log/network1.log”. This is an action with the type “omfile”. To configure the file where to store the messages, the action is “action (type=”omfile” File=”-filename-“). So in this example, it will look like this:
```
Action (type=”omfile” file=”/var/log/network1.log”)
}
```

All the lines together now are

Module (load=“imudp“)

Input (type=”imudp” port=”514”)

If $fromhost-ip == “172.19.1.135“ then {
Action (type=”omfile” File=”/var/log/network1.log”)
}

All in all it means: The input for rsyslog will listen to syslog via UDP on port 514. If the IP from the Computer, which sends the messages, is 172.19.1.135, then the action in the brackets will get activated for these. In the action the messages will be stored in the file /var/log/network1.log.

Rsyslog and rulesets

Rulesets are a bit more complicated. A ruleset is a set of rules, as the name implies. These are bound to an input. This works by adding an option to the input, namely “ruleset=”-rulesetname-“”. For example, if I want to bind a ruleset “rs1” to a input the line will look like this:

Input (type=”imudp” port=”514” ruleset=”rs1”)

But you still have to define, what the ruleset should do. In this guide I will limit myself to explain, how to create a ruleset, which has one action: to store all the messages in a file. In my example I want to store the messages in the file /var/log/network1.log”.

You define a ruleset like the normal configuration. To define it, you first name it with ruleset (name=”-rulesetname-“). After this you write what it does, in my example the action action (type=”omfile” file=”/var/log/network1.log”). This action you write in these curly brackets: {}.

So my full example looks like this

Module (load=”imudp”)

Input (type=”imudp” port=”514” ruleset=”rs1”)

Ruleset (name=”rs1”) {
Action (type=”omfile” file=”/var/log/network1.log”)
}

In that second example for configurations you can see, how to store all messages from the input into a file by using a ruleset. A rulesset can consist of multiple rules, but without binding it to the input it is useless. It can be bound to an input multiple times or even other rulesets can be called.

Final Conclusion

In this guide I explained how to install rsyslog, how to configure it and how to use rulesets. After you read this guide you are able to do exactly this: you can install rsyslog, configure it and have basic knowlege about rulesets. If you want to learn more about rsyslog, how to configure it or about rulesets, you can find information in the other guides or in the documentation.

Installing rsyslog from RPM

By Adiscon SupportPosted on April 23, 2013Posted in Basic Configuration, Guides for rsyslogTagged CentOS, repo, repository, RHEL, RPM, rsyslog

We want to describe here, how to install rsyslog from the RPM repository on RHEL/CentOS 5 and 6. Currently, we have RPMs available for the most recent versions of rsyslog.

To use the repository, please follow these steps.

1. Obtain the .repo file

To be able to use the RPM repository, you need a .repo file. Using your webbrowser, go to http://rpms.adiscon.com. Here, either download the rsyslogall.repo file or go to the subfolder for the desired version (e.g. v7-stable) and download the rsyslog.repo file from there.

2. Placing the file in the correct location

You now need to place the downloaded file in the correct folder. The .repo file must be placed in

/etc/yum.repos.d/

3. Finding updates

You should now do a “yum update”. The update for rsyslog will be detected automatically. Note, that the file rsyslogall.repo contains the repository configuration for each available branch, whereas the rsyslog.repo only provides repository configuration for the selected branch.

A note on signed RPM’s

On RHEL/CentOS 5 it is not possible to check for signed RPM’s. This is due to a bug in the RPM/YUM subsystem. So, Please DO NOT USE gpgcheck=1! If you however couldn’t resist trying it, and your system stucks with “Header V3 RSA/SHA1 signature: BAD, key ID e00b8985″, follow these steps:

Set gpgcheck in your rsyslog.repo file back to 0.
Run this command to remove the gpg key from your system: rpm –erase gpg-pubkey-e00b8985-512dde96

Sending messages with tags larger than 32 characters

By Rainer GerhardsPosted on October 21, 2011Posted in Basic Configuration, Guides for rsyslog, The recipiesTagged config snippet, interoperability, rsyslog, syslog, tag

The relevant syslog RFCs 3164 and 5424 limit the syslog tag to 32 characters max. Messages with larger tag length are malformed and may be discarded by receivers. Anyhow, some folks sometimes need to send tags longer than permitted.

To do so, a new template must be created and used when sending. The simplest way is to start with the standard forwarding template. The standard templates are hardcoded inside rsyslog. Thus they do not show up in your configuration file (but you can obtain them from the source, of course). In 5.8.6, the forwarding template is defined as follows:

template (name="ForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
%syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")

NOTE: all templates are on one line in rsyslog.conf. They are broken here for readability.

This template is RFC-compliant. Now look at the part in red. It specifies the tag. Note that, via the property replacer, it is restricted to 32 characters (from position 1 to position 32 inclusive). This is what you need to change. To remove the limit … just remove it ;-) This leads to a template like this:

template (name="LongTagForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")

Note that I have renamed the template in order to avoid conflicts with build-in templates. As it is a custom template, it is not hardcoded, so you need to actually configure it in your rsyslog.conf. Then, you need to use that template if you want to send messages to a remote host. This can be done via the usual way. Let’s assume you use legacy plain TCP syslog. Then the line looks as follows:

action(type="omfwd" 
Target="server.example.net"
Port="10514"
Protocol="tcp"
Template="LongTagForwardFormat"
)

This will bind the forwarding action to the newly defined template. Now tags of any size will be forwarded. Please keep in mind that receivers may have problems with large tags and may truncate them or drop the whole message. So check twice that the receiver handles long tags well.

Rsyslog supports tags to a build-defined maximum. The current (5.8.6) default is 511 characters, but this may be different if you install from a package, use a newer version of rsyslog or use sources obtained from someone else. So double-check.

Using the syslog receiver module

By Adiscon SupportPosted on March 29, 2011Posted in Basic ConfigurationTagged config snippet, rsyslog, syslog, TCP, UDP

We want to use rsyslog in its general purpose. We want to receive syslog. In rsyslog, we have two possibilities to achieve that.

Things to think about

First of all, we will determine, which way of syslog reception we want to use. We can receive syslog via UDP or TCP. The config statements are each a bit different in both cases.

In most cases, UDP syslog should be fully sufficient and performing well. But, you should be aware, that on large message bursts messages can be dropped. That is not the case with TCP syslog, since the sender and receiver communicate about the arrival of network packets. That makes TCP syslog more suitable for environments where log messages may not be lost or that must ensure PCI compliance (like banks).

Config Statements

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

and

module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")

How it works

The configuration part for the syslog server is pretty simple basically. Though, there are some parameters that can be set for both modules. But this is not necessary in most cases.

In general, first the module needs to be loaded. This is done via the directive module().

module(load="imudp / imtcp")

The module must be loaded, because the directives and functions rely on it. Just by using the right commands, rsyslog will not know where to get the functionality code from.

And next is the command that runs the syslog server itself is called input().

input(type="imudp" port="514")
input(type="imtcp" port="514")

Basically, the command says to run a syslog server on a specific port. Depending on the command, you can easily determine the UDP and the TCP server.

You can of course use both types of syslog server at the same time, too. You just need to load both modules for that and configure the server command to listen to specific ports. Then you can receive both UDP and TCP syslog.

Important

In general, we suggest to use TCP syslog. It is way more reliable than UDP syslog and still pretty fast. The main reason is, that UDP might suffer of message loss. This happens when the syslog server must receive large bursts of messages. If the system buffer for UDP is full, all other messages will be dropped. With TCP, this will not happen. But sometimes it might be good to have a UDP server configured as well. That is, because some devices (like routers) are not able to send TCP syslog by design. In that case, you would need both syslog server types to have everything covered. If you need both syslog server types configured, please make sure they run on proper ports. By default UDP syslog is received on port 514. TCP syslog needs a different port because often the RPC service is using this port as well.

Using the Text File Input Module

By Adiscon SupportPosted on March 25, 2011Posted in Basic ConfigurationTagged config snippet, Guides for rsyslog, imfile, rsyslog, text file input module

Log files should be processed by rsyslog. Here is some information on how the file monitor works. This will only describe setting up the Text File Input Module. Further configuration like processing rules or output methods will not be described.

Things to think about

The configuration given here should be placed on top of the rsyslog.conf file.

Config Statements

module(load="imfile" PollingInterval="10")
# needs to be done just once. PollingInterval is a module directive and is only set once when loading the module

# File 1
input(type="imfile" File="/path/to/file1" 
Tag="tag1" 
StateFile="/var/spool/rsyslog/statefile1" 
Severity="error" 
Facility="local7")
# File 2
input(type="imfile" File="/path/to/file2" 
Tag="tag2" 
StateFile="/var/spool/rsyslog/statefile2")
# ... and so on ...
#

How it works

The configuration for using the Text File Input Module is very extensive. At the beginning of your rsyslog configuration file, you always load the modules. There you need to load the module for Text File Input as well. Like all other modules, this has to be made just once. Please note that the directive PollingInterval is a module directive which needs to be set when loading the module.

module(load="imfile" PollingInterval="10")

Next up comes the input and its parameters. We configure a input of a certain type and then set the parameters to be used by this input. This is basically the same principle for all inputs:

# File 1
input(type="imfile" File="/path/to/file1" 
Tag="tag1" 
StateFile="/var/spool/rsyslog/statefile1" 
Severity="error" 
Facility="local7")

File specifies, the path and name of the text file that should be monitored. The file name must be absolute.

Tag will set a tag in front of each message pulled from the file. If you want a colon after the tag you must set it as well, it will not be added automatically.

StateFile will create a file where rsyslog keeps track of the position it currently is in a file. You only need to set the filename. This file always is created in the rsyslog working directory (configurable via $WorkDirectory). This file is important so rsyslog will not pull messages from the beginning of the file when being restarted.

Severity will give all log messages of a file the same severity. This is optional. By default all mesages will be set to “notice”.

Facility gives alle log messages of a file the same facility. Again, this is optional. By default all messages will be set to “local0”.

These statements are needed for monitoring a file. There are other statements described in the doc, which you might want to use. If you want to monitor another file the statements must be repeated.

Since the files cannot be monitored in genuine real time (which generates too much processing effort) you need to set a polling interval:

PollingInterval 10

This is a module setting and it defines the interval in which the log files will be polled. By default this value is set to 10 seconds. If you want this to get more near realtime, you can decrease the value, though this is not suggested due to increasing processing load. Setting this to 0 is supported, but not suggested. Rsyslog will continue reading the file as long as there are unprocessed messages in it. The interval only takes effect once rsyslog reaches the end of the file.

Important

The StateFile needs to be unique for every file that is monitored. If not, strange things could happen.