rsyslog error 51 [warning]
File has been truncated
This is not a real error but rather a warning message. Most probably it occurs when monitoring files with imfile and “reopenOnTruncate” has been set to “on”. In this case, it indicates truncation of the file has been detected, and as such imfile begins to read it’s content from the beginning again.
Please see the imfile documentation for limitations of truncation processing.
rsyslog error 2351
Crypto provider only supported for disk-associated queues.
This is a stub entry: If you have questions please post a comment or visit the github issue tracker.
rsyslog error 2151
(Target) host was not specified where it was needed.
This is a stub entry: If you have questions please post a comment or visit the github issue tracker.
rsyslog error 2051
Syntax error (in RainerScript)
This most often occurs with RainerScript (though in theory the error code may be issued for other syntax errors as well).
An incorrect syntax was found inside a RainerScript expression. To fix it, check the expression closely (if you don’t find the error, you may want to post the expression to the rsyslog mailing list).
This is a stub entry: If you have questions please post a comment or visit the github issue tracker.
rsyslog error 2207
(Fatal) error parsing config file.
Something is syntactically wrong with the mentioned config file, so that further interpretation is impossible. The error most often actually is on the quoted line, or at least very close in front of it. A syntax error can be caused by invalid spelling of RainerScript commands, e.g. “stopp” instead of the correct “stop”.
Sample Data:
rsyslogd: version 7.1.11, config validation run (level 3), master config /etc/rsyslog.conf
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 51: syntax error [try http://www.rsyslog.com/e/2207 ]
rsyslogd: CONFIG ERROR: could not interpret master config file ‘/etc/rsyslog.conf’. [try http://www.rsyslog.com/e/2207 ]
rsyslogd: run failed with error -2207 (see rsyslog.h or try http://www.rsyslog.com/e/2207 to learn what that number means)
This is a stub entry: If you have questions please post a comment or visit the github issue tracker.
Tutorial: Sending impstats Metrics to Elasticsearch Using Rulesets and Queues
Originally posted on the Sematext blog: Monitoring rsyslog’s Performance with impstats and Elasticsearch
If you’re using rsyslog for processing lots of logs (and, as we’ve shown before, rsyslog is good at processing lots of logs), you’re probably interested in monitoring it. To do that, you can use impstats, which comes from input module for process stats. impstats produces information like:
– input stats, like how many events went through each input
– queue stats, like the maximum size of a queue
– action (output or message modification) stats, like how many events were forwarded by each action
– general stats, like CPU time or memory usage
In this post, we’ll show you how to send those stats to Elasticsearch (or Logsene — essentially hosted ELK, our log analytics service, that exposes the Elasticsearch API), where you can explore them with a nice UI, like Kibana. For example get the number of logs going through each input/output per hour:
More precisely, we’ll look at:
– useful options around impstats
– how to use those stats and what they’re about
– how to ship stats to Elasticsearch/Logsene by using rsyslog’s Elasticsearch output
– how to do this shipping in a fast and reliable way. This will apply to most rsyslog use-cases, not only impstats
Continue reading “Tutorial: Sending impstats Metrics to Elasticsearch Using Rulesets and Queues”
Change Log
---------------------------------------------------------------------------
Version 8.1.6 [devel] 2014-01-20
- omfile: permit to set global defaults for action parameters
Thanks to Nathan Brown for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/23
- add capability to escape control characters in the C way of doing it
adds new global parameter "parser.escapeControlCharactersCStyle"
Thanks to Nathan Brown for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/13
- parser global parameters can now be set using RainerScript global()
Thanks to Nathan Brown for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/23
- omprog: guard program-to-be-executed against CTL-C
This can frequently happen in debug mode, where rsyslog is terminated
by ctl-c. In any case, SIGINT is not meant to control the child process,
so it should be blocked.
- omprog bugfix: parameter "forceSingleInstance" is NOT mandatory
- add new jsonr property replacer option
Thanks to Nathan Brown for the patch.
- added external plugin interface
- ommongodb: add authentication support (untested)
Thanks to JT for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/17
---------------------------------------------------------------------------
Version 8.1.5 [devel] 2014-01-24
- omprog: ability to execute multiple program instances per action
It can now execute one program instance per worker thread. This is
generally a very good thing the have performance wise. Usually, this
should cause no problems with the invoked program. For that reason,
we have decided to make this the default mode of operation. If not
desired, it can be turned off via the 'forceSingleInstance="on"'
action parameter.
CHANGE OF BEHAVIOUR: previous versions did always execute only one
instance per action, no matter how many workers were active. If
your program has special needs, you need to change your configuration.
- imfile now supports inotify (but must be explicitely turned on)
- imfile no longer has a limit on number of monitored files
- added ProcessInternalMessages global system parameter
This permits to inject rsyslog status messages into *another* main
syslogd or the journal.
- new dependency: liblogging-stdlog (for submitting to external logger)
- bugfix: imuxsock input parameters were not accepted
due to copy&paste error. Thanks to Andy Goldstein for the fix.
---------------------------------------------------------------------------
Version 8.1.4 [devel] 2014-01-10
- add exec_template() RainerScript function
- imrelp: support for TCP KEEPALIVE added
- bumped librelp dependency to 1.2.2 to support new KEEPALIVE feature
- Add directives for numerically specifying GIDs/UIDs
The already present directives (FileOwner, FileGroup, DirOwner,
DirGroup) translate names to numerical IDs, which depends on the user
information being available during rsyslog's startup. This can fail if
the information is obtained over a network or from a service such as
SSSD. The new directives provide a way to specify the numerical IDs
directly and bypass the lookup.
Thanks to Tomas Heinrich for the patch.
- bugfix: action commitTransaction() processing did not properly handle
suspended actions
- bugfix: omelasticsearch fail.es stats counter was improperly maitained
---------------------------------------------------------------------------
Version 8.1.3 [devel] 2013-12-06
THIS VERSION CAN BE CONSIDERED A "NORMAL" DEVEL RELEASE. It's no longer
highly experimental. This assertion is based on real-world feedback.
- changes to the strgen module interface
- new output module interface for transactional modules
- performance improvements
* reduced number of malloc/frees due to further changes to the
output module interface
* reduced number of malloc/frees during string template processing
We now re-use once allocated string template memory for as long
as the worker thread exists. This saves us from doing new memory
allocs (and their free counterpart) when the next message is
processed. The drawback is that the cache always is the size of
the so-far largest message processed. This is not considered a
problem, as in any case a single messages' memory footprint should
be far lower than that of a whole set of messages (especially on
busy servers).
* used variable qualifiers (const, __restrict__) to hopefully help
the compiler generate somewhat faster code
- failed action detection more precisely for a number of actions
If an action uses string parameter passing but is non-transactional
it can be executed immediately, giving a quicker indicatio of
action failure.
- bugfix: limiting queue disk space did not work properly
* queue.maxdiskspace actually initializes queue.maxfilesize
* total size of queue files was not checked against
queue.maxdiskspace for disk assisted queues.
Thanks to Karol Jurak for the patch.
---------------------------------------------------------------------------
Version 8.1.2 [experimental] 2013-11-28
- support for liblognorm1 added - results in performance improvements
Thanks to Pavel Levshin for his work in this regard.
- support for jemalloc added via --enable-jemalloc
Thanks to Pavel Levshin for suggesting jemalloc
Note that build system is experimental at this stage.
- queue defaults have changed
* high water mark is now dynamically 90% of queue size
* low water makr is now dynamically 70% of queue size
* queue.discardMark is now dynamically 98% of queue size
* queue.workerThreadMinimumMessage set to queue.size / num workers
For queues with very low queue.maxSize (< 100), "emergency" defaults
will be used.
- bugfix: disk queues created files in wrong working directory
if the $WorkDirectory was changed multiple times, all queues only
used the last value set.
- bugfix: legacy directive $ActionQueueWorkerThreads was not honored
- bugfix: mmrfc5424addhmac: "key" parameter was not properly processed
---------------------------------------------------------------------------
Version 8.1.1 [experimental] 2013-11-19
- bugfix: STOP/discard(~) was mostly NOT honored
This lead to execution of config code that was not meant to be executed.
- bugfix: memory leak on worker thread termination
- bugfix: potential segfault in omfile under heavy load
Thanks to Pavel Levshin for alerting us.
- bugfix: mmsequence: instance mode did not work
Thanks to Pavel Levshin for the patch
- bugfix: segfault on startup when certain script constructs are used
e.g. "if not $msg ..."
- omhiredis: now supports v8 output module interface and works again
Thanks to Pavel Levshin for the patch
- mmaudit: now supports v8 output module interface and work again
- bugfix: potential abort on startup in debug mode
This depends on template type being used. The root cause was a
non-necessary debug output, which were at the wrong spot (leftover from
initial testing).
Thanks to Pavel Levshin for alerting us and providing a patch
proposal.
---------------------------------------------------------------------------
Version 8.1.0 [experimental] 2013-11-15
- rewritten core engine for higher performance and new features
In detail:
* completely rewritten rule execution engine
* completely changed output module interface
* remodelled output module interface
* enabled important output modules to support full concurrent
operation
The core engine has been considerably changed and must be considered
experimental at this stage. Note that it does not yet include all
features planned for v8, but is close to this goal. In theory, the
engine should perform much better, especially on complex configurations
and busy servers. Most importantly, actions instances can now be called
concurrently from worker threads and many important output modules
support multiple concurrent action instances natively.
- module omruleset is no longer enabled by default.
Note that it has been deprecated in v7 and been replaced by the "call"
statement. Also, it can still be build without problems, the option must
just explicitely be given.
------------------------------Using TLS with RELP
In this guide, we want to describe how to setup rsyslog with a RELP connection which is to be secured with TLS. For this guide you need at least rsyslog 7.5.1 and librelp 1.1.3 as well as gnutls 2.10.0 or above. These need to be installed on the server as well on the clients. The guide will split up into 3 parts.
- Creating the certificates
- Client Configuration
- Server Configuration
Step 1 – Creating the certificates
1.1 Setting up the CA
The first step is to set up a certificate authority (CA). It must be maintained by a trustworthy person (or group) and approves the indentities of all machines. It does so by issuing their certificates. In a small setup, the administrator can provide the CA function. What is important is the the CA’s private key is well-protocted and machine certificates are only issued if it is know they are valid (in a single-admin case that means the admin should not issue certificates to anyone else except himself). The CA creates a so-called self-signed certificate. That is, it approves its own authenticy. This sounds useless, but the key point to understand is that every machine will be provided a copy of the CA’s certificate. Accepting this certificate is a matter of trust. So by configuring the CA certificate, the administrator tells rsyslog which certificates to trust. This is the root of all trust under this model. That is why the CA’s private key is so important – everyone getting hold of it is trusted by our rsyslog instances.
- generate the private key:
certtool --generate-privkey --outfile ca-key.pem
This takes a short while. Be sure to do some work on your workstation, it waits for radom input. Switching between windows is sufficient ;)
- now create the (self-signed) CA certificate itself:
certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem
This generates the CA certificate. This command queries you for a number of things. Use appropriate responses. When it comes to certificate validity, keep in mind that you need to recreate all certificates when this one expires. So it may be a good idea to use a long period, eg. 3650 days (roughly 10 years). You need to specify that the certificates belongs to an authority. The certificate is used to sign other certificates.
Sample Screen Session
Text in red is user input. Please note that for some questions, there is no user input given. This means the default was accepted by simply pressing the enter key.
[root@rgf9dev sample]# certtool --generate-privkey --outfile ca-key.pem --bits 2048
Generating a 2048 bit RSA private key...
[root@rgf9dev sample]# certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem
Generating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Country name (2 chars): US
Organization name: SomeOrg
Organizational unit name: SomeOU
Locality name: Somewhere
State or province order research paper online name: CA
Common name: someName (not necessarily DNS!)
UID:
This field should not be used in new certificates.
E-mail: Enter the certificate's serial number (decimal):
Activation/Expiration time.
The certificate will expire in (days): 3650
Extensions.
Does the certificate belong to an authority? (Y/N): y
Path length constraint (decimal, -1 for no constraint):
Is this a TLS web client certificate? (Y/N):
Is this also a TLS web server certificate? (Y/N):
Enter the e-mail of the subject of the certificate: someone@example.net
Will the certificate be used to sign other certificates? (Y/N): y
Will the certificate be used to sign CRLs? (Y/N):
Will the certificate be used to sign code? (Y/N):
Will the certificate be used to sign OCSP requests? (Y/N):
Will the certificate be used for time stamping? (Y/N):
Enter the URI of the CRL distribution point:
X.509 Certificate Information:
Version: 3
Serial Number (hex): 485a365e
Validity:
Not Before: Thu Jun 19 10:35:12 UTC 2008
Not After: Sun Jun 17 10:35:25 UTC 2018
Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=someName (not necessarily DNS!)
Subject Public Key Algorithm:
RSA Modulus (bits 2048):
d9:9c:82:46:24:7f:34:8f:60:cf:05:77:71:82:61:66
05:13:28:06:7a:70:41:bf:32:85:12:5c:25:a7:1a:5a
28:11:02:1a:78:c1:da:34:ee:b4:7e:12:9b:81:24:70
ff:e4:89:88:ca:05:30:0a:3f:d7:58:0b:38:24:a9:b7
2e:a2:b6:8a:1d:60:53:2f:ec:e9:38:36:3b:9b:77:93
5d:64:76:31:07:30:a5:31:0c:e2:ec:e3:8d:5d:13:01
11:3d:0b:5e:3c:4a:32:d8:f3:b3:56:22:32:cb:de:7d
64:9a:2b:91:d9:f0:0b:82:c1:29:d4:15:2c:41:0b:97
Exponent:
01:00:01
Extensions:
Basic Constraints (critical):
Certificate Authority (CA): TRUE
Subject Alternative Name (not critical):
RFC822name: someone@example.net
Key Usage (critical):
Certificate signing.
Subject Key Identifier (not critical):
fbfe968d10a73ae5b70d7b434886c8f872997b89
Other Information:
Public Key Id:
fbfe968d10a73ae5b70d7b434886c8f872997b89
Is the above information ok? (Y/N): y
Signing certificate...
[root@rgf9dev sample]# chmod 400 ca-key.pem
[root@rgf9dev sample]# ls -l
total 8
-r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem
-rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem
[root@rgf9dev sample]#Be sure to safeguard ca-key.pem! Nobody except the CA itself needs to have it. If some third party obtains it, you security is broken!
1.2 Generating the machine certificate
In this step, we generate certificates for each of the machines. Please note that both clients and servers need certificates. The certificate identifies each machine to the remote peer. The DNSName specified inside the certificate can be specified inside the tls.PermittedPeer config statements. For now, we assume that a single person (or group) is responsible for the whole rsyslog system and thus it is OK if that single person is in posession of all machine’s private keys. This simplification permits us to use a somewhat less complicated way of generating the machine certificates. So, we generate both the private and public key on the CA (which is NOT a server!) and then copy them over to the respective machines. If the roles of machine and CA administrators are split, the private key must be generated by the machine administrator. This is done via a certificate request. This request is then sent to the CA admin, which in turn generates the certificate (containing the public key). The CA admin then sends back the certificate to the machine admin, who installs it. That way, the CA admin never get’s hold of the machine’s private key. Instructions for this mode will be given in a later revision of this document. In any case, it is vital that the machine’s private key is protected. Anybody able to obtain that private key can imporsonate as the machine to which it belongs, thus breaching your security.
Sample Screen Session Text in red is user input. Please note that for some questions, there is no user input given. This means the default was accepted by simply pressing the enter key. Please note: you need to substitute the names specified below with values that match your environment. Most importantly, machine.example.net must be replaced by the actual name of the machine that will be using this certificate. For example, if you generate a certificate for a machine named “server.example.com”, you need to use that name. If you generate a certificate for “client.example.com”, you need to use this name. Make sure that each machine certificate has a unique name. If not, you can not apply proper access control.
[root@rgf9dev sample]# certtool --generate-privkey --outfile key.pem --bits 2048
Generating a 2048 bit RSA private key...
[root@rgf9dev sample]# certtool --generate-request --load-privkey key.pem --outfile request.pem
Generating a PKCS #10 certificate request...
Country name (2 chars): US
Organization name: SomeOrg
Organizational unit name: SomeOU
Locality name: Somewhere
State or province name: CA
Common name: machine.example.net
UID: Enter a dnsName of the subject of the certificate:
Enter the IP address of the subject of the certificate:
Enter the e-mail of the subject of the certificate:
Enter a challange password:
Does the certificate belong to an authority? (y/N): n
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N):
Will the certificate be used for encryption (RSA ciphersuites)? (y/N):
Is this a TLS web client certificate? (y/N): y
Is this also a TLS web server certificate? (y/N): y
[root@rgf9dev sample]# certtool --generate-certificate --load-request request.pem --outfile cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem
Generating a signed certificate...
Enter the certificate's serial number (decimal):
Activation/Expiration time.
The certificate will expire in (days): 1000
Extensions.
Do you want to honour the extensions from the request? (y/N):
Does the certificate belong to an authority? (Y/N): n
Is this a TLS web client certificate? (Y/N): y
Is this also a TLS web server certificate? (Y/N): y
Enter the dnsName of the subject of the certificate: machine.example.net {This is the name of the machine that will use the certificate}
Enter the IP address of the subject of certificate:
Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/N):
Will the certificate be used for encryption (RSA ciphersuites)? (Y/N):
X.509 Certificate Information:
Version: 3 Serial Number (hex): 485a3819
Validity:
Not Before: Thu Jun 19 10:42:54 UTC 2008
Not After: Wed Mar 16 10:42:57 UTC 2011
Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=machine.example.net
Subject Public Key Algorithm: RSA
Modulus (bits 2048):
b2:4e:5b:a9:48:1e:ff:2e:73:a1:33:ee:d8:a2:af:ae
2f:23:76:91:b8:39:94:00:23:f2:6f:25:ad:c9:6a:ab
2d:e6:f3:62:d8:3e:6e:8a:d6:1e:3f:72:e5:d8:b9:e0
d0:79:c2:94:21:65:0b:10:53:66:b0:36:a6:a7:cd:46
1e:2c:6a:9b:79:c6:ee:c6:e2:ed:b0:a9:59:e2:49:da
c7:e3:f0:1c:e0:53:98:87:0d:d5:28:db:a4:82:36:ed
3a:1e:d1:5c:07:13:95:5d:b3:28:05:17:2a:2b:b6:8e
8e:78:d2:cf:ac:87:13:15:fc:17:43:6b:15:c3:7d:b9
Exponent: 01:00:01
Extensions:
Basic Constraints (critical):
Certificate Authority (CA): FALSE
Key Purpose (not critical):
TLS WWW Client. TLS WWW Server.
Subject Alternative Name (not critical):
DNSname: machine.example.net
Subject Key Identifier (not critical):
0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac
Authority Key Identifier (not critical):
fbfe968d10a73ae5b70d7b434886c8f872997b89
Other Information:
Public Key Id:
0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac
Is the above information ok? (Y/N): y
Signing certificate...
[root@rgf9dev sample]# rm -f request.pem
[root@rgf9dev sample]# ls -l
total 16
-r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem
-rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem
-rw-r--r-- 1 root root 1074 2008-06-19 12:43 cert.pem
-rw-r--r-- 1 root root 887 2008-06-19 12:40 key.pem
[root@rgf9dev sample]# # it may be a good idea to rename the files to indicate where they belong to
[root@rgf9dev sample]# mv cert.pem machine-cert.pem
[root@rgf9dev sample]# mv key.pem machine-key.pem
[root@rgf9dev sample]# Distributing Files Provide the machine with:
- a copy of ca.pem
- cert.pem
- key.pem
Never provide anyone with ca-key.pem! Also, make sure nobody but the machine in question gets hold of key.pem.
Step 2: Client Configuration
Usually, this is a rather simple matter. It is basically a configuration to forward messages via RELP, just with a few extra parameters. A configuration could look like this:
module(load="imuxsock") module(load="omrelp") module(load="imtcp") input(type="imtcp" port="514") action(type="omrelp" target="192.168.233.153" port="20514" tls="on" tls.caCert="/home/test/cert/ca.pem" tls.myCert="/home/test/cert/ubuntu1-cert.pem" tls.myPrivKey="/home/test/cert/ubuntu1-key.pem" tls.authmode="name" tls.permittedpeer=["ubuntu-server"] )
The configuration is basically made of 3 parts.
First we load the modules that we need. In this case it is imuxsock, omrelp and imtcp. While imtcp and imuxsock are input modules and thus basically server for “generating” log data, omrelp is a output module and server to forward messages to a server via RELP.
The second part is a input. While imuxsock is always listening when the module is loaded, imtcp needs to be configured as an extra item. This also holds the possibility to configure multiple TCP listeners which could listen to different ports or have different parameters.
The third part is the action. In our case, we simply take all messages and forward them via RELP to our central server. Basically for a RELP connection it would be sufficient to configure the target and the port to send on. For this case, we enable TLS transportation with tls=”on” and secure it with our certificates. As you can see, we need to load our three certificate files as mentioned in the first section. They are all three needed. Also, we define the authentication mode as “name”. By that, the certificate of the other machine will be checked against the hostname to ensure the right machine uses the certificate. Also we allow a certain peer with tls.permittedpeer to answer to this machine.
Step 3: Server Configuration
The server configuration is only a little more complicated than the client configuration. Here is the example.
module(load="imuxsock")
module(load="imrelp" ruleset="relp")
input(type="imrelp" port="20514"
tls="on"
tls.caCert="/home/test/cert/ca.pem"
tls.myCert="/home/test/cert/server-cert.pem"
tls.myPrivKey="/home/test/cert/server-key.pem"
tls.authMode="name"
tls.permittedpeer=["ubuntu-client1","ubuntu-client2","ubuntu-client3"] )
ruleset (name="relp") { action(type="omfile" file="/var/log/relp_log") }Again, the configuration is made of 3 sections.
First we load the modules. In this case, we load imuxsock for local logging ability (though not further used here) and imrelp for receiving log messages via RELP. As you can see, the module has the ruleset parameter given. That means, that the whole module is bound to a ruleset. This means, that all configured listeners will automatically use the later configured ruleset. This is contrary to other modules where you can configure a separate ruleset for each listener, but this is not possible with imrelp.
The second portion is the input. For a basic configuration it would be sufficient to specify the port to listen on. But since we want to enable TLS-functionality, we need to use tls=”on” again. Also we need to configure the parameters for all the parts of the certificate. Please note, that except for the CA certificate (which is the same on all machines) the certificate and key need to be for this single machine. Also we configure the authentication mode to “name” and an array of permitted peers.
Finally, as third step, we configure the ruleset and the action. In this case, we simply use the omfile module. The functionality is automatically loaded through rsyslog and we do not need to configure the module explicitely at the beginning.
Verdict
One would agree, that setting up rsyslog to use TLS-secured RELP for transferring log messages is basically very easy. Though, creating and maintaing all the certificates can be a tedious amount of work. But, a secure log transmission sure is worth the effort. Also, despite what has been mentioned, a machine certificate could also be created for multiple machines at once, though this is not recommended by Adiscon.
Something that is also noteworthy concerns the authentication mode. There are two other options that can be used, but these are not as secure as authentication against the name. Anonymous authentication can be reached by simply using tls=”on”, but not setting any other parameters. This only secures transportation of log messages, but does not ensure, that no false log data can be injected by a compromised machine or a machine not belonging to your network. A different approach would be fingerprint authentication. This can be used by using the fingerprint of a machine certificate on the peer machine. The fingerprint can be either received by the error log of a machine and then copying and pasting it into tls.permittedpeer, or by using certtool to review certificate details. Please note, that this is not very secure as well as the peers only check for the fingerprint of the certificate and not the certificate itself.
Newbie guide to rsyslog
Written by Jan Gerhards
Here are some different guides for people, who never worked with rsyslog. First I’ll explain, how to install rsyslog. Besides, I will explain how to install some packages, which you will need to install rsyslog. There are two important ones and some minor ones. After that, I will show how to do some easy configurations. Questions like how can I configure a module? Or How can I configure the input? Will be answered. In the example I will configure rsyslog to receive messages, filter them for one specific IP and store only the messages in a file. In the end I’ll explain something about easy rulesets. In my example I will try the same like I did with the configuration, but I will work with rulesets.
Before you begin
Please note, there are many ways to distribute rsyslog. But, make sure that the platform you build rsyslog on is the same platform as where it should be used. You cannot build rsyslog on CentOS and use it on Ubuntu. The differences between the platforms are just to big. The same applies to different versions of the same platform. When building on a older platform and using it on a newer version, this may work, but with restrictions. Whereas building on a newer version and using it on a older version will probably not work at all. So, if you build rsyslog from Source and want to use it on another machine, make sure the platform is the same.
Preliminary actions
There are generally two very often needed packages, that you have to install before rsyslog: libestr and libee. These can be downloaded here:
Libestr: http://libestr.adiscon.com/download/
Libee: http://www.libee.org/download/
Both are being installed the same, so here is only one step-by-step instruction.
- Download the file
Just download libestr or libee and “cd” into the folder, where you want them. - “tar xzf” the file
after you “cd” to the folder, where the file is, type “tar xzf -name of file”. Your command line should look like this:tar xzf -name of file-
- “Cd” into the new folder
“cd” into the made directory. It will have the same name like the downloaded file. Your command line should look like this:cd -name of file-
- Run “./configure configure –libdir=/usr/lib –includedir=/usr/include”
After you “cd” into that file, just run ” ./configure –libdir=/usr/lib –includedir=/usr/include”. Your command line should look like this:./configure --libdir=/usr/lib --includedir=/usr/include
- Type “sudo make”
Just type this and let the Computer work. Your command line should look like this:sudo make
- Type “sudo make install”
This is the last step. Like with “sudo make”, just let the computer work. Your command line should look like thissudo make install
- The package should work now
Congratulation! You have installed the package! If it doesn’t work properly, you should check if you followed every step right and if you have, contact the support.
There might also be some other packages like libjson0-dev, uuid-dev, bison, flex, libz-dev or python-docutils. These you’ll have to install before installing rsyslog, too. This guide was made on Ubuntu, so if you use any other system, they might be named differently. You can install them with the command
sudo apt-get install -package name-
After you got all the packages, you can install rsyslog. Here is a How to for installing rsyslog.
How to install rsyslog
In this guide is explained how to install rsyslog with the download from the homepage.
- Download rsyslog
You can download rsyslog from the rsyslog webpage at http://www.rsyslog.com/download/ - “tar xzf” the file
Open a terminal. Then you “cd” to where you want the file and then type “tar xzf -name of downloaded file”. Your command line should look like this:tar xzf -name of downloaded file-
- “cd” into the new folder
Then “cd” into the made directory. The new directory will have the same name like the downloaded file. Your command line should look like this:cd -name of file-
- Type “./configure –prefix=/usr”
You just need to run “./configure –prefix=/usr”. There might be an Error that you need to install some packages to precede with installing. For two specific packages I’ve explained how to install them, because you will need them often. Your command line should look like this:./configure --prefix=/usr
- Run “sudo make”
it’s easy, just run “sudo make” and let the computer work. Your command line should look like this:sudo make
- Run “sudo make install”
Just like “sudo make”. Type it, press enter and let the Computer work. Again, there might be some missing packages. Just install them. Your command line should look like this:sudo make install
- Rsyslog should now be installed
Congratulation! You have installed rsyslog! You can check the version of rsyslog by typing: “rsyslogd -v”.
If it doesn’t work properly, you should check if you followed every step right and if you have, contact the support.
Now after you installed rsyslog, you have to configure it. How to do this is explained here.
Configure rsyslog
In this part I’ll explain some basic configuration steps for rsyslog. We configure rsyslog to recive UDP messages, to filter them depending on the IP of the host and to store them in a file.
- How to configure the module
The module has to be configured first. The general line for this configuration is: “module (load=”im-type of protocol-”). So in our example, where we want UDP, it will look like this:Module (load=”imudp”)
- How to configure the input for rsyslog
For the input, you have to give two different information to rsyslog. The first information needed is the protocol type of the input; in my example again UDP. Like in the first line there is an “im-” in front of the protocol-type. The other information is to configure a port for rsyslog, in my example 514. These two information are together in only one line. The line is: “Input (type=”-protocol of input-“port=”-number of port-“). This means for my example, the line has to beInput (type=”imudp” port=”514”)
- How to configure a filter for fromhost-IPs and store them in a file
A filter always has, like a normal conditional sentence, an “if…then” part. If you want to configure it to do something with all notes from a specific IP, between “if” and “then” will be the property “$fromhost-ip ==”-IP, you want to filter-”. After this stays a “then” and after the “then” follows an action in brackets, which I will explain later. In my example I want only the notes from the host with the IP 172.19.1.135. So the line will beIf $fromhost-ip == “172.19.1.135” then {After this we have to tell the computer, what to do if that case is given. In this example we want him to store these messages in the file “/var/log/network1.log”. This is an action with the type “omfile”. To configure the file where to store the messages, the action is “action (type=”omfile” File=”-filename-“). So in this example, it will look like this:
Action (type=”omfile” file=”/var/log/network1.log”) }
All the lines together now are
Module (load=“imudp“) Input (type=”imudp” port=”514”)
If $fromhost-ip == “172.19.1.135“ then {
Action (type=”omfile” File=”/var/log/network1.log”)
}All in all it means: The input for rsyslog will listen to syslog via UDP on port 514. If the IP from the Computer, which sends the messages, is 172.19.1.135, then the action in the brackets will get activated for these. In the action the messages will be stored in the file /var/log/network1.log.
Rsyslog and rulesets
Rulesets are a bit more complicated. A ruleset is a set of rules, as the name implies. These are bound to an input. This works by adding an option to the input, namely “ruleset=”-rulesetname-“”. For example, if I want to bind a ruleset “rs1” to a input the line will look like this:
Input (type=”imudp” port=”514” ruleset=”rs1”)But you still have to define, what the ruleset should do. In this guide I will limit myself to explain, how to create a ruleset, which has one action: to store all the messages in a file. In my example I want to store the messages in the file /var/log/network1.log”.
You define a ruleset like the normal configuration. To define it, you first name it with ruleset (name=”-rulesetname-“). After this you write what it does, in my example the action action (type=”omfile” file=”/var/log/network1.log”). This action you write in these curly brackets: {}.
So my full example looks like this
Module (load=”imudp”)
Input (type=”imudp” port=”514” ruleset=”rs1”)
Ruleset (name=”rs1”) {
Action (type=”omfile” file=”/var/log/network1.log”)
}In that second example for configurations you can see, how to store all messages from the input into a file by using a ruleset. A rulesset can consist of multiple rules, but without binding it to the input it is useless. It can be bound to an input multiple times or even other rulesets can be called.
Final Conclusion
In this guide I explained how to install rsyslog, how to configure it and how to use rulesets. After you read this guide you are able to do exactly this: you can install rsyslog, configure it and have basic knowlege about rulesets. If you want to learn more about rsyslog, how to configure it or about rulesets, you can find information in the other guides or in the documentation.
rsyslog 6.3.3 config format improvements
In rsyslog 6.3.3, the config processor has finally changed. The old legacy processor (and with it the early RainerScript implementation) is thrown out and has been replaced by the so-called RainerScript processor (why that crazy name?). This is an extremely important step for rsyslog, as it now has the foundation for a much better and intuitive rsyslog.conf format. However, most of that can not be seen in 6.3.3, as it requires more work, especially in the plugin arena. Still, there are a couple of smaller improvements available.
Most importantly, the performance of script based filters has been considerably enhanced. Preliminary testing shows a three times speedup (we’ll do more benchmarking at a later stage; there is also still lots of room for optimization ;-)).
The ugliness of continuation lines has been removed. They may still be used, and this may make a lot of sense with some actions, but you are usually no longer forced to use continuation lines. Take this config snippet from a leading distro:
if ( \ /* kernel up to warning except of firewall */ \ ($syslogfacility-text == 'kern') and \ ($syslogseverity <= 4 /* warning */ ) and not \ ($msg contains 'IN=' and $msg contains 'OUT=') \ ) or ( \ /* up to errors except of facility authpriv */ \ ($syslogseverity <= 3 /* errors */ ) and not \ ($syslogfacility-text == 'authpriv') \ ) \ then /dev/tty10 & |/dev/xconsole
This can now be written as follows:
if ( /* kernel up to warning except of firewall */ ($syslogfacility-text == 'kern') and ($syslogseverity <= 4 /* warning */ ) and not ($msg contains 'IN=' and $msg contains 'OUT=') ) or ( /* up to errors except of facility authpriv */ ($syslogseverity <= 3 /* errors */ ) and not ($syslogfacility-text == 'authpriv') ) then /dev/tty10 & |/dev/xconsole