rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Second try to test rate limiting

Adiscon Supportsub-faq, , , , ,

Go back to How to build test-tools?

After building the test tool syslog caller, we can give the testing another try. The syslog caller tool will be one process that creates the designated amount of messages. Since it has a very large output, we need it to produce a lot of messages to see the proper effects. We leave the rsyslog config as it is now.

Before we start, delete the logfile and restart rsyslog.

rm /var/log/everything.log
/etc/rc.d/init.d/rsyslog restart

Now we start the syslog caller tool with the following command:

/path/to/rsyslog/tests/syslog_caller -m 1000000

This will run for a little while and then bring you back onto the command prompt. With “-m 1000000” we determine how many messages will be generated. In this case we try 1 million messages, just to be on the safe side.

The entries in the logfile should look like this:

Oct  5 18:22:51 client kernel: imklog 5.7.1, log source = /proc/kmsg started.
Oct  5 18:22:51 client rsyslogd: [origin software="rsyslogd" swVersion="5.7.1" x-pid="7193" x-info="http://www.rsyslog.com"] startOct  5 18:23:52 client syslog_caller[7200]: test message nbr 0, severity=6
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 0, severity=6
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 1, severity=6
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 2, severity=6
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 3, severity=6
.
.
.
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 47, severity=6
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 48, severity=6
Oct  5 18:23:52 client syslog_caller[7200]: test message nbr 49, severity=6
Oct  5 18:23:52 client rsyslogd-2177: imuxsock begins to drop messages from pid 7200 due to rate-limiting
Oct  5 18:23:57 client rsyslogd-2177: imuxsock lost 111673 messages from pid 7200 due to rate-limiting
Oct  5 18:23:57 client syslog_caller[7200]: test message nbr 111723, severity=6
Oct  5 18:23:57 client syslog_caller[7200]: test message nbr 111724, severity=6
Oct  5 18:23:57 client syslog_caller[7200]: test message nbr 111725, severity=6
Oct  5 18:23:57 client syslog_caller[7200]: test message nbr 111726, severity=6
Oct  5 18:23:57 client syslog_caller[7200]: test message nbr 111727, severity=6

The first two messages that we have in the logfile are the startup messages. Right after that, the messages from syslog_caller appear. In the brackets behind the process name we find again the PID. If you compare the PID to the other log messages, you will find that is always the same. Syslog_caller is definately suitable for the test. In the shown log above, i skipped some messages, but the essence is visible. Take a look at message 49 (which is message 50 obviously). Right after that, you find a message logged from rsyslog directly. It states that imuxsock starts dropping messages from PID 7200, in this case the syslog_caller. The next message is again from rsyslog and states the amount of messages that have been lost due to rate-limiting. After this, messages are logged again, at least until rate limiting intervenes again.

This is how we can test the rate limiting properly.

What is imuxsock?

Adiscon Supportsub-faq,

Go back to How to use rate limiting

What is imuxsock some might ask. It is the name of the module that handles Unix Socket Input. For some who might not know, this module manages the delivery of syslog calls from a logging process to rsyslog. The module listens to the log sockets of a Unix system and gives rsyslog the log messages when they occur.

As a module, imuxsock has to be called in the configuration file of rsyslog. The entry looks like this:

$ModLoad imuxsock.so

Because this is the most commonly used module, this entry is in every rsyslog config by default. And because without it, no local logging could work.

The module itself has some options. The default values don’t have to be changed necessarily, because they are pretty balanced. But in some cases this has to be done, like, if a service does separate log socket or when you want to make rate limiting stricter. I won’t give a description to all the options here, because they are documented relatively good in the documentation.

Go on to Changing the settings

Changelog for 5.7.1 (v5-devel)

Adiscon SupportChangelog, , , ,

Version 5.7.1  [V5-DEVEL] (rgerhards), 2010-10-05

  • support for Hadoop’s HDFS added (via omhdfs)
  • imuxsock now optionally use SCM_CREDENTIALS to pull the pid from the log socket itself (thanks to Lennart Poettering for the suggesting this feature)
  • imuxsock now optionally uses per-process input rate limiting, guarding the user against processes spamming the system log (thanks to Lennart Poettering for suggesting this feature)
  • added new config statements
    • $InputUnixListenSocketUsePIDFromSystem
    • $SystemLogUsePIDFromSystem
    • $SystemLogRateLimitInterval
    • $SystemLogRateLimitBurst
    • $SystemLogRateLimitSeverity
    • $IMUxSockRateLimitInterval
    • $IMUxSockRateLimitBurst
    • $IMUxSockRateLimitSeverity
  • imuxsock now supports up to 50 different sockets for input
  • some code cleanup in imuxsock (consider this a release a major modification, especially if problems show up)
  • bugfix: /dev/log was unlinked even when passed in from systemd in which case it should be preserved as systemd owns it

How to build the testing tools?

Adiscon SupportFAQ, sub-faq, , , , ,

This article has been tested with rsyslog 5.7.1 on Fedora 13. It is part of the article “How to use rate limiting?”

Go back to First try to test rate limiting (fail)

When building a configuration for rsyslog, you will sometimes stumble upon the question, if your setup really works. To prove this in your environment is very complicated in most cases. For this we have made some test tools, which help you test your configuration. The main reason for us to create these tools was for testing new features and try out scenarios. But in the end we deliver them with every release. We will describe the way how to create and use the testing tools with the example of the tool syslog_caller. This tool creates a bunch of messages for the local loghost.

First you need to navigate to your rsyslog folder. From there you need to go to the subfolder tests. The path could look like this:

/home/username/rsyslog/tests/

We need to “make” the tests first before we can use it.

# make syslog_caller

The make process gives you some output. It looks like this:

CC syslog_caller.or
CCLD syslog_caller

After that, we are ready to use the test-tool. Use the following command:

./syslog_caller

This will generate 500 messages by default. You can change the default values with some switches. The switches are usually documented in the .c file of the tool. In this case, when using the following:

./syslog_caller -m 10000

will generate 10000 messages. With the test tools, you can do some stress-testing for your rsyslog installation. A good way to proof, if your configuration is working correctly.

Go on to Second try to test rate limiting (success)

How to use debug on demand?

Adiscon SupportFAQ,

This article should show how the “debug on demand” function works in rsyslog. It has been tested and verified with rsyslog 6.1.0 (v6-Devel) on Fedora 13 (steps may vary on different OS). It is assumed, that rsyslog is already working on the system. On notes regarding the installation and configuration, please take a look at the documentation. Debug on Demand is available since version 4.5.7.

To get this option working, we need to enable it first. This is done by enabling it and specifying a path for the debug log file. Open up a terminal with root rights and use the following commands:

export RSYSLOG_DEBUG="DebugOnDemand NoStdOut"
export RSYSLOG_DEBUGLOG=/somepath/example.log

The first option enables the “debug on demand” itself without standard output. This is necessary to have it disabled when starting the service. The second option specifies the path and name of the log file.

Now we have to stop the rsyslog service, before we are ready. Use the following command:

/etc/rc.d/init.d/rsyslog stop

We have now stopped the service and need to start it again. In the current configuration we need to start rsyslog as a foreground application. Due to it being much simpler to configure it this way, rsyslog needs to be a foreground application for “debug on demand” to work instead of a background service. Start it again as foreground application with the following command:

rsyslogd -n

Open up a second terminal (again with root) so we can enable or disable the “debug on demand”. Use this command:

kill -USR1 `cat /var/run/rsyslogd.pid`

Using this command once will enable the debug mode. Using it again will disable it. Please note that you definately need “-USR1” else the rsyslog will be really killed. Further make sure to use the backticks. These are important.

That’s it basically. You can now review your “debug on demand” log files.

Periodic statistics on rsyslog counters

adisconteamFAQ, , ,

Since rsyslog 5.7.0 (V5-Devel) there is a new module called impstats.

This module provides periodic output of rsyslog internal counters. Note that the whole statistics system is currently under development. So availabilty and format of counters may change and is not yet stable (so be prepared to change your trending scripts when you upgrade to a newer rsyslog version).

The set of available counters will be output as a set of syslog messages. This output is periodic, with the interval being configurable (default is 5 minutes). Be sure that your configuration records the counter messages (default is syslog.info).

Note that loading this module has impact on rsyslog performance. Depending on settings, this impact may be severe (for high-load environments).

Please find below some simple steps how to use that module.

Of course you have to name it additionally in the configure. The configure should look like this (please note that the parameters can be different to your configure, the important thing is ––enable-impstats; this example is for Fedora 13 )

./configure --enable-impstats --sbindir=/sbin --libdir=/lib

The next steps are make and make install

make
make install

Furthermore we have to add that module in the rsyslog config. Below you will find a short excerpt of a sample config.

#### Modules ####
$ModLoad imuxsock
$ModLoad imklog
#$ModLoad immark

$ModLoad impstats
$PStatsInterval 600
$PStatsSeverity 7

syslog.debug  /var/log/rsyslog-stats

$ModLoad impstats – tells rsyslog to load the module impstats

$PStatsInterval 600 – is a configuration directive of impstats

$PStatsSeverity 7 – is also a configuration directive of impstats

Here you will find all information about the impstats module and its configuration directives.

If we have a look at the mentioned output file rsyslog-stats we can see the results of the static module.

The content of that file should look like this

Sep 17 11:43:49 localhost rsyslogd-pstats: umxsock: submitted=16
Sep 17 11:43:49 localhost rsyslogd-pstats: main Q: size=1 enqueued=2403 full=0 maxqsize=2

At the actual point all objects are shown in the results. Every Object has its own counter like

umxsock: (= object) submitted=16 (=counter; 16 logs received by the object umxsock). The main queue of rsyslog is also shown -> main Q with the parameters size (messages in the queue), enqueued (all received messages), full (how often was the queue full) and maxqsize (the maximal amount of messages in the queue).

Please be sure that the features and options of that module will be develeloped soon.

All information about the periodic statistics module you will find in the documentation. There is also a list of rsyslog impstats counters available on the web site.

Scroll to top