Howto store remote messages in a separate file

In this ~8 minute video Rainer Gerhards describes Howto store remote messages in a separate file. It’s actually one of the most frequently asked question on rsyslog forum and mailing list.

Note: the tutorial is for legacy config format in order to help most people gain benefit from it.

Please accept YouTube cookies to play this video. By accepting you will be accessing content from YouTube, a service provided by an external third party.

YouTube privacy policy

If you accept this notice, your choice will be saved and the page will refresh.

How to use tcpflood for rsyslog testing?

First of all tcpflood is a testing tool of the rsyslog testbench which is able to send a lot of messages via tcp.

This tool is included in the local rsyslog folder in ‘tests’. Before you can use tcpflood you have to build it first. Just switch in the mentioned folder and type

make tcpflood

Now you can use tcpflood by using the following command

./tcpflood -"options"

as example, just insert -m10000 as “options” to send 10000 messages.

./tcpflood -m10000

Please note that tcpflood uses port 13514 by default, so you have to adjust your tcp listener in your rsyslog config (or the port of tcpflood using the option -p).

tcpflood and the testbench

Following up, we assume that tcpflood is present in the current directory but not the system search path. The tool has many command line options. Many of them have defaults, but these defaults are mostly suitable for use inside rsyslog’s testbench. For performance testing, a proper set of options must be selected. It is suggested that tcpflood, together with its option set, is kept inside a script file for easy test reproduction at later times.

The table below lists tcpflood command line options present at time of this writing. Note that tcpflood is still not a tool meant for average user consumption and thus under heavy development based solely on the needs of the rsyslog developers. We suggest that you open the tcpflood.c source code. Right at the top, there is a comment block describing the actual options available with the version you obtained.

-ttarget address (default
-ptarget port (default 13514)
-nnumber of target ports (targets are in range -p..(-p+-n-1) Note -c must also be set to at LEAST the number of -n!
-cnumber of connections (default 1)
-mnumber of messages to send (connection is random)
-iinitial message number (optional)
-PPRI to be used for generated messages (default is 167). Specify the plain number without leading zeros
-damount of extra data to add to message. If present, the number itself will be added as third field, and the data bytes as forth. Add -r to randomize the amount of extra data included in the range 1..(value of -d).
-rrandomize amount of extra data added (-d must be > 0)
-s(silent) do not show progress indicator (never done on non-tty)
-fsupport for testing dynafiles. If given, include a dynafile ID in the range 0..(f-1) as the SECOND field, shifting all field values one field to the right. Zero (default) disables this functionality.
-Mthe message to be sent. Disables all message format options, as only that exact same message is sent.
-Iread specified input file, do NOT generate own test data. The test completes when eof is reached.
-BThe specified file (-I) is binary. No data processing is done by tcpflood. If multiple connections are specified, data is read in chunks and spread across the connections without taking any record delemiters into account.
-Cwhen input from a file is read, this file is transmitted -C times (C like cycle, running out of meaningful option switches ;))
-Drandomly drop and re-establish connections. Useful for stress-testing the TCP receiver.
-FUSASCII value for frame delimiter (in octet-stuffing mode), default LF
-Rnumber of times the test shall be run (very useful for gathering performance data and other repetitive things). Default: 1
-Snumber of seconds to sleep between different runs (-R) Default: 30
-Xgenerate sTats data records. Default: off
-eencode output in CSV (not yet everywhere supported) for performance data: each inidividual line has the runtime of one test the last line has 0 in field 1, followed by numberRuns,TotalRuntime, Average,min,max
-Ttransport to use. Currently supported: “udp”, “tcp” (default) Note: UDP supports a single target port, only
-Wwait time between sending batches of messages, in microseconds (Default: 0)
-bnumber of messages within a batch (default: 100 million)
-Yuse multiple threads, one per connection (which means 1 if one only connection is configured!)
-zprivate key file for TLS mode
-Zcert (public key) file for TLS mode
-Lloglevel to use for GnuTLS troubleshooting (0-off to 10-all, 0 default)

How to use rate limiting in rsyslog?

This article is tested with rsyslog 5.7.1 on Fedora 13. It will not work with version of rsyslog prior to 5.7.1.

In rsyslog 5.7.1 we introduced rate limiting. This is a option for the Unix Socket Input module called imuxsock. In short, this option limits the amount of messages written into logfiles by a process, if the process tries to write huge amounts of messages in a short period of time.

If you look a bit more into the details, this new feature makes a lot of sense. Because when regular logging happens, this makes no difference. But if a process goes wild (and I bet everyone has experienced this at least once) and starts to create large bulks of log messages, this affects the whole system in general. This blocks not only your I/O, but creates a large amount of logs that usually is simply not necessary and doesn’t really hold useful information. Therefore such messages are now discarded automatically. We will describe later how you can customize these settings for this.

This article is aimed for those who want to learn a bit more about how the rate limit works or who are new to this topic. Therefore we will not only describe how to set this up or change settings, but we want to show you some ways of testing, how imuxsock works and where some caveats are lying.

Please go on to the following pages.

What is imuxsock?

Changing the settings

First try to test rate limiting (fail)

How to build test-tools?

Second try to test rate limiting (success)

Scroll to top