Load balancing for rsyslog
To use rsyslog effectively in a cluster, one could use the iptables CLUSTERIP feature on linux to setup one IP address that gets shared across the cluster of systems. A heartbeat (with the pacemaker cluster management layer) can keep track of the cluster and make sure that there is always a box handling the traffic.
It does use a multicast MAC address to send the traffic to multiple machines. The kernel does a hash on (one or more of) the source IP, source port, destination IP and destination port. It then divides this hash into buckets (machine 1 of 10) and if it falls into the bucket for this machine, it then sends the packet on to the application. Otherwise the kernel drops the packet.
This has the advantage of not needing any other systems. It can be done entirely on the receiving cluster.
A different approach would be to setup a LVS (Linux Virtual Server) load balancer (or any other commercial load balancer) to divide the TCP traffic.
Note: In any of these configurations, one will want to consider the tcprebindinterval config directive of rsyslog on the sending machines, so they will periodically close and re-open their connection (so that the source port changes). Otherwise one can end up with the traffic being unbalanced between the systems without any way to re-balance the load.
Changelog for 5.9.5 (v5-devel)
Version 5.9.5 [V5-DEVEL], 2012-01-27
- improved impstats subsystem, added many new counters
- enhanced module loader to not rely on PATH_MAX
- refactored imklog linux driver, now combined with BSD driver
The Linux driver no longer supports outdated kernel symbol resolution,
which was disabled by default for very long. Also overall cleanup,
resulting in much smaller code. Linux and BSD are now covered by a
single small driver. - $IMUXSockRateLimitInterval DEFAULT CHANGED, was 5, now 0
The new default turns off rate limiting. This was chosen as people
experienced problems with rate-limiting activated by default. Now it
needs an explicit opt-in by setting this parameter.
Thanks to Chris Gaffney for suggesting to make it opt-in; thanks to
many unnamed others who already had complained at the time Chris made
the suggestion ;-)
rsyslog 5.9.5 (v5-devel)
Download file name: rsyslog 5.9.5 (devel)
rsyslog 5.9.5 (devel)
md5sum: f515212d97fb1901ebdbe4dea892d46f
Author: Rainer Gerhards (rgerhards@adiscon.com)
Version: 5.9.5 File size: 2.277 MB
rsyslog 5.9.5 (v5-devel) released
This release brings many additional statistics counters and a couple of bug fixes. Note that the default setting of $IMUxSockRateLimitInterval was changed to 0, as rate limiting seems to have caused more trouble than it was worth. To enable it, simply set it to 200, the previous default.
ChangeLog:
http://www.rsyslog.com/changelog-for-5-9-5-v5-devel/
Download:
http://www.rsyslog.com/rsyslog-5-9-5-devel/
As always, feedback is appreciated.
Best regards,
Tim Eifler
rsyslog 5.8.7 (v5-stable) released
This is a bug-fixing release. The most important patches resolve instabilities with RFC5424 header fields and information loss when non-wellformed messages are submitted to the system log socket.
ChangeLog:
http://www.rsyslog.com/changelog-for-5-8-7-v5-stable/
Download:
http://www.rsyslog.com/rsyslog-5-8-7-v5-stable/
As always, feedback is appreciated.
Best regards,
Tim Eifler
rsyslog 5.8.7 (v5-stable)
Download file name: rsyslog 5.8.7 (stable)
rsyslog 5.8.7 (stable)
md5sum: d806a91de534d6e33d7d1c1138cfdd37
Author: Rainer Gerhards (rgerhards@adiscon.com)
Version: 5.8.7 File size: 2.383 MB
Changelog for 5.8.7 (v5-stable)
Version 5.8.7 [V5-stable] 2012-01-17
- bugfix: instabilities when using RFC5424 header fields
Thanks to Kaiwang Chen for the patch - bugfix: imuxsock did truncate part of received message if it did not contain a proper date. The truncation occured because we removed that part of the messages that was expected to be the date.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=295 - bugfix: potential abort after reading invalid X.509 certificate
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=290
Thanks to Tomas Heinrich for the patch - bugfix: stats counter were not properly initialized on creation FQDN hostname for multihomed host was not always set to the correct name if multiple aliases existed. Thanks to Tomas Heinreich for the patch.
Using RSyslog Windows Agent to forward log files
in this article we describe how to use the RSyslog Windows Agent to forward log messages that are stored in plain text files. The article itself will be made of two larger steps. Both steps contain some substeps which will be shown in detail in one of the smaller articles. We assume, that no basic configuration is currently available.
This time, we want to use textfiles as log sources. Many programs for Windows do not use the EventLog system. They use simple and plain text files to store their log messages. Though, the information that is logged there could be as important as EventLogs.
Basically, the configuration of RSyslog Windows Agent consists of 3 parts.
1. A so-called service which generates the log data to be processed by, for example, a file monitor.
2. Rules with Filters. Filters give you the power to decide which log messages are important enough to be kept or not.
3. The action that has to be taken. In our case, forwarding the syslog messages.
Step 1: Setting up the ruleset and action.
Usually we start by creating the ruleset, rule and action. The reason lies in the configuration structure. So we will first create the mentioned items. In the end, we will have a basic rule with no particular filter and a forward via syslog action. That means, that all messages will be forwarded to a central host.
Step 2: Setting up the service.
Now we will set up the service. We need to create a File Monitor Service. The File Monitor Service is able to monitor a file or a directory with files. It will check the specified file(s) periodically for new lines (which would be new log messages) and use them for further processing.
That’s it, you are already finished. Easy wasn’t it? Now you should be able to poll log files and forward the log messages to your central syslog server.
rsyslog 6.2.0 (v6-stable) released
This is the initial stable release of rsyslog v6. It is basically the last beta version plus some more fixes. This version provides functional and performance enhancements, for example the Hadoop (HDFS) support has been considerably speeded up by supporting batched insert mode. Also, TCP transmission overhead for TLS has been dramatically improved. TCP now also supports input worker thread pools. Most importantly, rsyslog now supports log normalization via liblognorm rule bases. This permits very high performance normalization of semantically equal messages from different devices (and thus in different syntaxes).
Note that config scoping, available in the beta versions, is NOT supported by v6-stable. This was decided because it would have been functionality equivalent to the new config language upcoming in v6.3 (already available as part of the devel version). As scoping was not available in any earlier versions, introducing it in v6.2 would have added, in the long term, just another method of doing some identicaly thing via the ugly old config language. This would have lead to user confusion and more complex than necessary code. If you are interested in the cleaner config language, we strongly encourage you to have a look at rsyslog 6.3.
With the arrival of the stable v6 version, version 4 will be retired and is no longer officially supported (but support is provided under maintenance contracts, of course).
ChangeLog:
http://www.rsyslog.com/changelog-for-6-2-0-v6-stable/
Download:
http://www.rsyslog.com/rsyslog-6-2-0-v6-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 6.2.0 (v6-stable)
ChangeLog (from 6.1.12 to 6.2.0):
- bugfix (kind of): removed numerical part from pri-text see v6 compatibility document for reasons
- bugfix: race condition when extracting program name, APPNAME, structured data and PROCID (RFC5424 fields) could lead to invalid characters e.g. in dynamic file names or during forwarding (general malfunction of these fields in templates, mostly under heavy load)
- bugfix: imuxsock did no longer ignore message-provided timestamp, if so configured (the *default*). Lead to no longer sub-second timestamps.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=281 - bugfix: omfile returns fatal error code for things that go really wrong previously, RS_RET_RESUME was returned, which lead to a loop inside the rule engine as omfile could not really recover.
- bugfix: rsyslogd -v always said 64 atomics were not present
thanks to mono_matsuko for the patch - bugfix: potential abort after reading invalid X.509 certificate closes:
http://bugzilla.adiscon.com/show_bug.cgi?id=290
Thanks to Tomas Heinrich for the patch - $Begin, $End, $StrictScoping directives have been removed as v6.4 will provide the same functionality in a far better way. So we do not want to clutter the code.
