rsyslog

The rocket-fast system for log processing pipelines

rsyslog

The rocket-fast system for log processing pipelines

rsyslog 5.5.6 (devel) released

adisconteamNews, Release Announcement, ,

Hi all,

We have just released rsyslog 5.5.6, a member of the devel branch.

The new release provides exciting performance enhancements: on multicore-machines it can be many times faster than version 5.5.5 (which already was quite fast). Most importantly, the enhancement provides much better scalability, so adding many additional core gains much more speedup than with any previous version.

A new concept of “strgen” modules has been implemented, which permit to use high speed C code as templates. Also, support for malformed “last message repated n times” messages, as emited by some syslogds, has been added in form of a custom message parser.

There are also a couple of bugfixes and minor improvements.

See ChangeLog for more details.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-5-6-devel/

Download:

http://www.rsyslog.com/rsyslog-5-5-6-devel

As always, feedback is appreciated.

Best regards,
Tom Bergfeld

Changelog for 5.5.6 (devel)

adisconteamChangelog, ,

Version 5.5.6  [DEVEL] (rgerhards), 2010-07-21

  • added parser modules
    • pmlastmsg, which supports the notoriously malformed “last message repeated n times” messages from some syslogd’s (namely sysklogd)
    • pmrfc3164sd (contributed), supports RFC5424 structured data in RFC3164 messages [untested]
  • added new module type “string generator”, used to speed up output processing. Expected speedup for (typical) rsyslog processing is roughly 5 to 6 percent compared to using string-based templates.
    They may also be used to do more complex formatting with custom C code, what provided greater flexibility and probably far higher speed, for example if using multiple regular expressions within a template.
  • added 4 string generators for
    • RSYSLOG_FileFormat
    • RSYSLOG_TraditionalFileFormat
    • RSYSLOG_ForwardFormat
    • RSYSLOG_TraditionalForwardFormat
  • bugfix: mutexes used to simulate atomic instructions were not destructed
  • bugfix: regression caused more locking action in msg.c than necessary
  • bugfix: “$ActionExecOnlyWhenPreviousIsSuspended on” was broken
  • bugfix: segfault on HUP when “HUPIsRestart” was set to “on” thanks varmojfekoj for the patch
  • bugfix: default for $OMFileFlushOnTXEnd was wrong (“off”).

This, in default mode, caused buffered writing to be used, what means that it looked like no output were written or partial lines. Thanks to Michael Biebl for pointing out this bug.

Changelog for 4.6.3 (v4-stable)

adisconteamChangelog,

Version 4.6.3 [v4-stable] (rgerhards), 2010-07-07

  • improvded testbench
  • added test with truly random data received via syslog to test robustness
  • added new configure option that permits to disable and enable an extended testbench
  • bugfix: segfault on HUP when “HUPIsRestart” was set to “on”
    thanks varmojfekoj for the patch
  • bugfix: default for $OMFileFlushOnTXEnd was wrong (“off”).
    This, in default mode, caused buffered writing to be used, what means that it looked like no output were written or partial lines. Thanks to Michael Biebl for pointing out this bug.
  • bugfix: testbench failed when not executed in UTC+1 timezone accidently, the time zone information was kept inside some to-be-checked-for responses
  • temporary bugfix replaced by permanent one for message-induced off-by-one error (potential segfault) (see 4.6.2)
    The analysis has been completed and a better fix been crafted and integrated.
  • some doc fixes; incorrect config samples could cause confusion
    thanks to Anthony Edwards for pointing the problems out

Preserving syslog sender over NAT

Adiscon SupportFAQ,

Question:
I have a number of syslog clients behind a NAT device. The receiver receives syslog messages that travelled over the NAT device. This leads the receiver to believe that all messages originated from the same IP address. With stock syslogd, I can not differentiate between the senders. Is there any way to record the correct sender of the message with rsyslog?

Answer:
OK, I’ve now had some real lab time. The good news in short: if you use rsyslog both on the senders as well as on the receiver, you do NOT have any problems with NAT.

To double-check (and out of curiosity), I also tried with stock syslogd. I used the ones that came with RedHat and FreeBSD. Neither of them reports the sending machine correctly, they all report the NAT address. Obviously, this is what made this thread appear, but it is a good verification for the correctness of my lab. Next, I tried rsyslogd on the sender and stock syslogd on the receiver (just RedHat this time). The machine was still incorrectly displayed as the NAT address. However, now the real machine name immediately followed the NAT address, so you could differentiate the different machines – but in a inconsistent way.

Finally, I tried to run the stock syslogds against rsyslogd. Again, the host was not properly displayed. Actually, this time the host was not displayed at all (with the default rsyslogd template). Instead, the tag showed up in the host field. So this configuration is basically unusable.

The root cause of the NAT issue with stock syslogd obviously is that it does NOT include the HOST header that should be sent as of RFC 3164. This requires the receiver to take the host from the socket, which – in a NATed environment – can only hold the mangled NAT address. Rsyslog instead includes the HOST header, so the actual host name can be taken from that (this is the way rsyslog works with the default templates).

I barely remember seeing this in code when I initially forked rsyslog from sysklogd. I have not verified it once again. I have also not tested with syslog-ng, simply because that is not my prime focus and a lab would have required too much time.

To make a long story short: If you use rsyslog on both the senders and receivers, NAT is no issue for you.

Build problems with 0.9.6

Adiscon SupportFAQ

Bennett Todd has just alerted me of some problems with the new release. In short: the build process seems not to work at all (nor does the install do).

This is actually a documentation issue. The way of compiling rsyslog has changed slightly but importantly. You need to CD into an distribution-specific subdirectory (use linux of in doubt) and then call make. Do NOT do this in the root directory of the rsyslog project. It’s documented here

http://www.rsyslog.com/Documentation-/install.html.phtml

Looks like I must find some other place to document it… Or find a
different solution (read my blog on why I ended up with this:

http://rgerhards.blogspot.com/2005/08/make-syntax-differences.html

Sorry for any hassle. Suggestions are welcome!
Rainer

Scroll to top