rsyslog

The rocket-fast system for log processing pipelines

rsyslog

The rocket-fast system for log processing pipelines

Changelog for 5.9.0 (v5-devel)

adisconteamChangelog, , , ,

Version 5.9.0 [V5-DEVEL] (rgerhards), 2011-06-08

  • imfile: added $InputFileMaxLinesAtOnce directive
  • enhanced imfile to support input batching
  • added capability for imtcp and imptcp to activate keep-alive packets
    at the socket layer. This has not been added to imttcp, as the latter is
    only an experimental module, and one which did not prove to be useful.
    reference: http://kb.monitorware.com/post20791.html

  • added support to control KEEPALIVE settings in imptcp
    this has not yet been added to imtcp, but could be done on request.

  • $ActionName is now also used for naming of queues in impstats
    as well as in the debug output

  • bugfix: do not open files with full privileges, if privs will be dropped
    This make the privilege drop code more bulletproof, but breaks Ubuntu’s
    work-around for log files created by external programs with the wrong
    user and/or group. Note that it was long said that this “functionality”
    would break once we go for serious privilege drop code, so hopefully
    nobody still depends on it (and, if so, they lost…).

  • bugfix: pipes not opened in full priv mode when privs are to be dropped
  • this begins a new devel branch for v5
  • better handling of queue i/o errors in disk queues. This is kind of a
    bugfix, but a very intrusive one, this it goes into the devel version
    first. Right now, “file not found” is handled and leads to the new
    emergency mode, in which disk action is stopped and the queue run
    in direct mode. An error message is emited if this happens.

  • added support for user-level PRI provided via systemd
  • added new config directive $InputTCPFlowControl to select if tcp
    received messages shall be flagged as light delayable or not.

  • enhanced omhdfs to support batching mode. This permits to increase
    performance, as we now call the HDFS API with much larger message
    sizes and far more infrequently

  • bugfix: failover did not work correctly if repeated msg reduction was on
    affected directive was: $ActionExecOnlyWhenPreviousIsSuspended on
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=236

rsyslog 6.3.1 (devel) released

Adiscon SupportNews, Release Announcement, , , ,

This is a first implementation of a full-blown DNS cache. While there were some optimizations for DNS queries in older releases, especially UDP sometimes suffered under slow DNS resolution performance.

This is solved with the new dnscache module. Note that the module will undergo some more enhancements in the next couple of weeks. Feedback on its effect would be deeply appreciated.

ChangeLog:

http://www.rsyslog.com/changelog-for-6-3-1-devel/

Download:

http://www.rsyslog.com/rsyslog-6-3-1-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Log normalization and the leading space

Adiscon SupportFAQ, , ,

Log normalization is simple, but has its quirks. A common pitfall is syslog message format as induced by RFC3164. Let’s look at a common case:  A log message has been sent to rsyslog. The message itself had no irregular characters. But, the message that should have been parsed by mmnormalize now has a leading space character. Basically, the message that should be parsed looks like this:

This is a test

Usually, one would think, that a simple parser can be used here. You might be correct, but there is a small caveat about this. The rulebase entry we currently have looks something like this:

rule=:%word1:word% %word2:word% %word3:word% %word4%

But strangely, rsyslog responds the following:

mmnormalize generated: {“originalmsg”: ” This is a test″, “unparsed-data”: ” This is a test″}

How comes, that rsyslog cannot parse the message? Why is there a leading space character in from of the message? The answer is, that messages are processed as RFC3164. In this RFC it is defined, that everything after the “:” of the syslog header is to be considered as the message. Thus, the message has a leading space now.

How is this to be solved? Simply insert the space to your rules in the rulebase. This will lead to a rule like this:

rule=: %word1:word% %word2:word% %word3:word% %word4%

Please note, that there has just the space character been added. Further, this is really only a example. The rule will fit to all messages that are 4 words long, so it is really not very suitable to be adopted to your configuration.

rsyslog 6.3.0 (devel) released

Adiscon SupportNews, Release Announcement, , , , ,

The release of rsyslog 6.3.0 introduces a new major feature. With this release we introduce a new config system. You can find some information about the new config system in Rainer’s blog:

http://blog.gerhards.net/2011/06/new-rsyslog-config-system-materializes.html

ChangeLog:

http://www.rsyslog.com/changelog-for-6-3-0-devel/

Download:

http://www.rsyslog.com/rsyslog-6-3-0-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

new rsyslog config system materializes

Adiscon SupportNews, , , ,

The past weeks we have worked pretty hard on the new rsyslog config system. The legacy system was quite different from what you expect in modern software. Most importantly, the legacy system applied directives as the config file was read, which made it extremely hard to restructure the config file format. That also prevented features like privilege drop from working fully correct.

We have now basically changed the config system so that there is a clear difference between the config load phase and applying the config. Most importantly, this means privilege drop now works correctly in all cases. Other than that, there are no user-visible enhancements at the moment. However, the internal plumbing has changed dramatically and enables future change. Most importantly, this finally creates a path to a new config language, as we now have a clear interface as part of the in-memory representation of the config, which is config language agnostic.

With this initial release, there may still be some things inside the core that can be optimized. Right now, the system aims at the capability to have multiple config objects loaded (but not active) at the same time. However, there are some data instances where this is not cleanly followed in order to reuse some code. This is not a problem, because the rest of the rsyslog engine does not support dynamic config reload (and thus multiple configs at runtime) at all.

Also it must be noted that the current code is quite experimental. So there is some risk involved in running the initial 6.3.0 version. However, all dramatic changes were made to the config system. That means if the system initializes correctly, it will most probably run without any issues. The risk window is constrained to the initial startup, what should be quite controllable. Users that use privilege drop are advised to check that their configurations work as expected. The previous system did some initialization with full privileges. This is no longer the case, except for modules that actually require full privileges (e.g. imtcp to bind privileged ports). Most importantly, files are now created with dropped privileges right from the beginning. I expect that some (unclean) configurations will run into trouble with that. The good news about that is that they would run into trouble with older releases as well, but only after a HUP. Now things break immediately, what makes them much easier to diagnose.

Scroll to top