rsyslog

The rocket-fast system for log processing pipelines

Changelog for 7.5.4 (v7-devel)

By Adiscon SupportPosted on Posted in ChangelogTagged , , , , , , , , , , , ,

Version 7.5.4 [devel] 2013-10-07

  • mmpstrucdata: new module to parse RFC5424 structured data into json message properties
  • change main/ruleset queue defaults to be more enterprise-like
    new defaults are queue.size 100,000 max workers 2, worker activation after 40,000 msgs are queued, batch size 256. These settings are much more useful for enterprises and will not hurt low-end systems that much. This is part of our re-focus on enterprise needs.
  • omfwd: new action parameter “maxErrorMessages” added
  • omfile: new module parameters to set action defaults added
    * dirCreateMode
    * fileCreateMode
  • mmutf8fix: new module to fix invalid UTF-8 sequences
  • imuxsock: handle unlimited number of additional listen sockets
  • doc: improve usability by linking to relevant web ressources
    The idea is to enable users to quickly find additional information, samples, HOWTOs and the like on the main site. At the same time, (very) slightly remove memory footprint when few listeners are monitored.
  • bugfix: omfwd parameter streamdrivermmode was not properly handled
    It was always overwritten by whatever value was set via the legacy directive $ActionSendStreamDriverMode
  • imtcp: add streamdriver.name module parameter
    permits overriding the system default stream driver (gtls, ptcp)
  • bugfix: build system: libgcrypt.h needed even if libgrcypt was disabled
    Thanks to Jonny Törnbom for reporting this problem
  • imported bugfixes from 7.4.4

Howto anonymize messages that go to specific files

By Rainer GerhardsPosted on Posted in FAQ

Rsyslog’s mmanon module is used to anonymize data. It is important to keep in you mind that mmanon actually modifies the message. So, as stated in the module documentation, the original, non-anonymized message can no longer be obtained once mmanon has been applied (except, of course, if the message was stored to a variable before calling mmanon). Continue reading “Howto anonymize messages that go to specific files”

How to add a HMAC to RFC5424 structured data messages

By Adiscon SupportPosted on Posted in FAQTagged , , , , ,

rsyslog features a new message modification module, that will check for the SD ID in RFC5424 messages and append a HMAC hash to the structured data part of the message. Please note, that even if the module works on all messages, only RFC5424 messages will be processed.

Before starting, you need a private Enterprise Number from IANA so you can use the module to add the HMAC hash to the message.

For the functionality you need the module “mmrfc5424addhmac”. This is currently available only in the git master branch and will be first released in the next devel release 7.5.4 and for stable in 7.6.0.

When doing the configure, please do not forget to enable this module:

./configure --prefix=/usr --enable-imtcp --enable-mmrfc5424addhmac

Now for the configuration:

module(load="imtcp")
module(load="mmrfc5424addhmac")
input(type="imtcp" port="514")
action(type="mmrfc5424addhmac" key="yourenterprisekey" hashFunction="sha256" sd_id="id@32473")
template(name="addhmac" type="string" string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% 
%APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")
action(type="omfile" file="/var/log/logfile" template="addhmac")

This is a relatively simple configuration. We load the modules imtcp and mmrfc5424addhmac. We will receive all RFC5424 messages through our tcp input. The really interesting part are the actions. Since we need to modify the messages, we need an action with the message modification module. As parameters for the action we define a key that will be used to create the hash, choose a hash function (basically all hash functions from openssl work) and define our SD ID which consists of a name, “@” and the ID received from IANA.

The message will be parsed for the ID, if it exists, a hash will be generated and appended to the structured data of the message.

Now we need to do something with these messages. The template above is a RFC5424 representation and gives out accordingly formatted messages. As follow-up action we will write all messages into a file. Alternatively, you could also forward them to another host or write them into a database.

How to add a module global parameter

By Rainer GerhardsPosted on Posted in faq-devel

In this git commit, two new module parameters are being added to omfile. They replace some hard-coded defaults for action parameters. This commit shows

  • how to add module parameters
  • how to apply them as defaults to action parameters

Please be sure to note the difference between module and action parameters.

Note that the new module parameters do not modify behavior of legacy (pre-v6) config statements. This is done intentionally to prevent complication with upward compatibility of old-style config files.

This commit is intentionally minimal in order to show the most important aspects. For a proper commit, the ChangeLog as well as the module documentation need to be updated as well. An example commit for the doc parts can be found here. It is usually suggested to do both tasks in a single commit (as they obviously closely belong together).

How to obtain a specific Doc Version?

By Rainer GerhardsPosted on Posted in FAQ

There are very many rsyslog versions out in the wild. Each of these versions have different capabilities and consequently there is also different documentation required to match an exact version.

The rsyslog site always contains documentation for the latest development version (sometimes even features that just reside in git). While chances are good that the online doc can be used to solve a specific question, this is not guaranteed. Most importantly, older versions may have quite different rsyslog.conf formats that they support. As such, folks tend to ask how they can obtain a version of the documentation that exactly matches their version.

The answer is quite straightforward: most of the doc on the site comes from rsyslog’s project doc set, and that doc set is contained in each release tarball. So if you have e.g. 5.8.10 installed and want it’s doc, simply go ahead and download the relevant tarball. The full doc is contained in the “./doc” subdirectory. It’s in HTML format, so all you need to do is to point your browser at ./doc/index.html.

In many distributions it is even simpler, as there is a package rsyslog-doc (or similarly named), which contains that exact doc folder.

rsyslog 7.5.3 (v7-devel) released

By Adiscon SupportPosted on Posted in News, Release AnnouncementTagged , , , , , , , ,
This release offers important new features like support for global and local variables, improvements in imfile multi-line handling and enhancements in the statistics subsystem. It also include bug fixes, including those imported from 7.4.4.

Note that the new imfile multi-line handling may cause a change of behaviour if you use it together with the input() config statement. For details, please see

http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html

Users of the rsyslog development branch are strongly encouraged to upgrade to the new version.

ChangeLog:

http://www.rsyslog.com/changelog-for-7-5-3-v7-devel/

Download:

http://www.rsyslog.com/rsyslog-7-5-3-v7-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Changelog for 7.5.3 (v7-devel)

By Adiscon SupportPosted on Posted in ChangelogTagged , , , , , , , ,

Version 7.5.3 [devel] 2013-09-11

  • imfile: support for escaping LF characters added embedded LF in syslog messages cause a lot of trouble. imfile now has the capability to escape them to “#012″ (just like the regular control character escape option). This requires new-style input statements to be used. If legacy configuration statements are used, LF escaping is always turned off to preserve compatibility.
    NOTE: if input() statements were already used, there is a CHANGE OF BEHAVIOUR: starting with this version, escaping is enabled by default. So if you do not want it, you need to add escapeLF=”off” to the input statement. Given the trouble LFs cause and the fact that the majority of installations still use legacy config, we considered this behaviour change acceptable and useful.
    see also: http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html
  • add support for global and local variables
  • bugfix: queue file size was not correctly processed
    this could lead to using one queue file per message for sizes >2GiB
    Thanks to Tomas Heinrich for the patch.
  • add main_queue() configuration object to configure main message queue
  • bugfix: stream compression in imptcp caused timestamp to be corrupted
  • imudp: add ability to specify SO_RCVBUF size (rcvbufSize parameter)
  • imudp: use inputname for statistics, if configured
  • impstats: add process resource usage counters [via getrusage()]
  • impstats: add paramter “resetCounters” to report delta values possible for most, but not all, counters. See doc for details.
  • librelp 1.2.0 is now required
  • make use of new librelp generic error reporting facility
    This leads to more error messages being passed to the user and thus simplified troubleshooting.
  • bugfix: very small memory leak in imrelp
    more or less cosmetic, a single memory block was not freed, but this only happens immediately before termination (when the OS automatically frees all memory). Still an annoyance e.g. in valgrind.
  • fix compile problem in debug build
  • imported fixes from 7.4.4
Scroll to top