rsyslog

The rocket-fast system for log processing

Howto anonymize messages that go to specific files

By Rainer GerhardsPosted on Posted in FAQ

Rsyslog’s mmanon module is used to anonymize data. It is important to keep in you mind that mmanon actually modifies the message. So, as stated in the module documentation, the original, non-anonymized message can no longer be obtained once mmanon has been applied (except, of course, if the message was stored to a variable before calling mmanon). Continue reading “Howto anonymize messages that go to specific files”

How to add a HMAC to RFC5424 structured data messages

By Adiscon SupportPosted on Posted in FAQTagged , , , , ,

rsyslog features a new message modification module, that will check for the SD ID in RFC5424 messages and append a HMAC hash to the structured data part of the message. Please note, that even if the module works on all messages, only RFC5424 messages will be processed.

Before starting, you need a private Enterprise Number from IANA so you can use the module to add the HMAC hash to the message.

For the functionality you need the module “mmrfc5424addhmac”. This is currently available only in the git master branch and will be first released in the next devel release 7.5.4 and for stable in 7.6.0.

When doing the configure, please do not forget to enable this module:

./configure --prefix=/usr --enable-imtcp --enable-mmrfc5424addhmac

Now for the configuration:

module(load="imtcp")
module(load="mmrfc5424addhmac")
input(type="imtcp" port="514")
action(type="mmrfc5424addhmac" key="yourenterprisekey" hashFunction="sha256" sd_id="id@32473")
template(name="addhmac" type="string" string="<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% 
%APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n")
action(type="omfile" file="/var/log/logfile" template="addhmac")

This is a relatively simple configuration. We load the modules imtcp and mmrfc5424addhmac. We will receive all RFC5424 messages through our tcp input. The really interesting part are the actions. Since we need to modify the messages, we need an action with the message modification module. As parameters for the action we define a key that will be used to create the hash, choose a hash function (basically all hash functions from openssl work) and define our SD ID which consists of a name, “@” and the ID received from IANA.

The message will be parsed for the ID, if it exists, a hash will be generated and appended to the structured data of the message.

Now we need to do something with these messages. The template above is a RFC5424 representation and gives out accordingly formatted messages. As follow-up action we will write all messages into a file. Alternatively, you could also forward them to another host or write them into a database.

How to add a module global parameter

By Rainer GerhardsPosted on Posted in faq-devel

In this git commit, two new module parameters are being added to omfile. They replace some hard-coded defaults for action parameters. This commit shows

  • how to add module parameters
  • how to apply them as defaults to action parameters

Please be sure to note the difference between module and action parameters.

Note that the new module parameters do not modify behavior of legacy (pre-v6) config statements. This is done intentionally to prevent complication with upward compatibility of old-style config files.

This commit is intentionally minimal in order to show the most important aspects. For a proper commit, the ChangeLog as well as the module documentation need to be updated as well. An example commit for the doc parts can be found here. It is usually suggested to do both tasks in a single commit (as they obviously closely belong together).

How to obtain a specific Doc Version?

By Rainer GerhardsPosted on Posted in FAQ

There are very many rsyslog versions out in the wild. Each of these versions have different capabilities and consequently there is also different documentation required to match an exact version.

The rsyslog site always contains documentation for the latest development version (sometimes even features that just reside in git). While chances are good that the online doc can be used to solve a specific question, this is not guaranteed. Most importantly, older versions may have quite different rsyslog.conf formats that they support. As such, folks tend to ask how they can obtain a version of the documentation that exactly matches their version.

The answer is quite straightforward: most of the doc on the site comes from rsyslog’s project doc set, and that doc set is contained in each release tarball. So if you have e.g. 5.8.10 installed and want it’s doc, simply go ahead and download the relevant tarball. The full doc is contained in the “./doc” subdirectory. It’s in HTML format, so all you need to do is to point your browser at ./doc/index.html.

In many distributions it is even simpler, as there is a package rsyslog-doc (or similarly named), which contains that exact doc folder.

rsyslog 7.5.3 (v7-devel) released

By Adiscon SupportPosted on Posted in News, Release AnnouncementTagged , , , , , , , ,
This release offers important new features like support for global and local variables, improvements in imfile multi-line handling and enhancements in the statistics subsystem. It also include bug fixes, including those imported from 7.4.4.

Note that the new imfile multi-line handling may cause a change of behaviour if you use it together with the input() config statement. For details, please see

http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html

Users of the rsyslog development branch are strongly encouraged to upgrade to the new version.

ChangeLog:

http://www.rsyslog.com/changelog-for-7-5-3-v7-devel/

Download:

http://www.rsyslog.com/rsyslog-7-5-3-v7-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Changelog for 7.5.3 (v7-devel)

By Adiscon SupportPosted on Posted in ChangelogTagged , , , , , , , ,

Version 7.5.3 [devel] 2013-09-11

  • imfile: support for escaping LF characters added embedded LF in syslog messages cause a lot of trouble. imfile now has the capability to escape them to “#012″ (just like the regular control character escape option). This requires new-style input statements to be used. If legacy configuration statements are used, LF escaping is always turned off to preserve compatibility.
    NOTE: if input() statements were already used, there is a CHANGE OF BEHAVIOUR: starting with this version, escaping is enabled by default. So if you do not want it, you need to add escapeLF=”off” to the input statement. Given the trouble LFs cause and the fact that the majority of installations still use legacy config, we considered this behaviour change acceptable and useful.
    see also: http://blog.gerhards.net/2013/09/imfile-multi-line-messages.html
  • add support for global and local variables
  • bugfix: queue file size was not correctly processed
    this could lead to using one queue file per message for sizes >2GiB
    Thanks to Tomas Heinrich for the patch.
  • add main_queue() configuration object to configure main message queue
  • bugfix: stream compression in imptcp caused timestamp to be corrupted
  • imudp: add ability to specify SO_RCVBUF size (rcvbufSize parameter)
  • imudp: use inputname for statistics, if configured
  • impstats: add process resource usage counters [via getrusage()]
  • impstats: add paramter “resetCounters” to report delta values possible for most, but not all, counters. See doc for details.
  • librelp 1.2.0 is now required
  • make use of new librelp generic error reporting facility
    This leads to more error messages being passed to the user and thus simplified troubleshooting.
  • bugfix: very small memory leak in imrelp
    more or less cosmetic, a single memory block was not freed, but this only happens immediately before termination (when the OS automatically frees all memory). Still an annoyance e.g. in valgrind.
  • fix compile problem in debug build
  • imported fixes from 7.4.4

Changelog for 7.4.4 (v7-stable)

By Adiscon SupportPosted on Posted in ChangelogTagged , , , , ,

Version 7.4.4  [v7.4-stable] 2013-09-03

  • better error messages in GuardTime signature provider
    Thanks to Ahto Truu for providing the patch.
  • make rsyslog use the new json-c pkgconfig file if available
    Thanks to the Gentoo team for the patches.
  • bugfix: imfile parameter “persistStateInterval” was unusable
    due to a case typo in imfile; work-around was to use legacy config
    Thanks to Brandon Murphy for reporting this bug.
  • bugfix: TLV16 flag encoding error in signature files from GT provider
    This fixes a problem where the TLV16 flag was improperly encoded. Unfortunately, existing files already have the bug and may not properly be processed. The fix uses constants from the GuardTime API lib to prevent such problems in the future.
    Thanks to Ahto Truu for providing the patch.
  • bugfix: slightly malformed SMTP handling in ommail
  • bugfix: segfault in omprog if no template was provided (now dflt is used)
  • bugfix: segfault in ompipe if no template was provided (now dflt is used)
  • bugfix: segfault in omsnmp if no template was provided (now dflt is used)
  • bugfix: some omsnmp optional config params were flagged as mandatory
  • bugfix: segfault in omelasticsearch when resuming queued messages after restarting Elasticsearch
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=464
  • bugfix: imtcp addtlframedelimiter could not be set to zero
    Thanks to Chris Norton for alerting us.
  • doc bugfix: remove no-longer existing omtemplate from developer doc was specifically mentioned as a sample for creating new plugins
    Thanks to Yannick Brosseau for alerting us of this problem.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=473
Scroll to top