rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Author : adisconteam

Changelog for 6.1.3 (v6-devel)

adisconteamChangelog, , ,

Version 6.1.3 [DEVEL] (rgerhards), 2011-02-01

  • experimental support for monogodb added
  • added $IMUDPSchedulingPolicy and $IMUDPSchedulingPriority config settings
  • added $LocalHostName config directive
  • improved tcpsrv performance by enabling multiple-entry epoll
    so far, we always pulled a single event from the epoll interface.
    Now 128, what should result in performance improvement (less API
    calls) on busy systems. Most importantly affects imtcp.

  • imptcp now supports non-cancel termination mode, a plus in stability
  • imptcp speedup: multiple worker threads can now be used to read data
  • new directive $InputIMPTcpHelperThreads added
  • bugfix: fixed build problems on some platforms
    namely those that have 32bit atomic operations but not 64 bit ones

  • bugfix: local hostname was pulled too-early, so that some config
    directives (namely FQDN settings) did not have any effect

  • enhanced tcpflood to support multiple sender threads
    this is required for some high-throughput scenarios (and necessary to
    run some performance tests, because otherwise the sender is too slow).

  • added some new custom parsers (snare, aix, some Cisco “specialities”)
    thanks to David Lang

Changelog for 5.6.3 (v5-stable)

adisconteamChangelog, , ,

Version 5.6.3  [V5-STABLE] (rgerhards), 2011-01-26

  • bugfix: action processor released memory too early, resulting inpotential issue in retry cases (but very unlikely due to anotherbug, which I also fixed — only after the fix this problem here became actually visible).
  • bugfix: batch processing flagged invalid message as “bad” under somecircumstances
  • bugfix: unitialized variable could cause issues under extreme conditionsplus some minor nits. This was found after a clang static code analyzeranalysis (great tool, and special thanks to Marcin for telling me about it!)
  • bugfix: batches which had actions in error were not properly retried inall cases
  • bugfix: imfile did duplicate messages under some circumstances

normalizer – first steps for mmnormalize

adisconteamFAQ, , , , , ,

Since rsyslog 6.1.2 there is a new module called mmnormalize. This module provides the capability to normalize log messages via liblognorm. (This example has been done on a Fedora 13 workstation. Please note that there can be some differences in the file paths.)

For using this module you need

libestr
libee
liblognorm
rsyslog 6.1.2

After downloading the mentioned items we have to install them.
You can find a guide for installing liblestr, libee and liblognorm at
http://www.liblognorm.com/help/first-steps-using-liblognorm/.

The next step would be to extract the downloaded rsyslog folder and install it. Please note that we habe to mention the mmnormalize module in the configure
(after extracting:)

$ ./configure --libdir=/usr/lib --sbindir=/sbin --enable-mmnormalize
$ make
$ make install

Now we have to load the mmnormalize module in the rsyslog.conf.

$ModLoad mmnormalize # loads the mmnormalize module
$mmnormalizeRuleBase rulebase.rb # specifies which rulebase is to use
*.* :mmnormalize: # applies normalization to all messages

Finally we need an action that uses the normalizer. Below you will find a simple sample for an action using normalizer

$template cee,"msg is '%msg%', %$!all-json%\n"
*.* /home/test/logfile;cee

This example will write everything to logfile and shows you the original message and what parts have been parsed and which not.

A further sample how to use mmnormalize to filter for IP in messages

$template cee, "ip is '%$!ip%', %$!all-json%\n"
if $!ip != '' then /home/test/logfile-cee;cee

With this example you can extract messages which have an IP in the ‘field’ ip. The available fields are applying to your rulebase, there they are specified. Fields are used in a template like properties but they use the additional signs ‘$!’ in their names, e.g. %$!name% instead of %name%.  Below you will find a link which tells you more about creating a rule base http://www.liblognorm.com/help/creating-a-rulebase/.

Scroll to top