rsyslog

The rocket-fast system for log processing

normalizer – first steps for mmnormalize

Since rsyslog 6.1.2 there is a new module called mmnormalize. This module provides the capability to normalize log messages via liblognorm. (This example has been done on a Fedora 13 workstation. Please note that there can be some differences in the file paths.)

For using this module you need

libestr
libee
liblognorm
rsyslog 6.1.2

After downloading the mentioned items we have to install them.
You can find a guide for installing liblestr, libee and liblognorm at
http://www.liblognorm.com/help/first-steps-using-liblognorm/.

The next step would be to extract the downloaded rsyslog folder and install it. Please note that we habe to mention the mmnormalize module in the configure
(after extracting:)

$ ./configure --libdir=/usr/lib --sbindir=/sbin --enable-mmnormalize
$ make
$ make install

Now we have to load the mmnormalize module in the rsyslog.conf.

$ModLoad mmnormalize # loads the mmnormalize module
$mmnormalizeRuleBase rulebase.rb # specifies which rulebase is to use
*.* :mmnormalize: # applies normalization to all messages

Finally we need an action that uses the normalizer. Below you will find a simple sample for an action using normalizer

$template cee,"msg is '%msg%', %$!all-json%\n"
*.* /home/test/logfile;cee

This example will write everything to logfile and shows you the original message and what parts have been parsed and which not.

A further sample how to use mmnormalize to filter for IP in messages

$template cee, "ip is '%$!ip%', %$!all-json%\n"
if $!ip != '' then /home/test/logfile-cee;cee

With this example you can extract messages which have an IP in the ‘field’ ip. The available fields are applying to your rulebase, there they are specified. Fields are used in a template like properties but they use the additional signs ‘$!’ in their names, e.g. %$!name% instead of %name%.  Below you will find a link which tells you more about creating a rule base http://www.liblognorm.com/help/creating-a-rulebase/.

3 thoughts on “normalizer – first steps for mmnormalize

  1. Pingback: rsyslog 6.1.2 (devel) released rsyslog

  2. This is awesome.
    Possibilities are endless.
    This is really the future of log management.
    If I can suggest one more thing :
    The limitation of this method is the database model, which is static.
    It would be nice to use a schemaless database, nosql like mongodb would be just perfect for this.
    see http://blog.mongodb.org/post/172254834/mongodb-is-fantastic-for-logging
    liblognorm would generate json, and load it in mongodb
    it would be much more flexible for insertion, requesting, reporting, graphing…
    liblognorm is ready for thisk, i hope loganalyzer is going in that direction as well ;)
    many thanks to rsyslog and loganalyzer teams !

Comments are closed.