Using MongoDB with rsyslog and LogAnalyzer
In this scenario we want to receive cee-formatted messages from a different system with rsyslog, store the messages with MongoDB and then display the stored messages with Adiscon LogAnalyzer. This is a very common use-case. Please read through the complete guide before starting.
We will split this guide in 3 main parts. These parts can consist of several steps.
- Setting up rsyslog
- Setting up MongoDB
- Setting up LogAnalyzer
This guide has been created with rsyslog 7.3.6 on ubuntu 12.04 LTS and Adiscon LogAnalyzer 3.6.3. All additional packages, services or applications where the latest version at that time.
Before you begin
Please note, there are many ways to distribute rsyslog. But, make sure that the platform you build rsyslog on is the same platform as where it should be used. You cannot build rsyslog on CentOS and use it on Ubuntu. The differences between the platforms are just to big. The same applies to different versions of the same platform. When building on a older platform and using it on a newer version, this may work, but with restrictions. Whereas building on a newer version and using it on a older version will probably not work at all. So, if you build rsyslog from Source and want to use it on another machine, make sure the platform is the same.
Step 1 – Setting up rsyslog
We need to setup rsyslog first. Not only do we need the core functionality, but several additional modules. For this case we want to receive the syslog messages via TCP, thus we need imtcp. For processing we need first mmjsonparse and ommongodb. Your configure should look like this.
./configure --prefix=/usr --enable-imtcp --enable-mmjsonparse --enable-ommongodb
The module mmjsonparse will be needed to verify and parse the @cee messages. Ommongodb will be used to write into the MongoDB. After the configure and installation, we can create our config. The config for our case looks like this:
module(load="imtcp") module(load="mmjsonparse") module(load="ommongodb")
input(type="imtcp" port="13514" Ruleset="mongodb")
template(name="mongodball" type="subtree" subtree="$!")
ruleset(name="mongodb") { action(type="mmjsonparse") if $parsesuccess == "OK" then { set $!time = $timestamp; set $!sys = $hostname; set $!procid = $syslogtag; set $!syslog_fac = $syslogfacility; set $!syslog_sever = $syslogpriority; set $!pid = $procid; action(type="ommongodb" server="127.0.0.1" db="logs" collection="syslog" template="mongodball") } }
As always, we first load the modules. The next part is the input. We need to receive tcp via imtcp. Please note, that we directly bind the input to a ruleset. The third part of the configuration is a template. We need it later when writing to MongoDB. Since we will automatically transform our @cee-message into json, we can use a subtree template. The template itself is basically the root of the subtree.
The last and most important part is the ruleset. Here all of our work is done. First, all messages are run through the mmjsonparse module. This will not only verify if we received a valid json message, but also transforms all the values into a json subtree. If the parsing was successful, we need to set several variables for the subtree. Information that is delivered in the syslog header will not be parsed into the subtree by mmjsonparse automatically. Thus we have to set subtree variables with the values of some default properties like timestamp, hostname and so on. After that we have basically all information from the complete syslog message in the subtree format. Finally a last action is needed. We need to write our log messages to MongoDB. In this example, MongoDB is installed on the same machine. We want to use the db “logs” and as collection we want to use “syslog”. And we use our subtree template to define the format that is written to MongoDB. Thus, all our parsed variables are stored separately. If we do not use this template, the @cee message gets written as it is into the msg field in MongoDB. But this is not what we want. We want all variables to be available separately.
That is basically it for rsyslog. You can now save the configuration and restart rsyslog. Though it won’t be able to be useful yet. We still need to install MongoDB.
Step 2 – Install MongoDB
Making a basic install for MongoDB is rather easy. Simply install the following packages:
mongodb mongodb-server php-pecl-mongo libmongo-client libglib2.0-dev
Please note, that package names may vary for different distributions.
After we have installed the packages, the MongoDB Server is already ready. By default, it is not secured by a user or password. Refer to the MongoDB manual for more information. Databases and collections (equivalent to tables) are created by rsyslog and do not need to be defined with the mongo shell. We will stick with the default setup to keep it simple.
Step 3 – Installing Adiscon LogAnalyzer
To run Adiscon LogAnalyzer, you need a webserver with PHP. The easiest way is to use apache2 and php5. To be able to access the MongoDB, we need to install an additional package. Run the following command
sudo pecl install mongo
You might need to install the package php-pear first, if it hasn’t been installed already with PHP.
After that, we need to put the following line into the file php.ini.
extension=mongo.so
Remember to restart your webserver after making changes to the php.ini. Without a lot of configuration, this should aready work.
We can now install Adiscon LogAnalyzer. Download the latest version from the Adiscon LogAnalyzer Download page and install it as it is described in the documentation.
The only difference we need to make is when setting up the log source in step 4.5. You need to set at least the following as shown in the screenshot:
Source Type: MongoDB Native Table Type: MongoDB Database Name: logs Database Tablename: syslog Database user: <clear this field>
The User and Password is of course needed, if you set it in your MongoDB setup.
After you have finished the installation of Adiscon LogAnalyzer, you should now be seeing the LogAnalyzer overview and the log messages in a table view (if you have already stored them in MongoDB). Now if you click on a specific message, you get to see the detail view of the log message.
Click on the picture for a bigger size
As you can see, you get a list of dynamic fields. These fields where already sent in @cee format from the original source and were parsed by rsyslog and mmjsonparse and finally they were automatically filled into the MongoDB.
With this setup, you are independent of a fixed database structure. Fields are inserted dynamically into the database as they are available and they are dynamically display by Adiscon LogAnalyzer as well.
Main Advantages of rsyslog v7 vs. v5
Why rsyslog V7:
- greatly improved configuration language – the new language is much more intuitive than the legacy format. It will also prevent some typical mistakes simply by not permitting these invalid constructs. Note that legacy format is still fully supported (and you can of course do the same mistakes if you use legacy format).
- greatly improved execution engine – with nested if/then/else constructs as well as the capability to modify variables during processing.
- full support for structured logging and project lumberjack / CEE – this includes everything from being able to create, interpret and handle JSON-based structured log messages, including the ability to normalize legacy text log messages.
- more plugins – like support for MongoDB, HDFS, and ElasticSearch as well as for the kernel’s new structured logging system.
- higher performance – many optimizations all over the code, like 5 to 10 times faster execution time for script-based filters, enhanced multithreaded TCP input plugin, DNS cache and many more.
Of course, there are many more improvements. This list contains just the most important ones. For full details, check the file ChangeLog.
How to use rsyslog and MongoDB
rsyslog is since version 6.1.3 capable of using a output module called “ommongodb” to write log messages to mongodb. To achieve this, several packages are needed:
mongodb mongodb-server php-pecl-mongo libmongo-client libglib2.0-dev
These and of course rsyslog are needed. The packages are easily installed via a package manager. And the installation steps for rsyslog are the usual ones. Except for the “Configure”. Please make sure to include “–enable-ommongodb“. And the other features you plan to use of course.
./configure --libdir=/usr/lib --sbindir=/usr/sbin --enable-ommongodb make make install
After this, you need to make some adjustments to the rsyslog.conf. You need to load the module of course and configure an action:
module (load="ommongodb") *.* action(type="ommongodb" server="127.0.0.1")
In this case we send everything to the local MongoDB server. With the default settings, this should work directly. No username or password is needed if it isn’t configured in MongoDB. And the “db” and “collection” will automatically be created. The default database that rsyslog will use is “syslog” and the default name for the collection is “log“. These can be changed by various parameters.
To review what is written to the database, simply open a command line and enter “mongo“. This is to open the MongoDB Shell. You now need to change the database that is used with
use syslog
That way all further commands will be used on the database that rsyslog uses. Next we let the shell show us the entries:
db.log.find()
Please make sure to use the exact command. Of course there are a lot of other commands that you may find useful. Or information on how to secure the database. Please refer to the MongoDB documentation for that.
rsyslog 6.3.8 (v6-devel) released
This is an important new release of the rsyslog v6 devel branch. Among others, it is the version that supports many of the new things done for project lumberjack, better cee-enhanced syslog support as well as a much-improved mongodb driver. The release also contains numerous other enhancements as well as bug-fixes. Please note that part of the feature set is still experimental and unstable in a sense that interfaces or similar thing “taken for granted” (e.g. the default MongoDB schema) may change in later releases.
As a preliminary measure, please make sure you are using the most current versions of libee (0.4.1), libestr (0.1.2) and liblognorm (0.3.4).
ChangeLog:
http://www.rsyslog.com/changelog-for-6-3-8-v6-devel/
Download:
http://www.rsyslog.com/rsyslog-6-3-8-v6-devel/
As always, feedback is appreciated.
Best regards,
Florian Riedl