LogAnalyzer: Facility and Severity is missing
Question: I use a logfile of rsyslog as source type in LogAnalyzer, everything is good but the facility and severity information tabs of the messages are missing, like in this screenshot.
Answer: The solution is rather simple, your current file template does not contain syslog priority or facility. Kindly switch to RSYSLOG_SyslogProtocol23Format which is RFC5424 format which contains the required information. You can use the template for a single action or you can use it as the default template. Below you can find a example for both cases:
Please note that this example only apply to a single logfile:
mail.* /var/log/maillog;RSYSLOG_SyslogProtocol23Format
This is the example for a default template:
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
Please note that you need to change the logfile type to RSyslog Format23 in your Loganalyzer logstream sources as well. You can do that by editing the “config.php” of LogAnalyzer.
Open the “config.php” with your favourite editor and add the following line to the correct source:
$CFG[‘Sources’][‘Source1’][‘LogLineType’] = “Syslog23”;
Afterwards the configuration should look like this.
Don’t forget to save the changes. Now you can refresh the Webpanel of LogAnalyzer and then you should see the facility and severity missing information tabs, like in this screenshot.
Using MongoDB with rsyslog and LogAnalyzer
In this scenario we want to receive cee-formatted messages from a different system with rsyslog, store the messages with MongoDB and then display the stored messages with Adiscon LogAnalyzer. This is a very common use-case. Please read through the complete guide before starting.
We will split this guide in 3 main parts. These parts can consist of several steps.
- Setting up rsyslog
- Setting up MongoDB
- Setting up LogAnalyzer
This guide has been created with rsyslog 7.3.6 on ubuntu 12.04 LTS and Adiscon LogAnalyzer 3.6.3. All additional packages, services or applications where the latest version at that time.
Before you begin
Please note, there are many ways to distribute rsyslog. But, make sure that the platform you build rsyslog on is the same platform as where it should be used. You cannot build rsyslog on CentOS and use it on Ubuntu. The differences between the platforms are just to big. The same applies to different versions of the same platform. When building on a older platform and using it on a newer version, this may work, but with restrictions. Whereas building on a newer version and using it on a older version will probably not work at all. So, if you build rsyslog from Source and want to use it on another machine, make sure the platform is the same.
Step 1 – Setting up rsyslog
We need to setup rsyslog first. Not only do we need the core functionality, but several additional modules. For this case we want to receive the syslog messages via TCP, thus we need imtcp. For processing we need first mmjsonparse and ommongodb. Your configure should look like this.
./configure --prefix=/usr --enable-imtcp --enable-mmjsonparse --enable-ommongodb
The module mmjsonparse will be needed to verify and parse the @cee messages. Ommongodb will be used to write into the MongoDB. After the configure and installation, we can create our config. The config for our case looks like this:
module(load="imtcp") module(load="mmjsonparse") module(load="ommongodb")
input(type="imtcp" port="13514" Ruleset="mongodb")
template(name="mongodball" type="subtree" subtree="$!")
ruleset(name="mongodb") {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
set $!time = $timestamp;
set $!sys = $hostname;
set $!procid = $syslogtag;
set $!syslog_fac = $syslogfacility;
set $!syslog_sever = $syslogpriority;
set $!pid = $procid;
action(type="ommongodb" server="127.0.0.1" db="logs" collection="syslog" template="mongodball")
}
}As always, we first load the modules. The next part is the input. We need to receive tcp via imtcp. Please note, that we directly bind the input to a ruleset. The third part of the configuration is a template. We need it later when writing to MongoDB. Since we will automatically transform our @cee-message into json, we can use a subtree template. The template itself is basically the root of the subtree.
The last and most important part is the ruleset. Here all of our work is done. First, all messages are run through the mmjsonparse module. This will not only verify if we received a valid json message, but also transforms all the values into a json subtree. If the parsing was successful, we need to set several variables for the subtree. Information that is delivered in the syslog header will not be parsed into the subtree by mmjsonparse automatically. Thus we have to set subtree variables with the values of some default properties like timestamp, hostname and so on. After that we have basically all information from the complete syslog message in the subtree format. Finally a last action is needed. We need to write our log messages to MongoDB. In this example, MongoDB is installed on the same machine. We want to use the db “logs” and as collection we want to use “syslog”. And we use our subtree template to define the format that is written to MongoDB. Thus, all our parsed variables are stored separately. If we do not use this template, the @cee message gets written as it is into the msg field in MongoDB. But this is not what we want. We want all variables to be available separately.
That is basically it for rsyslog. You can now save the configuration and restart rsyslog. Though it won’t be able to be useful yet. We still need to install MongoDB.
Step 2 – Install MongoDB
Making a basic install for MongoDB is rather easy. Simply install the following packages:
mongodb mongodb-server php-pecl-mongo libmongo-client libglib2.0-dev
Please note, that package names may vary for different distributions.
After we have installed the packages, the MongoDB Server is already ready. By default, it is not secured by a user or password. Refer to the MongoDB manual for more information. Databases and collections (equivalent to tables) are created by rsyslog and do not need to be defined with the mongo shell. We will stick with the default setup to keep it simple.
Step 3 – Installing Adiscon LogAnalyzer
To run Adiscon LogAnalyzer, you need a webserver with PHP. The easiest way is to use apache2 and php5. To be able to access the MongoDB, we need to install an additional package. Run the following command
sudo pecl install mongo
You might need to install the package php-pear first, if it hasn’t been installed already with PHP.
After that, we need to put the following line into the file php.ini.
extension=mongo.so
Remember to restart your webserver after making changes to the php.ini. Without a lot of configuration, this should aready work.
We can now install Adiscon LogAnalyzer. Download the latest version from the Adiscon LogAnalyzer Download page and install it as it is described in the documentation.
The only difference we need to make is when setting up the log source in step 4.5. You need to set at least the following as shown in the screenshot:
Source Type: MongoDB Native Table Type: MongoDB Database Name: logs Database Tablename: syslog Database user: <clear this field>
The User and Password is of course needed, if you set it in your MongoDB setup.
After you have finished the installation of Adiscon LogAnalyzer, you should now be seeing the LogAnalyzer overview and the log messages in a table view (if you have already stored them in MongoDB). Now if you click on a specific message, you get to see the detail view of the log message.
Click on the picture for a bigger size
As you can see, you get a list of dynamic fields. These fields where already sent in @cee format from the original source and were parsed by rsyslog and mmjsonparse and finally they were automatically filled into the MongoDB.
With this setup, you are independent of a fixed database structure. Fields are inserted dynamically into the database as they are available and they are dynamically display by Adiscon LogAnalyzer as well.
Using rsyslog mmnormalize module effectively with Adiscon LogAnalyzer
Using the mmnormalize module in rsyslog is a bit complicated at first. We want to describe in this article how to set up the basic components for using log normalization. In addition to that we will show how to configure these components so messages will be split into pieces of information. These pieces of information should then be written into a database for review with Adiscon LogAnalyzer.
This guide has been tested with rsyslog v5.8.0 and liblognorm 0.3, libee 0.3.
The goal of this guide is to have a setup, that will have a message parsed by the normalizing tool, put some content of the message into specific properties. These properties will then be filled into a special database format, which will should be reviewed by Adiscon LogAnalyzer.
For using normalization we need the following:
- rsyslog
- liblognorm
- libee
- libestr
In the further process of the article we need additional elements:
- apache webserver with PHP5
- mysql database (usually with phpmyadmin)
- Adiscon LogAnalyzer
Step 1: Setting up rsyslog and log normalization
First of all we need to setup rsyslog for log normalization. So before installing rsyslog, we will install liblognorm, libee and libestr. They can be installed according to this guide. rsyslog can now be installed. We assume you have downloaded and extracted a tarball from the rsyslog download page. Change into the directory you installed rsyslog in. Now use the following commands to setup rsyslog correctly:
./configure --libdir=/lib --sbindir=/sbin --enable-mysql --enable-mmnormalize make make install
If everything is correct, the installation procedure should complete successfully. We can now start configuring rsyslog itself. We need a configuration that looks like this:
$ModLoad immark $ModLoad imuxsock $ModLoad imklog $ModLoad mmnormalize $ModLoad ommysql.so $ModLoad imudp.so
$UDPServerRun 514
$mmnormalizeUseRawMSG 1 $mmnormalizeRuleBase /rsyslog/rulebase.rb *.* :mmnormalize:
$template database,"insert into normalized (date, uhost, msgnumber, protocol, ipin, ipout, portin, portout)
values ('%$!date%', '%$!uhost%', '%$!msgnumber%', '%$!protocol%', '%$!ipin%', '%$!ipout%', '%$!portin%',
'%$!portout%')",SQL*.* :ommysql:172.19.3.17,syslog, test, test;database
That is all for our rsyslog config. Looks pretty complicated right now. Basically, we load all necessary modules at the top. After that we start the UDP syslog server. It is needed to receive the messages. The next 3 lines are the parameters to initiate the normalization of messages. We declare, that the raw message should be used. Our rulebase for the normalization lies in the rsyslog directory (this path has to be changed if your directory lies somewhere else). And after that, we tell rsyslog to use normalization on all messages. The next line describes the template for the processed message. In the end, there should be a sql insert statement that puts all the parsed variables into their corresponding fields in the table “normalized”. The last line is finally the action that makes rsyslog write all messages (the ones created by the template – the sql statement – into a remote database.
After the configuration, we still need to setup a rulebase. This is done in a separate file. For our example, the rulebase should be the following file: /rsyslog/rulebase.rb
The file should look like this:
rule=:%date:date-rfc3164% %uhost:word% %tag:word% %notused:char-to:x3a%: %msgnumber:char-to:x3a%: access-list inside_access_in permitted %protocol:word% inside/%ipin:ipv4%(portin:number%) -> outside/%ipout:ipv4%(portout:number%) %notused2:char-to:]%]
The rule is basically one line. It might be shown otherwise here due to restrictions of the webdesign. It is basically a format of a message. The different parameters of a rule are shown in a different guide. The rule we have here should resemble the following message:
May 16 07:23:09 BHG-FW : %ASA-4-106100: access-list inside_access_in permitted tcp inside/10.200.22.183(2969) -> outside/67.192.232.82(80) hit-cnt 1 first hit [0x48e9c345, 0x386bad81]
If you want to have multiple messages, where the format differs, you need multiple rules as well of course. The rules must be as precise as possible to resemble the message. If a message does not fit any listed rules, it will not be processed further. Something else that needs to be pointed out, is to keep the rules variable enough as well. Like in our example, there are some parts that will be the same for every message of this kind. Other parts might be with different content. And even if we do not need the content further, it should be put into a variable. Else the message might again not fit to the rule.
Step 2: Setting up the database
We suppose, that you already have a server with a database and webserver installed. The installation of the components has to be made according to the instructions given by the manufacturer of the software. Therefore we cannot give any examples for that.
But we need a specific database scheme for our example here. So we need to show this at least. As you have seen before, we have some specific parts of the message filled into properties. These properties should be written to the database. So here is the basic SQL statement to create the table according to our needs:
CREATE TABLE normalize ( ID int unsigned not null auto_increment primary key, date datetime NULL, host varchar(255) NULL, msgnumber varchar(20), protocol varchar(60) NULL, ipin varchar(60) NULL, ipout varchar(60) NULL, portin int NULL, portout int NULL )
You can execute this statement as you like. It is currently designed for a MySQL database, so you might need to change some bits if you are using a different database.
3. Using Adiscon LogAnalyzer with this database
Adiscon LogAnalyzer can be used to review the data from this database. Installation of Adiscon LogAnalyzer is shown here. Please note, that we will need the admin center. So please think of creating a user database when installing.
Point your browser to your Adiscon LogAnalyzer installation. Now we need to go to the admin center. There we have to set some parts to fit our custom format.
Edit Fields
First, we need to add some Fields. We need to do this, so we can use the custom fields in our database with LogAnalyzer. By default, the list of fields only reflects basically the MonitorWare Database Scheme. When clicking on Fields in the Admin Center, a list of the currently available fields will be shown:
By clicking on Add new Field, we can create a new Field.
We need to create 7 new fields only, though we have 8 custom fields in the table. Since date is the same, we can use the already formatted field. So we only need to create the fields for host, msgnumber, protocol, ipin, ipout, portin and portout.
Basically, the Field details should look like this:
To finally create the Field, click on the button “Add new Field”. Now the list should appear again with the newly created Field. Repeat this step for the other fieldnames as well.
Edit DBMappings
In conjunction with the Fields which are only for the internal use in Adiscon LogAnalyzer, we need to create a custom database mapping. Therefore go to DBMappings in the Admin Center. You will see a list of the currently available database mappings.
Click on Add new Database Mapping:
Here we need to tell Adiscon LogAnalyzer, which Field we created depends on which database field. Give your database mapping a name first. After that, choose the Fields we need from the dropdown menu and click on “Add Field Mapping into list”. The final step will be to enter the database field names into the list. It should now look like this now:
Finally click on “Add new Database Mapping”. This will save the mapping and get you back to the list of DBMappings.
Edit Views
The next step we need to adjust is the Views. In Views you can configure, what LogAnalyzer should show. This is related to the data that is stored in the database. Basically, a View should represent the kind of logs that are stored. For example if you use the View for Windows Event Logs, but have a database where Linux syslog is stored, many Fields will be shown as empty, because they are not filled like from Windows Event Logs. Therefore we need a custom view.
You will get there by clicking on Views in the Admin Center.
There are already pre-configured Views for Windows EventLog, Syslog and Webserver Logs. We need a completely different View though. A new View can be configured by clicking on “Add new View” at the bottom of the list.
You need to give your view a name. If you want, you can restrict the use of this view to certain users or groups, but we will skip that for now. The most important part is to select the Fields that should be displayed. This is done at “Configured Columns”. Before clicking on “Add new View” it should look like this:
After clicking the button, the new View should appear in our list.
Edit Sources
Finally, we need to create a Source. When installing Adiscon LogAnalyzer, you can already configure a Source. For our example, we need to create another Source. Therefore go to Sources in the Admin Center.
You will see a list of the configured Sources. It currently holds one Source. By clicking on Add new Source you can create another one.
Basically, we need to insert a Source Name. If you want, you can also create a description. Change the Source Type to MYSQL Native. You can also select a default View. Choose our lognorm View we created earlier. No more general options need to be set. If you want, you can again restrict the source to a user or group.
We still need to change the database Type options. As you can see, the fields have changed by setting the Source Type to MYSQL Native. As table type choose the lognorm type we created before. Insert the details as your database needs them. The complete form should look like this now:
Finish the new Source by clicking on Add new Source. It should now appear in the list.
Final Thoughts
Though this scenario seems very complex it shows in the end how easy some things can be afterwards. This setup shows exactly, how different products from the Adiscon product line can work together. And we have a good example for how normalizing works.