rsyslog -devel packages are being removed soon
If you use rsyslog’s devel packages on your system, you will receive errors soon. Be sure to read the complete posting to avoid trouble!
As part of rsyslog’s new release schedule and version naming, devel releases will no longer be named according to the “normal” numbering scheme. This also means that the previous “devel” branches will disappear, as git master branch now is the always-current devel version.
Keep on your mind that we previously had a release cycle of 3 to 9 month for a new feature to appear in a stable version. That was because new feature releases were only done when a complete devel turnaround was done, and relatively many new features were added. For this reason, some people opted to run devel versions in production, and thus needed specific tarballs (and packages) for them.
With the new six week release cycle, we get new features rather quickly into the stable builds. So it usually should be no problem to wait for the next stable to use that recently-implemented new feature. As such, there is no need any longer for special devel releases, and thus no need for devel tarballs and packages.
Well… almost. One thing we would like to have is a “daily devel version”. The idea is that if the testbench runs are OK, a new tarball and a set of packages is generated automatically and posted to a special archive. In general, that archive should receive an update once a day. So people really interested in the [b]leading edge can simply install from that daily package archive — and report bugs quickly, so helping the development process. Unfortunately, time is precious and we don’t know when and if we can setup the required automation. Most probably not before January 2015, and how it works out then needs to be seen.
In the interim, we will begin to delete the -devel packages. The old -devel tarballs will remain available, at least for the time being. The problem with -devel packages is that folks may have set their system to use the -devel repro. If we would just keep it as is, those systems would never again receive any updates, neither security-releated nor others, simply because -devel versions no longer exist in the way they were. That would pose a potentially big security risk. As such, we will delete the -devel content, and begin to do so early next week. If you use the -devel packages, be sure to switch the v8-stable instead.
Ubuntu Repository legacy
The Adiscon Ubuntu Repository has been setup to provide support for the latest rsyslog versions on Ubuntu including support for necessary third party packages. Please note that the Ubuntu Repository is open for testing at the moment, and contains the latest RSyslog version. Currently supported Ubuntu Versions:
- Ubuntu 14.04 (Trusty)
- Ubuntu 13.10 (Saucy Salamander)
- Ubuntu 12.04 (Precise)
Ubuntu 12.10 (Quantal) (Not Updated anymore)
Currently supported RSyslog Branches:
- v8-devel
- v8-stable
v7-devel (Not Updated anymore)- v7-stable
The new packages are based in the original and latest Ubuntu 12 rsyslog packages, so in most cases an simple sudo apt-get update && sudo apt-get upgrade will be enough to update rsyslog. Please note that these packages are currently experimental. Use at your own risk.
Add our PPA Repository using the classic method:
- Install our PGP Key into your apt system
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com AEF0CF8E gpg --export --armor AEF0CF8E | sudo apt-key add -
- Edit your /etc/apt/sources.list and add these lines to the end replace v8-devel with v7-stable or v8-stable if you want to use the RSYSLOG stable packages.
For Ubuntu (12.04 LTS) Precise # Adiscon repository deb http://ppa.launchpad.net/adiscon/v8-devel/ubuntu precise main deb-src http://ppa.launchpad.net/adiscon/v8-devel/ubuntu precise main
For Ubuntu (13.10) Saucy # Adiscon repository deb http://ppa.launchpad.net/adiscon/v8-devel/ubuntu saucy main deb-src http://ppa.launchpad.net/adiscon/v8-devel/ubuntu saucy main
- Once done perform these commands to update your apt cache and install the latest RSYSLOG version
sudo apt-get update && sudo apt-get upgrade
- If you receive a message like this while upgrading follow these steps below:
The following packages have been kept back: rsyslog 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.sudo apt-get install rsyslog
We highly appriciate any feedback or bug reports.
Performance Tuning&Tests for the Elasticsearch Output
Original post: Rsyslog 8.1 Elasticsearch Output Performance by @Sematext
Version 8 brings major changes in rsyslog’s core – see Rainer’s presentation about it for more details. Those changes should give outputs better performance, and the Elasticsearch one should benefit a lot. Since we’re using rsyslog and Elasticsearch in Sematext‘s own log analytics product, Logsene, we had to take the new version for a spin.
The Weapon and the Target
For testing, we used a good-old i3 laptop, with 8GB of RAM. We generated 20 million logs, sent them to rsyslog via TCP and from there to Elasticsearch in the Logstash format, so they can get explored with Kibana. The objective was to stuff as many events per second into Elasticsearch as possible.
Rsyslog Architecture Overview
In order to tweak rsyslog effectively, one needs to understand its architecture, which is not that obvious (although there’s an ongoing effort to improve the documentation). The gist of it its architecture represented in the figure below.
- you have input modules taking messages (from files, TCP/UDP, journal, etc.) and pushing them to a main queue
- one or more main queue threads take those events and parse them. By default, they parse syslog formats (RFC-3164, RFC-5424 and various derivatives), but you can configure rsyslog to use message modifier modules to do additional parsing (e.g. CEE-formatted JSON messages). Either way, this parsing generates structured events, made out of properties
- after parsing, the main queue threads push events to the action queue. Or queues, if there are multiple actions and you want to fan-out
- for each defined action, one or more action queue threads takes properties from events according to templates, and makes messages that would be sent to the destination. In Elasticsearch’s case, a template should make Elasticsearch JSON documents, and the destination would be the REST API endpoint
There are two more things to say about rsyslog’s architecture before we move on to the actual test:
- you can have multiple independent flows (like the one in the figure above) in the same rsyslog process by using rulesets. Think of rulesets as swim-lanes. They’re useful for example when you want to process local logs and remote logs in a completely separate manner
- queues can be in-memory, on disk, or a combination called disk-assisted. Here, we’ll use in-memory because they’re the fastest. For more information about how queues work, take a look here
Configuration
To generate messages, we used tcpflood, a small and light tool that’s part of rsyslog’s testbench. It generates messages and sends them over to the local syslog via TCP.
Rsyslog took received those messages with the imtcp input module, queued them and forwarded them to Elasticsearch 0.90.7, which was also installed locally. We also tried with Elasticsearch 1.0 and got the same results (see below).
The flow of messages in this test is represented in the following figure:
The actual rsyslog config is listed below (in the new configuration format). It can be tuned further (for example by using the multithreaded imptcp input module), but we didn’t get significant improvements in this particular scenario.
module(load="imtcp") # TCP input module module(load="omelasticsearch") # Elasticsearch output module input(type="imtcp" port="13514") # where to listen for TCP messages main_queue( queue.size="1000000" # capacity of the main queue queue.dequeuebatchsize="1000" # process messages in batches of 1000 and move them to the action queues queue.workerthreads="2" # 2 threads for the main queue ) # template to generate JSON documents for Elasticsearch in Logstash format template(name="plain-syslog" type="list") { constant(value="{") constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339") constant(value="\",\"host\":\"") property(name="hostname") constant(value="\",\"severity\":\"") property(name="syslogseverity-text") constant(value="\",\"facility\":\"") property(name="syslogfacility-text") constant(value="\",\"syslogtag\":\"") property(name="syslogtag" format="json") constant(value="\",\"message\":\"") property(name="msg" format="json") constant(value="\"}") } action(type="omelasticsearch" template="plain-syslog" # use the template defined earlier searchIndex="test-index" bulkmode="on" # use the Bulk API queue.dequeuebatchsize="5000" # ES bulk size queue.size="100000" # capacity of the action queue queue.workerthreads="5" # 5 workers for the action action.resumeretrycount="-1" # retry indefinitely if ES is unreachable )
You can see from the configuration that:
- both main and action queues have a defined size in number of messages
- both have number of threads that deliver messages to the next step. The action needs more because it has to wait for Elasticsearch to reply
- moving of messages from the queues happens in batches. For the Elasticsearch output, the batch of messages is sent through the Bulk API, which makes queue.dequeuebatchsize effectively the bulk size
Results
We started with default Elasticsearch settings. Then we tuned them to leave rsyslog with a more significant slice of the CPU. We monitored the indexing rate with SPM for Elasticsearch. Here are the average results over 20 million indexed events:
- with default Elasticsearch settings, we got 8,000 events per second
- after setting Elasticsearch up more production-like (5 second refresh interval, increased index buffer size, translog thresholds, etc), and the throughput went up to average of 20,000 events per second
- in the end, we went berserk and used in-memory indices, updated the mapping to disable any storing or indexing for any field, to have Elasticsearch do as little work as possible and make room for rsyslog. Got an average of 30,000 events per second. In this scenario, rsyslog was using between 1 and 1.5 of the 4 virtual CPU cores, with tcpflood using 0.5 and Elasticsearch using from 2 to 2.5
Conclusion
20K EPS on a low-end machine with production-like configuration means Elasticsearch is quick at indexing. This is very good for logs, where you typically have lots of messages being generated, compared to how often you search.
If you need some tool to ship your logs to Elasticsearch with minimum overhead, rsyslog version 8 may well be your best bet.
Related posts:
librelp 1.2.4
librelp 1.2.4 [download]
This version of librelp is a correction for the API/ABI change in v1.2.3. Everything else stays the same.
———————————————————————-
Version 1.2.4 – 2014-03-17
– correct API/ABI change in 1.2.3
My reasoning was flawed, and we could run into problems with
apps that required the new version but could not detect that an
older one was installed.
Thanks to Michael Biebl for pointing this out.
What we have done is:
– revert back to previous state (return void)
* relpSrvEnableTLS();
* relpSrvEnableTLSZip();
These functions are now deprecated.
– introduce new functions that return a state
* relpSrvEnableTLS2();
* relpSrvEnableTLSZip2();
Version 1.2.4 – 2014-03-17
– correct API/ABI change in 1.2.3
My reasoning was flawed, and we could run into problems with
apps that required the new version but could not detect that an
older one was installed.
Thanks to Michael Biebl for pointing this out.
What we have done is:
– revert back to previous state (return void)
* relpSrvEnableTLS();
* relpSrvEnableTLSZip();
These functions are now deprecated.
– introduce new functions that return a state
* relpSrvEnableTLS2();
* relpSrvEnableTLSZip2();
sha256sum: cf4f26f9a75991eedf3eaf414280c8da3532c38e619a465d23008c714f5c1cf1
Output to Elasticsearch in Logstash format (Kibana-friendly)
Original post: Recipe rsyslog+Elasticsearch+Kibana by @Sematext
In this post you’ll see how you can take your logs with rsyslog and ship them directly to Elasticsearch (running on your own servers, or the one behind Logsene’s Elasticsearch API) in a format that plays nicely with Logstash. So you can use Kibana to search, analyze and make pretty graphs out of them.
This is especially useful when you have a lot of servers logging [a lot of data] to their syslog daemons and you want a way to search them quickly or do statistics on the logs. You can use rsyslog’s Elasticsearch output to get your logs into Elasticsearch, and Kibana to visualize them. The only challenge is to get your rsyslog configuration right, so your logs end up where Kibana is expecting them. And this is exactly what we’re doing here.
Getting all the ingredients
Here’s what you’ll need:
- a recent version of rsyslog (v8+ is recommended for best performance, although the Elasticsearch output is available since 6.4.0). You can download and compile it yourself, or you can get it from the RHEL/CentOS or Ubuntu repositories
- the Elasticsearch output plugin for rsyslog. If you compile rsyslog from sources, you’ll need to add the –enable-elasticsearch parameter to the configure script. If you use the repositories, just install the rsyslog-elasticsearch package
- Elasticsearch :). You have a DEB and a RPM there, which should get you started in no time. If you choose the tar.gz archive, you might find the installation instructions useful
- Kibana 3 and a web server to serve it. There are installation instructions on the GitHub page. To get started quickly, you can try the tar.gz archive from the download page that gets you Elasticsearch, too
Then, you’ll probably need to edit config.js to change the Elasticsearch host name from “localhost” to the actual FQDN of the host that’s running Elasticsearch. This applies even if Kibana is on the same machine as Elasticsearch. “localhost” only works if your browser is on the same machine as Elasticsearch, because Kibana talks to Elasticsearch directly from your browser.
Finally, you can serve the Kibana page with any HTTP server you prefer. If you want to get started quickly, you can try SimpleHTTPServer, which should be embedded to any recent Python, by running this command from the “kibana” directory:
python -m SimpleHTTPServer
Putting them all together
Kibana is, by default, expecting Logstash to send logs to Elasticsearch. So “putting them all together” here means “configuring rsyslog to send logs to Elasticsearch in the same manner Logstash does”. And Logstash, by default, has some particular ways when it comes to naming the indices and formatting the logs:
- indices should be formatted like logstash-YYYY.MM.DD. You can change the pattern Kibana is looking for, but we won’t do that here
- logs must have a timestamp, and that timestamp must be stored in the @timestamp field. It’s also nice to put the message part in the message field – because Kibana shows it by default
To satisfy the requirements above, here’s a rsyslog configuration that should work for sending your local syslog logs to Elasticsearch in a Logstash/Kibana-friendly way:
module(load="imuxsock") # for listening to /dev/log
module(load="omelasticsearch") # for outputting to Elasticsearch
# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")After restarting rsyslog, you can go to http://host-serving-Kibana:8000/ in your browser and start searching and graphing your logs:
More tips
Now that you got the essentials working, here are some tips that might help you go even further with your centralized logging setup:
- you might not want to put the new rsyslog and omelasticsearch on all your servers. In this case you can forward them over the network to a central rsyslog that has omelasticsearch, and push your logs to Elasticsearch from there. Some information on forwarding logs via TCP can be found here and here
- you might want rsyslog to buffer your logs (in memory, on disk, or some combination of the two), in case Elasticsearch is not available for some reason. Buffering will also help performance, as you can send messages in bulks instead of one by one. There’s a reference on buffers with rsyslog&omelasticsearch here
- you might want to parse JSON-formatted (CEE) syslog messages. If you’re using them, check our earlier post on the subject: JSON logging with rsyslog and Elasticsearch
You can also hook rsyslog up to a log analytics service like Logsene, by either shipping logs via omelasticsearch or by sending them via UDP/TCP/RELP syslog protocols.
Guides
Basic Configuration
This first section will describe some basic configuration. Here you will not find complete configurations, but snippets on how to use different modules correctly and some description on how they are working.
Newbie guide to RSYSLOG
Installing RSYSLOG from RPM
Sending messages with tags larger than 32 characters
Using the syslog receiver module
Using the Text File Input Module
Some core configs
This section contains some basics. Things, that are used ever and ever again. It also contains some more in-depth description of what rsyslog does and why. It is recommended to at least briefly read through this part before going to more complex scenarios.
TLS secured syslog via RELP
Action’s with directives
Writing specific messages to a file and discarding them
Sending Messages to a Remote Syslog Server
Receiving Messages from a Remote System
Using a different log Format for all Files
Discarding unwanted messages
More complex scenarios
RSYSLOG and ElasticSearch
Parsing JSON (CEE) Logs and Sending them to Elasticsearch
Using TLS with RELP
Log normalization for different formats
Using MongoDB with RSYSLOG and LogAnalyzer
Normalizing Cisco ASA messages
Receiving CEE enhanced syslog in RSYSLOG
Storing and forwarding remote messages
How to write to a local socket?
Storing Messages from a Remote System into a specific File
Integration with “standard” syslogd
Enterprise Support
| Options | Enterprise Support | Extended Enterprise Support | Unlimited Enterprise Support |
| Max. hours | 20 | 40 | 40 |
| Email support |  | | |
| Phone support | | | |
| Response time | 1 – 2 business days | 24 hours on business days | 24 hours on business days |
| Cost | 1999 EURO | 3899 EURO | 19999 EURO |
| Number of installations | 5 | 5 | Unlimited |
Purchase RSYSLOG support directly from the source. This contract provides priority email support and phone support. It is a great option if you need to provide proof of software support in your organization. This contract provides
- All contracts run for a period of one year.
- unlimited email support tickets during validity
- fixes for current RSYSLOG releases
- advise on how to implement rsyslog in the best possible way.
All contracts run for a period of one year.
Under this contract, fixes for RSYSLOG releases will be provided / created, assuming it is possible to do that with the code base in question.
Change Log
---------------------------------------------------------------------------
Version 8.1.6 [devel] 2014-01-20
- omfile: permit to set global defaults for action parameters
Thanks to Nathan Brown for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/23
- add capability to escape control characters in the C way of doing it
adds new global parameter "parser.escapeControlCharactersCStyle"
Thanks to Nathan Brown for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/13
- parser global parameters can now be set using RainerScript global()
Thanks to Nathan Brown for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/23
- omprog: guard program-to-be-executed against CTL-C
This can frequently happen in debug mode, where rsyslog is terminated
by ctl-c. In any case, SIGINT is not meant to control the child process,
so it should be blocked.
- omprog bugfix: parameter "forceSingleInstance" is NOT mandatory
- add new jsonr property replacer option
Thanks to Nathan Brown for the patch.
- added external plugin interface
- ommongodb: add authentication support (untested)
Thanks to JT for the patch.
See also: https://github.com/rsyslog/rsyslog/pull/17
---------------------------------------------------------------------------
Version 8.1.5 [devel] 2014-01-24
- omprog: ability to execute multiple program instances per action
It can now execute one program instance per worker thread. This is
generally a very good thing the have performance wise. Usually, this
should cause no problems with the invoked program. For that reason,
we have decided to make this the default mode of operation. If not
desired, it can be turned off via the 'forceSingleInstance="on"'
action parameter.
CHANGE OF BEHAVIOUR: previous versions did always execute only one
instance per action, no matter how many workers were active. If
your program has special needs, you need to change your configuration.
- imfile now supports inotify (but must be explicitely turned on)
- imfile no longer has a limit on number of monitored files
- added ProcessInternalMessages global system parameter
This permits to inject rsyslog status messages into *another* main
syslogd or the journal.
- new dependency: liblogging-stdlog (for submitting to external logger)
- bugfix: imuxsock input parameters were not accepted
due to copy&paste error. Thanks to Andy Goldstein for the fix.
---------------------------------------------------------------------------
Version 8.1.4 [devel] 2014-01-10
- add exec_template() RainerScript function
- imrelp: support for TCP KEEPALIVE added
- bumped librelp dependency to 1.2.2 to support new KEEPALIVE feature
- Add directives for numerically specifying GIDs/UIDs
The already present directives (FileOwner, FileGroup, DirOwner,
DirGroup) translate names to numerical IDs, which depends on the user
information being available during rsyslog's startup. This can fail if
the information is obtained over a network or from a service such as
SSSD. The new directives provide a way to specify the numerical IDs
directly and bypass the lookup.
Thanks to Tomas Heinrich for the patch.
- bugfix: action commitTransaction() processing did not properly handle
suspended actions
- bugfix: omelasticsearch fail.es stats counter was improperly maitained
---------------------------------------------------------------------------
Version 8.1.3 [devel] 2013-12-06
THIS VERSION CAN BE CONSIDERED A "NORMAL" DEVEL RELEASE. It's no longer
highly experimental. This assertion is based on real-world feedback.
- changes to the strgen module interface
- new output module interface for transactional modules
- performance improvements
* reduced number of malloc/frees due to further changes to the
output module interface
* reduced number of malloc/frees during string template processing
We now re-use once allocated string template memory for as long
as the worker thread exists. This saves us from doing new memory
allocs (and their free counterpart) when the next message is
processed. The drawback is that the cache always is the size of
the so-far largest message processed. This is not considered a
problem, as in any case a single messages' memory footprint should
be far lower than that of a whole set of messages (especially on
busy servers).
* used variable qualifiers (const, __restrict__) to hopefully help
the compiler generate somewhat faster code
- failed action detection more precisely for a number of actions
If an action uses string parameter passing but is non-transactional
it can be executed immediately, giving a quicker indicatio of
action failure.
- bugfix: limiting queue disk space did not work properly
* queue.maxdiskspace actually initializes queue.maxfilesize
* total size of queue files was not checked against
queue.maxdiskspace for disk assisted queues.
Thanks to Karol Jurak for the patch.
---------------------------------------------------------------------------
Version 8.1.2 [experimental] 2013-11-28
- support for liblognorm1 added - results in performance improvements
Thanks to Pavel Levshin for his work in this regard.
- support for jemalloc added via --enable-jemalloc
Thanks to Pavel Levshin for suggesting jemalloc
Note that build system is experimental at this stage.
- queue defaults have changed
* high water mark is now dynamically 90% of queue size
* low water makr is now dynamically 70% of queue size
* queue.discardMark is now dynamically 98% of queue size
* queue.workerThreadMinimumMessage set to queue.size / num workers
For queues with very low queue.maxSize (< 100), "emergency" defaults
will be used.
- bugfix: disk queues created files in wrong working directory
if the $WorkDirectory was changed multiple times, all queues only
used the last value set.
- bugfix: legacy directive $ActionQueueWorkerThreads was not honored
- bugfix: mmrfc5424addhmac: "key" parameter was not properly processed
---------------------------------------------------------------------------
Version 8.1.1 [experimental] 2013-11-19
- bugfix: STOP/discard(~) was mostly NOT honored
This lead to execution of config code that was not meant to be executed.
- bugfix: memory leak on worker thread termination
- bugfix: potential segfault in omfile under heavy load
Thanks to Pavel Levshin for alerting us.
- bugfix: mmsequence: instance mode did not work
Thanks to Pavel Levshin for the patch
- bugfix: segfault on startup when certain script constructs are used
e.g. "if not $msg ..."
- omhiredis: now supports v8 output module interface and works again
Thanks to Pavel Levshin for the patch
- mmaudit: now supports v8 output module interface and work again
- bugfix: potential abort on startup in debug mode
This depends on template type being used. The root cause was a
non-necessary debug output, which were at the wrong spot (leftover from
initial testing).
Thanks to Pavel Levshin for alerting us and providing a patch
proposal.
---------------------------------------------------------------------------
Version 8.1.0 [experimental] 2013-11-15
- rewritten core engine for higher performance and new features
In detail:
* completely rewritten rule execution engine
* completely changed output module interface
* remodelled output module interface
* enabled important output modules to support full concurrent
operation
The core engine has been considerably changed and must be considered
experimental at this stage. Note that it does not yet include all
features planned for v8, but is close to this goal. In theory, the
engine should perform much better, especially on complex configurations
and busy servers. Most importantly, actions instances can now be called
concurrently from worker threads and many important output modules
support multiple concurrent action instances natively.
- module omruleset is no longer enabled by default.
Note that it has been deprecated in v7 and been replaced by the "call"
statement. Also, it can still be build without problems, the option must
just explicitely be given.
------------------------------Downloads
Apart from installing via tarball or git, rsyslog is also available in package form on some distributions. All packages currently maintained by Adiscon are listed below. For more detailed information about the packages and how to install rsyslog with them can be found at the specific package page.
- Ubuntu
- RHEL/CentOS
- Debian
rsyslog and ElasticSearch
by Micah Yoder, originally published on rackspace. Minor changes through Adiscon.
There is a clear benefit to being able to aggregate logs from various servers and services into one place and be able to search them for any sort of arbitrary event. Traditional syslog can aggregate logs, but aggregating events from them sometimes involves grep and convoluted regular expressions. Logging structured data to a database makes a lot of sense. rsyslog and ElasticSearch can do that, but figuring out how to get it to work from the rsyslog documentation can be difficult. Let’s start from the beginning.
First, you need the newest stable rsyslog, 7.4.x. The older 7.2 won’t cut it. You need the plug-ins mmnormalize and omelasticsearch, both of which are available from rsyslog’s yum repositories for RHEL/CentOS. mmnormalize requires some packages from EPEL so that will need to be added to the server as well.
Let’s walk through this from the beginning, starting with the most basic rsyslog configuration:
module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog") # provides kernel logging support (previously done by rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
*.* /var/log/messages
This simply loads the necessary modules for basic logging operation, sets the normal log format, and sends everything to /var/log/messages. With this if you issue this command:
# logger 'hello world!'
You get this in /var/log/messages:
Mar 27 14:16:47 micahsyslog root: hello world!
There are several parts here: the date/time, the hostname, the syslog tag or program (root), and the message. Another example:
# service sshd restart
yields:
Mar 27 14:19:27 micahsyslog sshd[23295]: Received signal 15; terminating.
Mar 27 14:19:27 micahsyslog sshd[16602]: Server listening on 0.0.0.0 port 22.
Mar 27 14:19:27 micahsyslog sshd[16602]: Server listening on :: port 22.
By the way, if we remove the ActionFileDefaultTemplate directive, we get the default line with an ISO time, which is nicer for automatic processing but a bit less readable for humans:
2013-03-27T21:51:29.139796+00:00 micahsyslog root: Hello world #2!
That is good to know, but actually irrelevant for putting data into ElasticSearch. If you also log to text files, you can decide whether to use the ISO time or the more human readable time.
Now let’s try to get it to normalize messages from the log! For this example we will need rsyslog running with ActionFileDefaultTemplate directive (keep it in for now, but you can remove it later if you want).
Create a rulebase file, which will be used by liblognorm. liblognorm is a library written by Rainer Gerhards, the author of rsyslog, which parses log messages according to a pattern and extracts variables from them. We can create the file as /etc/rsyslog.d/rules.rb and put in these contents:
prefix=%date:date-rfc3164% %hostname:word% %tag:word%
rule=user: new user: name=%user:char-to:,%, UID=%uid:number%, GID=%gid:number%, home=%home:char-to:,%, shell=%shell:word%
rule=test: Hello %x:number%
Now if we log something:
# logger “Hello 55”
We should be able to get something like this:
# lognormalizer -r rsyslog.d/rules.rb -t test < /var/log/messages
[cee@115 event.tags="test" x="55" tag="root:" hostname="micahsyslog" date="Mar 27 22:53:30"]
See what happened? The lognormalizer command, which uses liblognorm, was able to pull out the number in the log as variable “x” according to our “test” rule.
A short description of the rule base is in order. The prefix line contains a prefix that will be applied to all following rules. A rule is defined by a line beginning with the word “rule”, followed by an “=” sign, an optional tag, then a “:”. A space should then follow – but the space is actually part of the rule! There will be a space after the prefix, so it needs to be there.
Documentation on the rulebase file can be found here: http://www.liblognorm.com/files/manual/index.html Then navigate to How to use liblognorm → Rulebase. This will show you all the directives you can use.
Now we need to get the data into ElasticSearch. First we need to install it, which is very simple (at least for a single test node). Just extract its tarball, change into the base directory, and run:
# bin/elasticsearch -f
The -f causes it to run in the foreground so you can see what is going on.
We then add this to rsyslog.conf:
module(load="mmnormalize")
module(load="omelasticsearch")
template(name="test" type="subtree" subtree="$!")
set $!fac = $syslogfacility;
set $!host = $hostname;
set $!sev = $syslogseverity;
set $!time = $timereported;
set $!tgen = $timegenerated;
set $!tag = $syslogtag;
set $!prog = $programname;
action(type="omelasticsearch" server="localhost" template="test")
So what is all this? First we load the modules to normalize the log messages and for export to ElasticSearch. These should be placed near the top of your rsyslog.conf file with any other includes.
The template statement tells which part of the CEE data should be sent to ElasticSearch. Here a brief explanation of the data structure is necessary.
JSON (the format for CEE logging data and the native format for ElasticSearch) is a hierarchical data structure. The root of the data structure in rsyslog is $! – something like the ‘{}’ in a JSON document. Assigning to $!data1 would put a value into the “data1” top level JSON element {“data”: “…”}. Trees can be built. If you assign to $!tree1!child1, you will get a JSON document like this: {“tree1: {“child1”: “…”}}
This rsyslog document lists the available properties that you can use to populate these CEE variables.
As of version 7.5.8, 7.6.0 and 8.1.4, not only the standard rsyslog properties and constants can be assigned to these variables. There is now a way to assign the result of an rsyslog template, which allows more complex values. That is what you need to put a sortable timestamp into the document. More information can be found in the guide to use “set variable and exec_template“.
As this was not available at the time of writing I needed to get around this limitation (sort of), so I created an ElasticSearch index with a timestamp. Here is the command I used:
curl -XPUT http://elasticsearch.hostname:9200/logs -d '{"mappings":{"events":{"_timestamp":{"enabled": true, "store": "yes"},"prog":{"store":"yes"},"host":{"store":"yes"}}}}'
This also “stores” the host and prog syslog fields, which should help with querying based on the host or program. This will create an index called “logs”. ElasticSearch by default inserts events into the “system” index so you will want to specify the index name in your omelasticsearch line in rsyslog.conf (or one of its includes):
action(type="omelasticsearch" server="elasticsearch.hostname" template="cee_template_name" searchIndex="logs" bulkmode="on")
We also enabled bulk mode, which allows rsyslog to send many events at once to ElasticSearch. This will greatly improve performance.
Now you can query it!
curl 'http://elasticsearch.hostname:9200/logs/_search?pretty=1&fields=_source,_timestamp&size=100' -d '{"sort":{"_timestamp": "desc"}}' | less
The “pretty=1” and piping it to less are simply there to pretty print the JSON result and make it easy for you to browse the data set. A real search program, of course, would not turn on pretty mode and the code would directly consume the JSON.
You can add parameters after the ‘size=100’ (which, of course, says to return the 100 most recent results).
- &q=word will search the logs for that word in any field.
- &q=prog:name will search for ‘name’ in the syslog program field. Examples: postfix, sudo, crontab
- &q=host:hostname will find events for that hostname. Just use the base hostname, not a FQDN.
And that should get you started! Hopefully this is helpful.