Use this documentation with care! It describes
the outdated version 7, which was actively
developed around 2014 and is considered dead by the
rsyslog team.
This documentation reflects the latest update of the
v7-stable branch. It describes the 7.6.8 version, which was never
released. As such, it contains some content that
does not apply to any released version.
To obtain the doc that properly matches your installed
v7 version, obtain the doc set from your distro. Each
version of rsyslog contained the version that exactly
matches it.
As general advise, it is strongly suggested to
upgrade to the current version supported by the rsyslog
project. The current version can always be found on
the right-hand side info box on the rsyslog web site.
Note that there is only limited rsyslog community support
available for the outdated v7 version (officially we do not
support it at all, but we usually are able to answer simple
questions). If you need to stick with v7, it probably is
best to ask your distribution for support.
omelasticsearch: Elasticsearch Output Module¶
Module Name: omelasticsearch
Author:Rainer Gerhards <rgerhards@adiscon.com>
Available since:6.4.0+
Description:
This module provides native support for logging to Elasticsearch.
Action Parameters:
- server Host name or IP address of the Elasticsearch server. Defaults to “localhost”
- serverport
- HTTP port to connect to Elasticsearch. Defaults to 9200
- searchIndex Elasticsearch index to send your logs to. Defaults to “system”
- dynSearchIndex<on/off> Whether the string provided for searchIndex should be taken as a template. Defaults to “off”, which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let’s assume you define a template named “date-days” containing “%timereported:1:10:date-rfc3339%”. Then, with dynSearchIndex=”on”, if you say searchIndex=”date-days”, each log will be sent to and index named after the first 10 characters of the timestamp, like “2013-03-22”.
- searchType Elasticsearch type to send your index to. Defaults to “events”
- dynSearchType <on/off> Like dynSearchIndex, it allows you to specify a template for searchType, instead of a static string.
- asyncrepl<on/off> By default, an indexing operation returns after all replica shards have indexed the document. With asyncrepl=”on” it will return after it was indexed on the primary shard only - thus trading some consistency for speed.
- timeout How long Elasticsearch will wait for a primary shard to be available for indexing your log before sending back an error. Defaults to “1m”.
- template This is the JSON document that will be indexed in Elasticsearch. The resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:
$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
Which will produce this sort of documents (pretty-printed here for readability):
{
"message": " this is a test message",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
}
- bulkmode<on/off> The default “off” setting means logs are shipped one by one. Each in its own HTTP request, using the Index API. Set it to “on” and it will use Elasticsearch’s Bulk API to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the dequeue batch size. More information about queues can be found here.
- parent Specifying a string here will index your logs with that string the parent ID of those logs. Please note that you need to define the parent field in your mapping for that to work. By default, logs are indexed without a parent.
- dynParent<on/off> Using the same parent for all the logs sent in the same action is quite unlikely. So you’d probably want to turn this “on” and specify a template that will provide meaningful parent IDs for your logs.
- uid If you have basic HTTP authentication deployed (eg: through the elasticsearch-basic plugin), you can specify your user-name here.
- pwd Password for basic authentication.
Samples:
The following sample does the following:
- loads the omelasticsearch module
- outputs all logs to Elasticsearch using the default settings
module(load="omelasticsearch")
*.* action(type="omelasticsearch")
The following sample does the following:
- loads the omelasticsearch module
- defines a template that will make the JSON contain the following
properties
- RFC-3339 timestamp when the event was generated
- the message part of the event
- hostname of the system that generated the message
- severity of the event, as a string
- facility, as a string
- the tag of the event
- outputs to Elasticsearch with the following settings
- host name of the server is myserver.local
- port is 9200
- JSON docs will look as defined in the template above
- index will be “test-index”
- type will be “test-type”
- activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300
- retry indefinitely if the HTTP request failed (eg: if the target server is down)
module(load="omelasticsearch")
template(name="testTemplate"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}")
}
action(type="omelasticsearch"
server="myserver.local"
serverport="9200"
template="testTemplate"
searchIndex="test-index"
searchType="test-type"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
This documentation is part of the rsyslog project. Copyright © 2008-2014 by Rainer Gerhards and Adiscon. Released under the ASL 2.0.