Module Name: omelasticsearch
Author:Rainer Gerhards <rgerhards@adiscon.com>
Available since:6.4.0+
Description:
This module provides native support for logging to Elasticsearch.
Action Parameters:
Host name or IP address of the Elasticsearch server. Defaults to
“localhost”
HTTP port to connect to Elasticsearch. Defaults to 9200
`Elasticsearch
index <http://www.elasticsearch.org/guide/appendix/glossary.html#index>`_ to send your logs to. Defaults to “system”
Whether the string provided for searchIndex should be taken as a
template. Defaults to “off”, which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let’s assume you define a template named “date-days” containing “%timereported:1:10:date-rfc3339%”. Then, with dynSearchIndex=”on”, if you say searchIndex=”date-days”, each log will be sent to and index named after the first 10 characters of the timestamp, like “2013-03-22”.
`Elasticsearch
type <http://www.elasticsearch.org/guide/appendix/glossary.html#type>`_ to send your index to. Defaults to “events”
Like dynSearchIndex, it allows you to specify a
template for searchType, instead of a static string.
By default, an indexing operation returns after all `replica
shards <http://www.elasticsearch.org/guide/appendix/glossary.html#replica_shard>`_ have indexed the document. With asyncrepl=”on” it will return after it was indexed on the primary shard only - thus trading some consistency for speed.
Send events over HTTPS instead of HTTP. Good for when you have
Elasticsearch behind Apache or something else that can add HTTPS.
How long Elasticsearch will wait for a primary shard to be available
for indexing your log before sending back an error. Defaults to “1m”.
This is the JSON document that will be indexed in Elasticsearch. The
resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:
$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
Which will produce this sort of documents (pretty-printed here for readability):
{
"message": " this is a test message",
"fromhost": "test-host",
"facility": "user",
"priority": "info",
"timereported": "2013-03-12T18:05:01.344864+02:00",
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
}
The default “off” setting means logs are shipped one by one. Each in
its own HTTP request, using the Index API. Set it to “on” and it will use Elasticsearch’s Bulk API to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the dequeue batch size. More information about queues can be found here.
Specifying a string here will index your logs with that string the
parent ID of those logs. Please note that you need to define the parent field in your mapping for that to work. By default, logs are indexed without a parent.
Using the same parent for all the logs sent in the same action is
quite unlikely. So you’d probably want to turn this “on” and specify a template that will provide meaningful parent IDs for your logs.
If you have basic HTTP authentication deployed (eg: through the
elasticsearch-basic plugin), you can specify your user-name here.
Password for basic authentication.
Samples:
The following sample does the following:
module(load="omelasticsearch")
*.* action(type="omelasticsearch")
The following sample does the following:
module(load="omelasticsearch")
template(name="testTemplate"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}")
}
*.* action(type="omelasticsearch"
server="myserver.local"
serverport="9200"
template="testTemplate"
searchIndex="test-index"
searchType="test-type"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
[`rsyslog.conf overview <rsyslog_conf.html>`_\ ] [`manual
index <manual.html>`_] [rsyslog site]
This documentation is part of the rsyslog project.
Copyright © 2008-2012 by `Rainer
Gerhards <http://www.gerhards.net/rainer>`_ and Adiscon. Released under the ASL 2.0.