rsyslog

The rocket-fast system for log processing

Output to Elasticsearch in Logstash format (Kibana-friendly)

Original post: Recipe rsyslog+Elasticsearch+Kibana by @Sematext

In this post you’ll see how you can take your logs with rsyslog and ship them directly to Elasticsearch (running on your own servers, or the one behind Logsene’s Elasticsearch API) in a format that plays nicely with Logstash. So you can use Kibana to search, analyze and make pretty graphs out of them.

This is especially useful when you have a lot of servers logging [a lot of data] to their syslog daemons and you want a way to search them quickly or do statistics on the logs. You can use rsyslog’s Elasticsearch output to get your logs into Elasticsearch, and Kibana to visualize them. The only challenge is to get your rsyslog configuration right, so your logs end up where Kibana is expecting them. And this is exactly what we’re doing here.

Getting all the ingredients

Here’s what you’ll need:

  • a recent version of rsyslog (v8+ is recommended for best performance, although the Elasticsearch output is available since 6.4.0). You can download and compile it yourself, or you can get it from the RHEL/CentOS or Ubuntu repositories
  • the Elasticsearch output plugin for rsyslog. If you compile rsyslog from sources, you’ll need to add the –enable-elasticsearch parameter to the configure script. If you use the repositories, just install the rsyslog-elasticsearch package
  • Elasticsearch :). You have a DEB and a RPM there, which should get you started in no time. If you choose the tar.gz archive, you might find the installation instructions useful
  • Kibana 3 and a web server to serve it. There are installation instructions on the GitHub page. To get started quickly, you can try the tar.gz archive from the download page that gets you Elasticsearch, too

Then, you’ll probably need to edit config.js to change the Elasticsearch host name from "localhost" to the actual FQDN of the host that’s running Elasticsearch. This applies even if Kibana is on the same machine as Elasticsearch. "localhost" only works if your browser is on the same machine as Elasticsearch, because Kibana talks to Elasticsearch directly from your browser.

Finally, you can serve the Kibana page with any HTTP server you prefer. If you want to get started quickly, you can try SimpleHTTPServer, which should be embedded to any recent Python, by running this command from the "kibana" directory:

python -m SimpleHTTPServer

Putting them all together

Kibana is, by default, expecting Logstash to send logs to Elasticsearch. So "putting them all together" here means "configuring rsyslog to send logs to Elasticsearch in the same manner Logstash does". And Logstash, by default, has some particular ways when it comes to naming the indices and formatting the logs:

  • indices should be formatted like logstash-YYYY.MM.DD. You can change the pattern Kibana is looking for, but we won’t do that here
  • logs must have a timestamp, and that timestamp must be stored in the @timestamp field. It’s also nice to put the message part in the message field – because Kibana shows it by default

To satisfy the requirements above, here’s a rsyslog configuration that should work for sending your local syslog logs to Elasticsearch in a Logstash/Kibana-friendly way:

module(load="imuxsock")             # for listening to /dev/log
module(load="omelasticsearch") # for outputting to Elasticsearch
# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
  type="list") {
    constant(value="logstash-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="logstash-index"
    dynSearchIndex="on")

After restarting rsyslog, you can go to http://host-serving-Kibana:8000/ in your browser and start searching and graphing your logs:

kibana-screenshot

More tips

Now that you got the essentials working, here are some tips that might help you go even further with your centralized logging setup:

  • you might not want to put the new rsyslog and omelasticsearch on all your servers. In this case you can forward them over the network to a central rsyslog that has omelasticsearch, and push your logs to Elasticsearch from there. Some information on forwarding logs via TCP can be found here and here
  • you might want rsyslog to buffer your logs (in memory, on disk, or some combination of the two), in case Elasticsearch is not available for some reason. Buffering will also help performance, as you can send messages in bulks instead of one by one. There’s a reference on buffers with rsyslog&omelasticsearch here
  • you might want to parse JSON-formatted (CEE) syslog messages. If you’re using them, check our earlier post on the subject: JSON logging with rsyslog and Elasticsearch

You can also hook rsyslog up to a log analytics service like Logsene, by either shipping logs via omelasticsearch or by sending them via UDP/TCP/RELP syslog protocols.

9 thoughts on “Output to Elasticsearch in Logstash format (Kibana-friendly)

  1. I can’t find rsyslog-elasticsearch*.rpm under http://rpms.adiscon.com/v8-stable/epel-5/x86_64/RPMS/. I also poked around and could not find a suitable version. Then I downloaded the src package but it complains on ./configure –enable-elasticsearch (well there are more problems like ./configure also complains on libestr >= 0.1.9 although I have installed this and the devel package. I’m not very good at creating packages). I also checked if the latest version includes it but omelasticsearch.so is not on my system.
    Would you be able to create a current elasticsearch package?
    Thanks.

  2. Is it possible to grep pattern in property(name="hostname") and replace the name as the pattern we want to assign?
    Thanks

  3. Hi,

    Can anyone please let me know how logstash indexing the field @timestamp in documents with time zone format? I need to derive the same logic where feeding data to elasticsearch from java api.

  4. Hi There, thanks for such an informative article, it has helped me a lot.
    I am trying to make my rsyslog send logs to elasticsearch and then view it via kibana. I am using nginx as my server, i could not see any logs the kibana dashboad is emapty. Is there something I am missing here? Thanks for your help in advance. If you could tell me how to debug or verify my configuration using curl commands its that would be greatly helpful.

  5. A second sexy tip. A two-for.

    You no longer have to install Kibana separately if you have installed the Marvel plugin. Marvel is built on Kibana. When you access the normal Marvel URL at http://elasticsearch:9200/_plugin/marvel/ it will forward you to the full URL. Change the ending marvel.overview.json to guided.json and begin to build out your page.

    When you name and save the page, it will forward you to the new URL, which you may bookmark.

  6. More helpful hints!

    The above text sends severity and facility as text fields, if you would like numerical fields so that you can easily sort for "warnings and above" as an example, change the following:


    template(name="plain-syslog"
    type="list") {
    constant(value="{")
    constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"") property(name="hostname")
    constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
    constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
    constant(value="\",\"message\":\"") property(name="msg" format="json")
    constant(value="\"}")
    }

    to


    template(name="plain-syslog"
    type="list") {
    constant(value="{")
    constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
    constant(value="\",\"host\":\"") property(name="hostname")
    constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
    constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
    constant(value="\",\"severity-num\":") property(name="syslogseverity")
    constant(value=",\"facility-num\":") property(name="syslogfacility")
    constant(value=",\"tag\":\"") property(name="syslogtag" format="json")
    constant(value="\",\"message\":\"") property(name="msg" format="json")
    constant(value="\"}")
    }

    This will keep the text within severity and facility, and put the numbers in the severity-num and facility-num field data. Not the removal of the quotes around severity-num and facility-num fields, this makes sure they are processed as a number.

Comments are closed.