The rocket-fast system for log processing
We have released rsyslog 8.21.0. This release is mostly for maintenance. There was a big change to how internal messages are handled. These are no longer logged via the internal bridge, but via the syslog() API call. For regular users, this should make not too much difference. Additionaly, the TLS syslog error messages have been […]
Original post: Recipe: rsyslog + Kafka + Logstash by @Sematext This recipe is similar to the previous rsyslog + Redis + Logstash one, except that we’ll use Kafka as a central buffer and connecting point instead of Redis. You’ll have more of the same advantages: rsyslog is light and crazy-fast, including when you want it […]
Original post: Recipe: Apache Logs + rsyslog (parsing) + Elasticsearch by @Sematext This recipe is about tailing Apache HTTPD logs with rsyslog, parsing them into structured JSON documents, and forwarding them to Elasticsearch (or a log analytics SaaS, like Logsene, which exposes the Elasticsearch API). Having them indexed in a structured way will allow you […]
Original post: Recipe: rsyslog + Redis + Logstash by @Sematext OK, so you want to hook up rsyslog with Logstash. If you don’t remember why you want that, let me give you a few hints: Logstash can do lots of things, it’s easy to set up but tends to be too heavy to put on […]
Originally posted on the Sematext blog: Monitoring rsyslog’s Performance with impstats and Elasticsearch If you’re using rsyslog for processing lots of logs (and, as we’ve shown before, rsyslog is good at processing lots of logs), you’re probably interested in monitoring it. To do that, you can use impstats, which comes from input module for process […]
Original post: Rsyslog 8.1 Elasticsearch Output Performance by @Sematext Version 8 brings major changes in rsyslog’s core – see Rainer’s presentation about it for more details. Those changes should give outputs better performance, and the Elasticsearch one should benefit a lot. Since we’re using rsyslog and Elasticsearch in Sematext‘s own log analytics product, Logsene, we […]
Queue For each queue inside the system its own set of statistics counters is created. If there are multiple action (or main) queues, this can become a rather lengthy list. The stats record begins with the queue name (e.g. “main Q” for the main queue; ruleset queues have the name of the ruleset they are […]
Version 8.1.2 [devel] 2013-11-28 support for liblognorm1 added – results in performance improvements Thanks to Pavel Levshin for his work in this regard. support for jemalloc added via –enable-jemalloc Thanks to Pavel Levshin for suggesting jemalloc queue defaults have changed high water mark is now dynamically 90% of queue size low water makr […]
Some users report that they do not receive all impstats log records or that these log records are delayed. The common thing about these questions is that those user tend to have very large main message queues. By default, impstats is run in-band, which means that it’s messages are submitted to the main message queue […]
This guide will tell you, how to quickly protect your disk queue through encryption. So you can be sure that unauthorized persons can’t read your queue. Please note that we only use the “disk” queue format in this guide to show you the encrypted files but normally we recommend you to use the “LinkedList” queue […]
