The rocket-fast system for log processing
Original post: Monitoring rsyslog with Kibana and SPM by @Sematext A while ago we published this post where we explained how you can get stats about rsyslog, such as the number of messages enqueued, the number of output errors and so on. The point was to send them to Elasticsearch (or Logsene, our logging SaaS, […]
Original post: Recipe: Apache Logs + rsyslog (parsing) + Elasticsearch by @Sematext This recipe is about tailing Apache HTTPD logs with rsyslog, parsing them into structured JSON documents, and forwarding them to Elasticsearch (or a log analytics SaaS, like Logsene, which exposes the Elasticsearch API). Having them indexed in a structured way will allow you […]
We have released rsyslog 8.13.0. This release sports a big number of changes. While most are bugfixes, there are also some additions to existing functionality, most notably the enhancements for ZeroMQ and Redis modules. For more details, please take a look at the Changelog. ChangeLog: http://www.rsyslog.com/changelog-for-8-13-0-v8-stable/ Download: http://www.rsyslog.com/downloads/download-v8-stable/ As always, feedback is appreciated. Best regards, […]
We have released rsyslog 8.7.0. Version 8.7.0 contains various improvements and additions to a wide array of modules, like imfile, imptcp, improvements to RainerScript and mmnormalize (thanks to Singh Janmejay) and a couple of other improvements. But, the biggest addition is the new omkafka module that now allows direct writing to Apache Kafka. This release […]
Version 8.7.0 [v8-stable] 2015-01-13 add message metadata “system” to msg object this permits to store metadata alongside the message imfile: add support for “filename” metadata this is useful in cases where wildcards are used imptcp: make stats counter names consistent with what imudp, imtcp uses added new module “omkafka” to support writing to Apache Kafka […]
This release offers some important bug fixes and exciting new features. Most importantly, imudp can now use multiple input threads for even higher UDP input throughput. And imfile now longer has a hardcoded limit on the number of files that can be monitored. You can now monitor as many files as you like. Support for […]
Version 7.5.5 [devel] 2013-10-16 imfile: permit to monitor an unlimited number of files imptcp: add “defaultTZ” input parameter imudp: support for multiple receiver threads added imudp: add “dfltTZ” input config parameter bugfix: memory leak in mmnormalize bugfix: mmutf8fix did not properly handle invalid UTF-8 at END of message if the very last character sequence was […]
This time, we want to parse log messages received from a Cisco ASA for specific parts. The log messages will be parsed by liblognorm and then they will be written into a specific file resembling the sent code. This guide has been tested with v7.3.4 of rsyslog. Things to think about We basically need a […]
When trying to normalize log messages via liblognorm and mmnormalize, you need to create a rulebase first. The rulebase is usually a representation of message formats. Due to the format of these rules, it is necessary to be cautious. Some messages and rule necessities could possibly cause confusion to the configuration interpreter. This mainly applies […]
Log normalization is simple, but has its quirks. A common pitfall is syslog message format as induced by RFC3164. Let’s look at a common case: A log message has been sent to rsyslog. The message itself had no irregular characters. But, the message that should have been parsed by mmnormalize now has a leading space […]