The rocket-fast system for log processing
Original post: Monitoring rsyslog with Kibana and SPM by @Sematext A while ago we published this post where we explained how you can get stats about rsyslog, such as the number of messages enqueued, the number of output errors and so on. The point was to send them to Elasticsearch (or Logsene, our logging SaaS, […]
Original post: Recipe: rsyslog + Kafka + Logstash by @Sematext This recipe is similar to the previous rsyslog + Redis + Logstash one, except that we’ll use Kafka as a central buffer and connecting point instead of Redis. You’ll have more of the same advantages: rsyslog is light and crazy-fast, including when you want it […]
In the first part, we will explain the new RSYSLOG v8 engine, its motivation and its benefits. Learn, for example, why writing to Elasticsearch is much faster with the new engine. We will describe the tuning parameters vital for making best use of the new features. In the second part we will explain how to […]
Version 8.3.0 [v8-devel] 2014-04-10 new plugin for anonymizing credit card numbers Thanks to Peter Slavov for providing the code. external message modification modules are now supported They are bound via the new native module “mmexternal”. Also, a sample skeleton for an external python message modification module has been added. new $jsonmesg property with JSON representation […]
Plugin – imuxsock This plugin maintains a global statistics with the following properties: submitted – total number of messages submitted for processing since startup ratelimit.discarded – number of messages discarded due to rate limiting ratelimit.numratelimiters – number of currently active rate limiters (small data structures used for the rate limiting logic) Back to statistics counter […]
Plugin – imudp This plugin maintains statistics for each listener and for each worker thread. The listener statistic is named starting with “imudp”, followed followed by the listener IP, a colon and port in parenthesis. For example, the counter for a listener on port 514 (on all IPs) with no set name is called “imudp(*:514)”. […]
Plugin – imtcp This plugin maintains statistics for each listener. The statistic is named after the given input name (or “imtcp” if none is configured), followed by the listener port in parenthesis. For example, the counter for a listener on port 514 with no set name is called “imtcp(514)”. The following properties are maintained for […]
Plugin – imptcp This plugin maintains statistics for each listener. The statistic is named “imtcp” , followed by the bound address, listener port and IP version in parenthesis. For example, the counter for a listener on port 514, bound to all interfaces and listening on IPv6 is called “imptcp(*/514/IPv6)”. The following properties are maintained for […]
Plugin – impstats (rsyslog 7.5.3+) The impstats plugin gathers some internal statistics. They have different names depending on the actual statistics. Obviously, they do not relate to the plugin itself but rather to a broader object – most notably the rsyslog process itself. The “resource-usage” counter maintains process statistics. They base on the getrusage() system […]
Plugin – omfile (rsyslog 7.3.6+) This plugin maintains statistics for each dynafile cache. Dynafile cache performance is critical for overall system performance, so reviewing these counters on a busy system (especially one experiencing performance problems) is advisable. The statistic is named “dynafile cache”, followed by the template name used for this dynafile action. The following […]