rsyslog 8.1.3 (v8-devel) released
We have just released 8.1.3 of the v8-devel branch.
Note that this release can be considered a “normal” devel version, with moderate risk associated to it. Experience in the past weeks suggest so. Special thanks to Pavel Levshin for all his work and tests. Still note that there is a considerably higher risk running the devel version than the stable one.
ChangeLog:
http://www.rsyslog.com/changelog-for-8-1-3-v8-devel/
Download:
http://www.rsyslog.com/rsyslog-8-1-3-v8-devel/
Feedback is *very much* appreciated.
Best regards,
Florian Riedl
Changelog for 8.1.3 (v8-devel)
Version 8.1.3 [devel] 2013-12-06
THIS VERSION CAN BE CONSIDERED A “NORMAL” DEVEL RELEASE. It’s no longer
highly experimental. This assertion is based on real-world feedback.
- changes to the strgen module interface
- new output module interface for transactional modules
- performance improvements
- reduced number of malloc/frees due to further changes to the output module interface
- reduced number of malloc/frees during string template processing
We now re-use once allocated string template memory for as long as the worker thread exists. This saves us from doing new memory allocs (and their free counterpart) when the next message is processed. The drawback is that the cache always is the size of the so-far largest message processed. This is not considered a problem, as in any case a single messages’ memory footprint should be far lower than that of a whole set of messages (especially on busy servers). - used variable qualifiers (const, __restrict__) to hopefully help the compiler generate somewhat faster code
- failed action detection more precisely for a number of actions
If an action uses string parameter passing but is non-transactional it can be executed immediately, giving a quicker indicatio of action failure. - bugfix: limiting queue disk space did not work properly
- queue.maxdiskspace actually initializes queue.maxfilesize
- total size of queue files was not checked against queue.maxdiskspace for disk assisted queues.
Thanks to Karol Jurak for the patch.
rsyslog 8.1.3 (v8-devel)
Download file name: rsyslog 8.1.3 (devel)
rsyslog 8.1.3 (devel)
sha256 hash: e2e5e12500b6d2a8be80558b2f71da
Author: Rainer Gerhards (rgerhards@adiscon.com)
Version: 8.1.3 File size: 2.608 MB
rsyslog statistic counter Queues
Queue
For each queue inside the system its own set of statistics counters is created. If there are multiple action (or main) queues, this can become a rather lengthy list. The stats record begins with the queue name (e.g. “main Q” for the main queue; ruleset queues have the name of the ruleset they are associated to, action queues the name of the action).
- size – currently active messages in queue
- enqueued – total number of messages enqueued into this queue since startup
- maxsize – maximum number of active messages the queue ever held
- full – number of times the queue was actually full and could not accept additional messages
- discarded.full – number of messages discarded because the queue was full
- discarded.nf – number of messages discarded because the queue was nearly full. Starting at this point, messages of lower-than-configured severity are discarded to save space for higher severity ones.
rsyslog statistic counter plugin imuxsocks
Plugin – imuxsock
This plugin maintains a global statistics with the following properties:
- submitted – total number of messages submitted for processing since startup
- ratelimit.discarded – number of messages discarded due to rate limiting
- ratelimit.numratelimiters – number of currently active rate limiters (small data structures used for the rate limiting logic)
rsyslog statistic counter plugin imudp
Plugin – imudp
This plugin maintains statistics for each listener and for each worker thread.
The listener statistic is named starting with “imudp”, followed followed by the listener IP, a colon and port in parenthesis. For example, the counter for a listener on port 514 (on all IPs) with no set name is called “imudp(*:514)”.
If an “inputname” is defined for a listener, that inputname is used instead of “imudp” as statistic name. For example, if the inputname is set to “myudpinut”, that corresponding statistic name in above case would be “myudpinput(*:514)”. This has been introduced in 7.5.3.
The following properties are maintained for each listener:
- submitted – total number of messages submitted for processing since startup
The worker thread (in short: worker) statistic is named “imudp(wX)” where “X” is the worker thread ID, which is an monotonically increasing integer starting at 0. This means the first worker will have the name “imudp(w0”), the second “imudp(w1)” and so on. Note that workers are all equal. It doesn’t really matter which worker processes which messages, so the actual worker ID is not of much concern. More interesting is to check how the load is spread between the worker. Also note that there is no fixed worker-to-listener relationship: all workers process messages from all listeners.
Note: worker thread statistics are available starting with rsyslog 7.5.5.
The following properties are maintained for each worker thread:
- called.recvmmsg – number of recvmmsg() OS calls done
- called.recvmsg – number of recvmsg() OS calls done
- msgs.received – number of actual messages received
Note: usually either one of “called.recvmmsg” or “called.recvmsg” is non-zero. This is because these are alternative kernel interfaces and the one to be used is selected based on kernel capabilities. However, there are two counters, as the system call type used has important performance implications, and so we thought this information needs to be exposed.
On systems supporting recvmmsg, the quotient msgs.received / caled.recvmmsg tells the average number of messages that could be pulled from the kernel buffers with a single syscall.
rsyslog statistic counter plugin imtcp
Plugin – imtcp
This plugin maintains statistics for each listener. The statistic is named after the given input name (or “imtcp” if none is configured), followed by the listener port in parenthesis. For example, the counter for a listener on port 514 with no set name is called “imtcp(514)”.
The following properties are maintained for each listener:
- submitted – total number of messages submitted for processing since startup
rsyslog statistic counter plugin imptcp
Plugin – imptcp
This plugin maintains statistics for each listener. The statistic is named “imtcp” , followed by the bound address, listener port and IP version in parenthesis. For example, the counter for a listener on port 514, bound to all interfaces and listening on IPv6 is called “imptcp(*/514/IPv6)”.
The following properties are maintained for each listener:
- submitted – total number of messages submitted for processing since startup
rsyslog statistic counter plugin impstats
Plugin – impstats (rsyslog 7.5.3+) The impstats plugin gathers some internal statistics. They have different names depending on the actual statistics. Obviously, they do not relate to the plugin itself but rather to a broader object – most notably the rsyslog process itself. The “resource-usage” counter maintains process statistics. They base on the getrusage() system call. The counters are named like getrusage returned data memebers. So for details, looking them up in “man getrusage” is highly recommended, especially as value may be different depending on the platform. A getrusage() call is done immediately before the counter is emitted. The follwowing individual counters are maintained:
- utime – this is the user time in microseconds (thus the timeval structure combined)
- stime – again, time given in microseconds
- maxrss
- minflt
- majflt
- inblock
- oublock
- nvcsw
- nivcsw
rsyslog statistic counter plugin omfile
Plugin – omfile (rsyslog 7.3.6+)
This plugin maintains statistics for each dynafile cache. Dynafile cache performance is critical for overall system performance, so reviewing these counters on a busy system (especially one experiencing performance problems) is advisable. The statistic is named “dynafile cache”, followed by the template name used for this dynafile action.
The following properties are maintained for each dynafile:
- requests – total number of requests made to obtain a dynafile
- level0 – requests for the current active file, so no real cache lookup needed to be done. These are extremely good.
- missed – cache misses, where the required file did not reside in cache. Even with a perfect cache, there will be at least one miss per file. That happens when the file is being accessed for the first time and brought into cache. So “missed” will always be at least as large as the number of different files processed.
- evicted – the number of times a file needed to be evicted from the cache as it ran out of space. These can simply happen when date-based files are used, and the previous date files are being removed from the cache as time progresses. It is better, though, to set an appropriate “closeTimeout” (counter described below), so that files are removed from the cache after they become no longer accessed. It is bad if active files need to be evicted from the cache. This is a very costly operation as an evict requires to close the file (thus a full flush, no matter of its buffer state) and a later access requires a re-open – and the eviction of another file, as the cache obviously has run out of free entries. If this happens frequently, it can severely affect performance. So a high eviction rate is a sign that the dynafile cache size should be increased. If it is already very high, it is recommended to re-think about the design of the file store, at least if the eviction process causes real performance problems.
- maxused – the maximum number of cache entries ever used. This can be used to trim the cache down to a value that’s actually useful but does not waste resources. Note that when date-based files are used and rsyslog is run for an extended period of time, the cache gradually fills up to the max configured value as older files are migrated out of it. This will make “maxused” questionable after some time. Frequently enough purging the cache can prevent this (usually, once a day is sufficient).
- closetimeouts – available since 8.3.3 – tells how often a file was closed due to timeout settings (“closeTimeout” action parameter). These are cases where dynafiles or static files have been closed by rsyslog due to inactivity. Note that if no “closeTimeout” is specified for the action, this counter always is zero. A high or low number in itself doesn’t mean anything good or bad. It totally depends on the use case, so no general advise can be given.
Note that the dynafile caches are purged when a HUP is sent.