impstats delayed or lost? – cause and cure
Some users report that they do not receive all impstats log records or that these log records are delayed. The common thing about these questions is that those user tend to have very large main message queues.
By default, impstats is run in-band, which means that it’s messages are submitted to the main message queue just like any other messages are. So if the main queue takes very long to processs, impstats messages get delayed as well. They may be discarded, too, based on queue settings and queue full status. In this scenarios, impstats obviously has problems reporting what is going on.
A simple solution is to run impstats off-band. This is done by simply instructing it to directly write to file. In that mode, the rsyslog engine is not used at all, and output will always be available and happen right on the interval. There is a drawback, though: as the rsyslog core engine is not involved, things like dynafiles, different templates or forwarding to another host are not possible.
Note that impstats can submit messages both to a file and the regular message stream. This may be an interesting alternative if the main queue causes trouble but usually logs shall be gathered at some central place.
Example for writing to a local file:
module(load="impstats" interval="600" severity="7" log.file="/var/log/impstats")
rsyslog 7.5.4 (v7-devel) released
This release offers some interesting features. It provides a new module called mmpstrucdata to parse RFC5424 structured data into json message properties. Also the default queue.size values have been altered to more suitable values. Omfwd and omfile received new parameters and we changed a bigger portion of the documentation to improve usability by linking relevant web ressources to quickly find additional information. Finally, there have been a few other changes and bugfixes.
More detailed information is available in the changelog.
ChangeLog:
http://www.rsyslog.com/changelog-for-7-5-4-v7-devel/
Download:
http://www.rsyslog.com/rsyslog-7-5-4-v7-devel/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 7.5.4 (v7-devel)
Version 7.5.4 [devel] 2013-10-07
- mmpstrucdata: new module to parse RFC5424 structured data into json message properties
- change main/ruleset queue defaults to be more enterprise-like
new defaults are queue.size 100,000 max workers 2, worker activation after 40,000 msgs are queued, batch size 256. These settings are much more useful for enterprises and will not hurt low-end systems that much. This is part of our re-focus on enterprise needs. - omfwd: new action parameter “maxErrorMessages” added
- omfile: new module parameters to set action defaults added
* dirCreateMode
* fileCreateMode - mmutf8fix: new module to fix invalid UTF-8 sequences
- imuxsock: handle unlimited number of additional listen sockets
- doc: improve usability by linking to relevant web ressources
The idea is to enable users to quickly find additional information, samples, HOWTOs and the like on the main site. At the same time, (very) slightly remove memory footprint when few listeners are monitored. - bugfix: omfwd parameter streamdrivermmode was not properly handled
It was always overwritten by whatever value was set via the legacy directive $ActionSendStreamDriverMode - imtcp: add streamdriver.name module parameter
permits overriding the system default stream driver (gtls, ptcp) - bugfix: build system: libgcrypt.h needed even if libgrcypt was disabled
Thanks to Jonny Törnbom for reporting this problem - imported bugfixes from 7.4.4
Changelog for 7.5.3 (v7-devel)
Version 7.5.3 [devel] 2013-09-11
- imfile: support for escaping LF characters added embedded LF in syslog messages cause a lot of trouble. imfile now has the capability to escape them to “#012″ (just like the regular control character escape option). This requires new-style input statements to be used. If legacy configuration statements are used, LF escaping is always turned off to preserve compatibility.
NOTE: if input() statements were already used, there is a CHANGE OF BEHAVIOUR: starting with this version, escaping is enabled by default. So if you do not want it, you need to add escapeLF=”off” to the input statement. Given the trouble LFs cause and the fact that the majority of installations still use legacy config, we considered this behaviour change acceptable and useful.
see also: http://blog.gerhards.net/2013/09/imfile-multi-line-messages. html - add support for global and local variables
- bugfix: queue file size was not correctly processed
this could lead to using one queue file per message for sizes >2GiB
Thanks to Tomas Heinrich for the patch. - add main_queue() configuration object to configure main message queue
- bugfix: stream compression in imptcp caused timestamp to be corrupted
- imudp: add ability to specify SO_RCVBUF size (rcvbufSize parameter)
- imudp: use inputname for statistics, if configured
- impstats: add process resource usage counters [via getrusage()]
- impstats: add paramter “resetCounters” to report delta values possible for most, but not all, counters. See doc for details.
- librelp 1.2.0 is now required
- make use of new librelp generic error reporting facility
This leads to more error messages being passed to the user and thus simplified troubleshooting. - bugfix: very small memory leak in imrelp
more or less cosmetic, a single memory block was not freed, but this only happens immediately before termination (when the OS automatically frees all memory). Still an annoyance e.g. in valgrind. - fix compile problem in debug build
- imported fixes from 7.4.4
Changelog for 7.4.4 (v7-stable)
Version 7.4.4 [v7.4-stable] 2013-09-03
- better error messages in GuardTime signature provider
Thanks to Ahto Truu for providing the patch. - make rsyslog use the new json-c pkgconfig file if available
Thanks to the Gentoo team for the patches. - bugfix: imfile parameter “persistStateInterval” was unusable
due to a case typo in imfile; work-around was to use legacy config
Thanks to Brandon Murphy for reporting this bug. - bugfix: TLV16 flag encoding error in signature files from GT provider
This fixes a problem where the TLV16 flag was improperly encoded. Unfortunately, existing files already have the bug and may not properly be processed. The fix uses constants from the GuardTime API lib to prevent such problems in the future.
Thanks to Ahto Truu for providing the patch. - bugfix: slightly malformed SMTP handling in ommail
- bugfix: segfault in omprog if no template was provided (now dflt is used)
- bugfix: segfault in ompipe if no template was provided (now dflt is used)
- bugfix: segfault in omsnmp if no template was provided (now dflt is used)
- bugfix: some omsnmp optional config params were flagged as mandatory
- bugfix: segfault in omelasticsearch when resuming queued messages after restarting Elasticsearch
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=464 - bugfix: imtcp addtlframedelimiter could not be set to zero
Thanks to Chris Norton for alerting us. - doc bugfix: remove no-longer existing omtemplate from developer doc was specifically mentioned as a sample for creating new plugins
Thanks to Yannick Brosseau for alerting us of this problem.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=473
7.4.0 – the new stable
new rsyslog 7.4.0 stable released
We just released rsyslog 7.4.0, a new stable release which replaces the 7.2 branch. After nine month of hard work, there are many exciting enhancements compared to 7.2, and I thought I give you a quick rundown of the more important new features. Note that while I list some “minor features” at the end of this posting, the list is not complete. I left out those things that are primarily of interest for smaller groups of users. So if you look for a specific feature not mentioned here, it may pay to look at the ChangeLog or post a question to the rsyslog mailing list.
With this release, the rsyslog project officially supports the 7.4 and 7.2 branches. If support for older versions is required, utilizing Adiscon’s professional services for rsyslog is recommended.
Note that I only list main headlines for each of the features. Follow links (where provided) to gain more in-depth information.
Security Package
- introduction of Linux-like rate-limiting for all inputs
- “Last message repeated n times” now done on a per-input basis; makes it much more useful AND increases processing speed.
- omjournal writes messages to the journal
- imjournal obtains messages including journal-specific meta data items from the journal
Performance Improvements
- Disk Queue Performance greatly improved
- DNS chache greatly improved and optimized
- omfile now supports fully async writing
- script optimizer now changes frequently-used long-running filters into equivalent quick ones when possible (this even affects some distros default configs and is a great performance saver)
Minor Features
- various plugins now support the new RainerScript based config language
- omlibdbi improvements, among them support for transactions
- ommysql now supports transactions
- improved omfile zip writing speed
- performance optimizations
- omelasticsearch improvements (bulk processing mode, local error file)
- omudpspoof now supports packets larger than 1472 by using fragmentation
- omrelp now supports session timeout
- contributed module (not project-supported) omrabbitmq was added
Changelog for 7.3.4 (v7-devel)
Version 7.3.4 [devel] 2012-11-23
- further (and rather drastically) improved disk queue performance
we now save one third of the IO calls - imklog: added ParseKernelTimestamp parameter (import from 5.10.2)
Thanks to Marius Tomaschewski for the patch. - imklog: added KeepKernelTimestamp parameter (import from 5.10.2)
Thanks to Marius Tomaschewski for the patch. - bugfix: improper handling of backslash in string-type template()s
- bugfix: leading quote (“) in string-type template() lead to thight loop
on startup - bugfix: no error msg on invalid field option in legacy/string template
- bugfix: imklog mistakenly took kernel timestamp subseconds as nanoseconds
… actually, they are microseconds. So the fractional part of the
timestamp was not properly formatted. (import from 5.10.2)
Thanks to Marius Tomaschewski for the bug report and the patch idea.
rsyslog 7.3.4 (v7-devel) released
This release most importantly contains drastic improvements of disk queue performance (including DA queues). The new code saves one third of file IO calls, improving both rsyslogd as well as overall system performance. It also contains some smaller enhancements as well as bug fixes.
ChangeLog:
http://www.rsyslog.com/changelog-for-7-3-4-v7-devel/
Download:
http://www.rsyslog.com/rsyslog-7-3-4-v7-devel/
As always, feedback is appreciated.
Best regards,
Tim Eifler
rsyslog 7.3.3 (v7-devel) released
We have just released 7.3.3 of the v7-devel branch. This release most importantly provides increased performance for the disk queue subsystem. Especially CPU-use will be much lower when unspooling disk queues. It also contains all known fixes for v7, some imported from v7-stable and earlier versions.
ChangeLog:
http://www.rsyslog.com/changelog-for-7-3-3-v7-devel/
Download:
http://www.rsyslog.com/rsyslog-7-3-3-v7-devel/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 7.3.3 (v7-devel)
Version 7.3.3 [devel] 2012-11-07
- improved disk queue performance
- bugfix: dynafile zip files could be corrupted
This could happen if a dynafile was destructed before the first write. In practice, this could happen if few lines were written to a file and it then became evicted from the dynafile cache. This would probably look very random, because it depended on the timing in regard to message volume and dynafile cache size.