omfile

rsyslog 8.23.0 (v8-stable) released

We have released rsyslog 8.23.0.

This release is packed with changes and enhancements. One of the most interesting might be the removal of the SHA2-224 hash algorithm for KSI signatures. This is considered insecure and is no longer supported by the KSI library. Also notable are the changes to imfile, omfile and omelasticsearch, among lots of others. Please take a look at the Changelog for a full overview.

Note: We delayed the next release by two weeks to the Jan 10, so we don’t have to deal with a release around the Christmas holidays. This also means that 8.23.0 is the final release for 2016.
ChangeLog:

Changelog for 8.12.0 (v8-stable)

Version 8.12.0 [v8-stable] 2015-08-11

  • Harmonize resetConfigVariables values and defaults
    see also https://github.com/rsyslog/rsyslog/pull/413
    Thanks to Tomas Heinrich for the patch.
  • GT/KSI: fix some issues in signature file format and add conversion tool
    The file format is incompatible to previous format, but tools have been upgraded to handle both and also an option been added to convert from old to new format.
  • bugfix: ommysql did not work when gnutls was enabled
    As it turned out, this was due to a check for GnuTLS functions with the side-effect that AC_CHECK_LIB, by default, adds the lib to LIBS, if there is no explicit action, what was the case here. So everything was now linked against GnuTLS, which in turn made ommysql fail.
    Thanks to Thomas D. (whissi) for the analysis of the ommysql/gnutls problem and Thomas Heinrich for pointing out that AC_CHECK_LIB might be the culprit.
  • bugfix omfile: potential memory leak on file close
    see also: https://github.com/rsyslog/rsyslog/pull/423
    Thanks to Robert Schiele for the patch.
  • bugfix omfile: potential race in dynafile detection/creation
    This could lead to a segfault.
    Thanks to Tomas Heinrich for the patch.
  • bugfix omfile: Fix race-condition detection in path-creation code
    The affected code is used to detect a race condition in between testing for the existence of a directory and creating it if it didn’t exist.  The variable tracking the number of attempts wasn’t reset for subsequent elements in the path, thus limiting the number of reattempts to one per the whole path, instead of one per each path element.
    This solution was provided by Martin Poole.
  • bugfix parser subsystem: potential misadressing in SanitizeMsg() could lead to a segfault
    Thanks to Tomas Heinrich for the patch.
  • imfile: files moved outside of directory are now (properly) handled
  • bugfix: imfile: segfault when using startmsg.regex if first log line doesn’t match
    Thanks to Ciprian Hacman for the patch.
  • bugfix imfile: file table was corrupted when on file deletion
    This could happen when a file that was statically configured (not via an wildcard) was deleted.
  • bugfix ompgsql: transaction were improperly handled
    Now transaction support is solidly disabled until we have enough requests to implement it again. Module still works fine in single insert mode.
    closes https://github.com/rsyslog/rsyslog/issues/399
  • bugfix mmjsonparse: memory leak if non-cee-json message is processed
    see also https://github.com/rsyslog/rsyslog/pull/383
    Thanks to Anton Matveenko for the patch
  • testbench: remove raciness from UDP based tests
  • testbench: added bash into all scripts makign it mandatory
  • bugfix testbench: Fixed problem building syslog_caller util when liblogging-stdlog is not available.
    Thanks to Louis Bouchard for the patch
  • bugfix rscryutil.1: Added fix checking for generate_man_pages condition
    Thanks to Radovan Sroka for the patch
  • bugfix freebsd console: \n (NL) is prepended with \r (CR) in console output on freebsd only. For more details see here:
    https://github.com/rsyslog/rsyslog/issues/372
    Thanks to AlexandreFenyo for the patch

Changelog for 7.6.4 (v7-stable)

Version 7.6.4 [v7.6-stable] 2014-09-12

  • add –enable-generate-man-pages configure switch (default: enabled)
    This forces generation of man pages, even if cached ones exists. This “fixes” a typical release tarball nit. While it is hackish, the benefit is clear given the history of failed tarball releases since we changed the cached man page handling. It was just too easy to get that wrong.
  • removed obsolete –disable-fsstnd configure option
    Thanks to Thomas D. for alerting us.
    Closes: https://github.com/rsyslog/rsyslog/issues/72
  • permits to build against json-c 0.12
    Unfortunately, json-c had an ABI breakage, so this is necessary. Note that versions prior to 0.12 had security issues (CVE-2013-6370, CVE-2013-6371) and so it is desirable to link against the new version.
    Thanks to Thomas D. for the patch. Note that at least some distros have fixed the security issue in older versions of json-c, so this seems to apply mostly when building from sources.
  • new omfile default module parameters
    • filecreatemode
    • fileowner
    • fileownernum
    • filegroup
    • filegroupnum
    • dirowner
    • dirownernum
    • dirgroup
    • dirgroupnum

    Thanks to Karol Jurak for the patch.

  • bugfix: memory leak in TCP TLS mode
  • bugfix: imfile: if a state file for a different file name was set, that different file (name) was monitored instead of the configured one. Now, the state file is deleted and the correct file monitored.
    closes: https://github.com/rsyslog/rsyslog/issues/103
  • bugfix: using UUID property could cause segfault
  • bugfix: mmutf8fix did not detect two invalid sequences
    Thanks to Axel Rau for the patch.
  • bugfix: file descriptor leak with Guardtime signatures
    When a .gtstate file is opened it is never closed. This is especially bad when dynafiles frequently get evicted from dynafile cache and be re-opened again.
  • bugfix: busy loop in tcp listener when running out of file descriptors
    Thanks to Susant Sahani for the patch.
  • bugfix: mishandling of input modules not supporting new input instances
    If they did not support this, accidently the output module part of the module union was written, leading to unpredictable results. Note: all core modules do support this interface, but some contributed or very old ones do not.
  • bugfix: double-free when ruleset() parser parameters were used
    While unlikely, this could cause stability issues even after the config phase.
  • bugfix: output modules with parameters with multiple passing modes could caused strange behaviour including aborts
    This was due to the fact that the action module only preserved and processed the last set passing mode. Note that this was not a problem for the plugins provided by the rsyslog git: none of them uses different passing modes.
    Thanks to Tomas Heinrich for providing a very detailled bug report.
  • various fixes after coverty scan
    These do not address issues seen in practice but those seen by the tool. Some of them may affect practical deployments.
    Thanks to Tomas Heinrich for the patches.
  • bugfix imuxsock: “Last message repeated…” was not emitted at shutdown
    The “Last message repeated…” notice didn’t get printed if rsyslog was shut down before the repetition was broken.
    Thanks to Tomas Heinrich for the patch.
  • bugfix: make dist failed when GUARDTIME or LIBGCRYPT feature was disabled
  • bugfix: mmjsonparse did not build with json-c < 0.10
    This was a regression introduced some time in the past in order to support API changes in json-c. Now we check for the version and use proper code.
  • bugfix: mmanon did not properly anonymize IP addresses starting with ‘9’
    Thanks to defa-at-so36.net for reporting this problem.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=529

rsyslog statistic counter plugin omfile

Plugin – omfile (rsyslog 7.3.6+)

This plugin maintains statistics for each dynafile cache. Dynafile cache performance is critical for overall system performance, so reviewing these counters on a busy system (especially one experiencing performance problems) is advisable. The statistic is named “dynafile cache”, followed by the template name used for this dynafile action.

The following properties are maintained for each dynafile:

  • requests – total number of requests made to obtain a dynafile
  • level0 – requests for the current active file, so no real cache lookup needed to be done. These are extremely good.
  • missed – cache misses, where the required file did not reside in cache. Even with a perfect cache, there will be at least one miss per file. That happens when the file is being accessed for the first time and brought into cache. So “missed” will always be at least as large as the number of different files processed.
  • evicted – the number of times a file needed to be evicted from the cache as it ran out of space. These can simply happen when date-based files are used, and the previous date files are being removed from the cache as time progresses. It is better, though, to set an appropriate “closeTimeout” (counter described below), so that files are removed from the cache after they become no longer accessed. It is bad if active files need to be evicted from the cache. This is a very costly operation as an evict requires to close the file (thus a full flush, no matter of its buffer state) and a later access requires a re-open – and the eviction of another file, as the cache obviously has run out of free entries. If this happens frequently, it can severely affect performance. So a high eviction rate is a sign that the dynafile cache size should be increased. If it is already very high, it is recommended to re-think about the design of the file store, at least if the eviction process causes real performance problems.
  • maxused – the maximum number of cache entries ever used. This can be used to trim the cache down to a value that’s actually useful but does not waste resources. Note that when date-based files are used and rsyslog is run for an extended period of time, the cache gradually fills up to the max configured value as older files are migrated out of it. This will make “maxused” questionable after some time. Frequently enough purging the cache can prevent this (usually, once a day is sufficient).
  • closetimeouts –  available since 8.3.3 – tells how often a file was closed due to timeout settings (“closeTimeout” action parameter). These are cases where dynafiles or static files have been closed by rsyslog due to inactivity. Note that if no “closeTimeout” is specified for the action, this counter always is zero. A high or low number in itself doesn’t mean anything good or bad. It totally depends on the use case, so no general advise can be given.

Note that the dynafile caches are purged when a HUP is sent.

Back to statistics counter overview

rsyslog 7.5.4 (v7-devel) released

This release offers some interesting features. It provides a new module called mmpstrucdata to parse RFC5424 structured data into json message properties. Also the default queue.size values have been altered to more suitable values. Omfwd and omfile received new parameters and we changed a bigger portion of the documentation to improve usability by linking relevant web ressources to quickly find additional information. Finally, there have been a few other changes and bugfixes.

More detailed information is available in the changelog.

ChangeLog:

http://www.rsyslog.com/changelog-for-7-5-4-v7-devel/

Download:

http://www.rsyslog.com/rsyslog-7-5-4-v7-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Changelog for 7.5.4 (v7-devel)

Version 7.5.4 [devel] 2013-10-07

  • mmpstrucdata: new module to parse RFC5424 structured data into json message properties
  • change main/ruleset queue defaults to be more enterprise-like
    new defaults are queue.size 100,000 max workers 2, worker activation after 40,000 msgs are queued, batch size 256. These settings are much more useful for enterprises and will not hurt low-end systems that much. This is part of our re-focus on enterprise needs.
  • omfwd: new action parameter “maxErrorMessages” added
  • omfile: new module parameters to set action defaults added
    * dirCreateMode
    * fileCreateMode
  • mmutf8fix: new module to fix invalid UTF-8 sequences
  • imuxsock: handle unlimited number of additional listen sockets
  • doc: improve usability by linking to relevant web ressources
    The idea is to enable users to quickly find additional information, samples, HOWTOs and the like on the main site. At the same time, (very) slightly remove memory footprint when few listeners are monitored.
  • bugfix: omfwd parameter streamdrivermmode was not properly handled
    It was always overwritten by whatever value was set via the legacy directive $ActionSendStreamDriverMode
  • imtcp: add streamdriver.name module parameter
    permits overriding the system default stream driver (gtls, ptcp)
  • bugfix: build system: libgcrypt.h needed even if libgrcypt was disabled
    Thanks to Jonny Törnbom for reporting this problem
  • imported bugfixes from 7.4.4

rsyslog 7.3.0 (v7-devel) released

This start the new 7.3.x development branch. The first feature introduced is an enhancement that dramatically reduces the size of omfile-generated ZIP files.

ChangeLog: http://www.rsyslog.com/changelog-for-7-3-0-v7-devel/

Download: http://www.rsyslog.com/rsyslog-7-3-0-v7-devel/

As always, feedback is appreciated.

Best regards, Tim Eifler

rsyslog statistics counter

Rsyslog supports statistic counters via the impstats module. It is important to know that impstats and friends only provides an infrastructure where core components and plugins can register statistics counter. This FAQ entry tries to describe all counters available, but please keep in mind that there may exist that we  do not know about.

When interpreting rsyslog statistics, please keep in mind that statistics records are processed as regular syslog messages. As such, the statistics messages themselves increment counters when they are emitted via the regular syslog stream, which is the default (and so counters keep slowly increasing even if there is absolutely no other traffic). Also keep in mind that a busy rsyslog system is very dynamic. Most importantly, this means that the counters may not be 100% consistent, but some slight differences may exist. Avoiding such inconsistencies would be possible only at the price of a very tight locking discipline, which would cause serious performance bottlenecks. Thus, this is not done. Finally, though extremely unlikely, some counters may experience an overflow and restart at 0 for that reasons. However, most counters are 64-bit, so this is extremely unlikely. Those which are not 64 bit are typically taken from some internal data structure that uses lower bits for performance reasons and guards against overflow.

The listing starts with the core component or plugin that creates the counters and than specifies various counters that exist for the sub-entities. The listing below is extended as new counters are added. Some counters probably do not exist in older releases of rsyslog.

Below you can find all available core components and plugins. Please note that every core component or plugin are linked to a information site.

Queue
Actions

PLUGINS

rsyslog 5.9.4 (devel) released

This release provides support for “trusted properties”, which may enhance overall system security. This is a new concept and feedback on it is appreciated. For more details on trusted properties, please visit

http://www.rsyslog.com/what-are-trusted-properties/

or Rainer’s blog post with some more background about trusted properties:

http://blog.gerhards.net/2011/11/trusted-properties-in-rsyslog.html

In addition to this feature, we have reduced dependency on libgcrypt and fixed some bugs.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-9-4-v5-devel/

Download:

http://www.rsyslog.com/rsyslog-5-9-4-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Changelog for 5.9.4 (v5-devel)

Version 5.9.4 [V5-DEVEL], 2011-11-29

  • imuxsock: added capability to “annotate” messages with “trusted information”, which contains some properties obtained from the system and as such is sure to not be faked. This is inspired by the similiar idea introduced in systemd. removed dependency on gcrypt for recently-enough GnuTLS
    see: http://bugzilla.adiscon.com/show_bug.cgi?id=289
  • bugfix: imuxsock did no longer ignore message-provided timestamp, if so configured (the *default*). Lead to no longer sub-second timestamps.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=281
  • bugfix: omfile returns fatal error code for things that go really wrong previously, RS_RET_RESUME was returned, which lead to a loop inside the rule engine as omfile could not really recover.
  • bugfix: rsyslogd -v always said 64 atomics were not present
    thanks to mono_matsuko for the patch
Scroll to top