rsyslog 8.30.0 (v8-stable) released
We have released rsyslog 8.30.0.
This release features a large number of changes. First we should mention the new build requirements for libfastjson 0.99.7 and the build recommendation for imjournal being libsystemd-journal >= 234.
Notable changes are that (JSON) variables are now handled case-insensitive by default, imjournal being able to switch to persistent journal in runtime and the complete refactoring of mmanon. Also, a lot of improvements have been added to the error reporting as well as many bugfixes.
For a complete list of changes, fixes and enhancements, please visit the ChangeLog.
The packages will follow when they are finished.
https://github.com/rsyslog/rsyslog/blob/v8-stable/ChangeLog
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.18.0 (v8-stable)
Version 8.18.0 [v8-stable] 2016-04-19
- testbench: When running privdrop tests testbench tries to drop
user to “rsyslog”, “syslog” or “daemon” when running as root and
you don’t explict set RSYSLOG_TESTUSER environment variable.
Make sure the unprivileged testuser can write into tests/ dir! - templates: add option to convert timestamps to UTC
closes https://github.com/rsyslog/rsyslog/issues/730 - omjournal: fix segfault (regression in 8.17.0)
- imptcp: added AF_UNIX support
Thanks to Nathan Brown for implementing this feature. - new template options
- compressSpace
- date-utc
- redis: support for authentication
Thanks to Manohar Ht for the patch - omkafka: makes kafka-producer on-HUP restart optional
As of now, omkafka kills and re-creates kafka-producer on HUP. This
is not always desirable. This change introduces an action param
(reopenOnHup=”on|off”) which allows user to control re-cycling of
kafka-producer.
It defaults to on (for backward compatibility). Off allows user to
ignore HUP as far as kafka-producer is concerned.
Thanks to Janmejay Singh for implementing this feature - imfile: new “FreshStartTail” input parameter
Thanks to Curu Wong for implementing this. - omjournal: fix libfastjson API issues
This module accessed private data members of libfastjson - ommongodb: fix json API issues
This module accessed private data members of libfastjson - testbench improvements (more tests and more thourough tests)
among others:- tests for omjournal added
- tests for KSI subsystem
- tests for priviledge drop statements
- basic test for RELP with TLS
- some previously disabled tests have been re-enabled
- dynamic stats subsystem: a couple of smaller changes
they also involve the format, which is slightly incompatible to
previous version. As this was out only very recently (last version),
we considered this as acceptable.
Thanks to Janmejay Singh for developing this. - foreach loop: now also iterates over objects (not just arrays)
Thanks to Janmejay Singh for developing this. - improvements to the CI environment
- enhancement: queue subsystem is more robst in regard to some corruptions
It is now detected if a .qi file states that the queue contains more
records than there are actually inside the queue files. Previously this
resulted in an emergency switch to direct mode, now the problem is only
reported but processing continues. - enhancement: Allow rsyslog to bind UDP ports even w/out specific
interface being up at the moment.
Alternatively, rsyslog could be ordered after networking, however,
that might have some negative side effects. Also IP_FREEBIND is
recommended by systemd documentation.
Thanks to Nirmoy Das and Marius Tomaschewski for the patch. - cleanup: removed no longer needed json-c compatibility layer
as we now always use libfastjson, we do not need to support old
versions of json-c (libfastjson was based on the newest json-c
version at the time of the fork, which is the newest in regard
to the compatibility layer) - new External plugin for sending metrics to SPM Monitoring SaaS
Thanks to Radu Gheorghe for the patch. - bugfix imfile: fix memory corruption bug when appending @cee
Thanks to Brian Knox for the patch. - bugfix: memory misallocation if position.from and position.to is used
a negative amount of memory is tried to be allocated if position.from
is smaller than the buffer size (at least with json variables). This
usually leads to a segfault.
closes https://github.com/rsyslog/rsyslog/issues/915 - bugfix: fix potential memleak in TCP allowed sender definition
depending on circumstances, a very small leak could happen on each
HUP. This was caused by an invalid macro definition which did not rule
out side effects. - bugfix: $PrivDropToGroupID actually did a name lookup
… instead of using the provided ID - bugfix: small memory leak in imfile
Thanks to Tomas Heinrich for the patch. - bugfix: double free in jsonmesg template
There has to be actual json data in the message (from mmjsonparse,
mmnormalize, imjournal, …) to trigger the crash.
Thanks to Tomas Heinrich for the patch. - bugfix: incorrect formatting of stats when CEE/Json format is used
This lead to ill-formed json being generated - bugfix omfwd: new-style keepalive action parameters did not work
due to being inconsistently spelled inside the code. Note that legacy
parameters $keepalive… always worked
see also: https://github.com/rsyslog/rsyslog/issues/916
Thanks to Devin Christensen for alerting us and an analysis of the
root cause. - bugfix: memory leaks in logctl utility
Detected by clang static analyzer. Note that these leaks CAN happen in
practice and may even be pretty large. This was probably never detected
because the tool is not often used. - bugfix omrelp: fix segfault if no port action parameter was given
closes https://github.com/rsyslog/rsyslog/issues/911 - bugfix imtcp: Messages not terminated by a NL were discarded
… upon connection termination.
Thanks to Tomas Heinrich for the patch.
rsyslog 8.17.0 (v8-stable) released
We have released rsyslog 8.17.0.
This release brings, among a few bugfixes, a lot of brand-new features. The most important change is probably the libfastjson requirement, which replaces the json-c dependency. There is a new contributed plugin called omampq1 for AMQP 1.0 compliant brokers, a new experimental lookup table support, dynamic statistics counters and many many more.
http://www.rsyslog.com/changelog-for-8-17-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.17.0 (v8-stable)
Version 8.17.0 [v8-stable] 2016-03-08
- NEW REQUIREMENT: libfastjson
see also:
http://blog.gerhards.net/2015/12/rsyslog-and-liblognorm-will-switch-to.html - new testbench requirement: faketime command line tool
This is used to generate a controlled environment for time-based tests; if
not available, tests will gracefully be skipped. - improve json variable performance
We use libfastjson’s alternative hash function, which has been
proven to be much faster than the default one (which stems
back to libjson-c). This should bring an overall performance
improvement for all operations involving variable processing.
closes https://github.com/rsyslog/rsyslog/issues/848 - new experimental feature: lookup table suport
Note that at this time, this is an experimental feature which is not yet
fully supported by the rsyslog team. It is introduced in order to gain
more feedback and to make it available as early as possible because many
people consider it useful.
Thanks to Janmejay Singh for implementing this feature - new feature: dynamic statistics counters
which may be changed during rule processing
Thanks to Janmejay Singh for suggesting and implementing this feature - new contributed plugin: omampq1 for AMQP 1.0-compliant brokers
Thanks to Ken Giusti for this module - new set of UTC-based $now family of variables ($now-utc, $year-utc, …)
- simplified locking when accessing message and local variables
this simlifies the code and slightly increases performance if such
variables are heavily accessed. - new global parameter “debug.unloadModules”
This permits to disable unloading of modules, e.g. to make valgrind
reports more useful (without a need to recompile). - timestamp handling: guard against invalid dates
We do not permit dates outside of the year 1970..2100
interval. Note that network-receivers do already guard
against this, so the new guard only guards against invalid
system time. - imfile: add “trimlineoverbytes” input paramter
Thanks to github user JindongChen for the patch. - ommongodb: add support for extended json format for dates
Thanks to Florian Bücklers for the patch. - omjournal: add support for templates
see also: https://github.com/rsyslog/rsyslog/pull/770
Thanks to github user bobthemighty for the patch - imuxsock: add “ruleset” input parameter
- testbench: framework improvement: configs can be included in test file
they do no longer need to be in a separate file, which saves a bit
of work when working with them. This is supported for simple tests with
a single running rsyslog instance
Thanks to Janmejay Singh for inspiring me with a similar method in
liblognorm testbench. - imptcp: performance improvements
Thanks to Janmejay Singh for implementing this improvement - made build compile (almost) without warnings
still some warnings are suppressed where this is currently required - improve interface definition in some modules, e.g. mmanon, mmsequence
This is more an internal cleanup and should have no actual affect to
the end user. - solaris build: MAXHOSTNAMELEN properly detected
- build system improvement: ability to detect old hiredis libs
This permits to automatically build omhiredis on systems where the
hiredis libs do not provide a pkgconfig file. Previsouly, this
required manual configuration.
Thanks to github user jaymell for the patch. - rsgtutil: dump mode improvements
- auto-detect signature file type
- ability to dump hash chains for log extraction files
- build system: fix build issues with clang
clang builds often failed with a missing external symbol
“rpl_malloc”. This was caused by checks in configure.ac,
which checked for specific GNU semantics. As we do not need
them (we never ask malloc for zero bytes), we can safely
remove the macros.
Note that we routinely run clang static analyer in CI and
it also detects such calls as invalid.
closes https://github.com/rsyslog/rsyslog/issues/834 - bugfix: unixtimestamp date format was incorrectly computed
The problem happened in leap year from March til then end
of year and healed itself at the begining of the next year.
During the problem period, the timestamp was 24 hours too low.
fixes https://github.com/rsyslog/rsyslog/issues/830 - bugfix: date-ordinal date format was incorrectly computed
same root cause aus for unixtimestamp and same triggering
condition. During the affected perido, the ordinal was one
too less. - bugfix: some race when shutting down input module threads
this had little, if at all, effect on real deployments as it resulted
in a small leak right before rsyslog termination. However, it caused
trouble with the testbench (and other QA tools).
Thanks to Peter Portante for the patch and both Peter and Janmejay
Singh for helping to analyze what was going on. - bugfix tcpflood: did not handle connection drops correct in TLS case
note that tcpflood is a testbench too. The bug caused some testbench
instability, but had no effect on deplyments. - bugfix: abort if global parameter value was wrong
If so, the abort happened during startup. Once started,
all was stable. - bugfix omkafka: fix potential NULL pointer addressing
this happened when the topic cache was full and an entry
needed to be evicted - bugfix impstats: @cee cookie was prefixed to wrong fromat (json vs. cee)
Thanks to Volker Fröhlich for the fix. - bugfix imfile: fix race during startup that could lead to some duplication
If a to-be-monitored file was created after inotify was initialized
but before startup was completed, the first chunk of data from this
file could be duplicated. This should have happened very rarely in
practice, but caused occasional testbench failures.
see also: https://github.com/rsyslog/rsyslog/issues/791 - bugfix: potential loss of single message at queue shutdown
see also: https://github.com/rsyslog/rsyslog/issues/262 - bugfix: potential deadlock with heavy variable access
When making havy use of global, local and message variables, a deadlock
could occur. While it is extremly unlikely to happen, we have at least
seen one incarnation of this problem in practice. - bugfix ommysql: on some platforms, serverport parameter had no effect
This was caused by an invalid code sequence which’s outcome depends on
compiler settings. - bugfix omelasticsearch: invalid pointer dereference
The actual practical impact is not clear. This came up when working
on compiler warnings.
Thanks to David Lang for the patch. - bugfix omhiredis: serverport config parameter did not reliably work
depended on environment/compiler used to build - bugfix rsgtutil: -h command line option did not work
Thanks to Henri Lakk for the patch. - bugfix lexer: hex numbers were not properly represented
see: https://github.com/rsyslog/rsyslog/pull/771
Thanks to Sam Hanes for the patch. - bugfix TLS syslog: intermittent errors while sending data
Regression from commit 1394e0b. A symptom often seen was the message
“unexpected GnuTLS error -50 in nsd_gtls.c:530” - bugfix imfile: abort on startup if no slash was present in file name param
Thanks to Brian Knox for the patch. - bugfix rsgtutil: fixed abort when using short command line options
Thanks to Henri Lakk - bugfix rsgtutil: invalid computation of log record extraction file
This caused verification to fail because the hash chain was actually
incorrect. Depended on the input data set.
closes https://github.com/rsyslog/rsyslog/issues/832 - bugfix build system: KSI components could only be build if in default path
Changelog for 8.15.0 (v8-stable)
——————————————————————————
Version 8.15.0 [v8-stable] 2015-12-15
- KSI Lib: Updated code to run with libksi 3.4.0.5
Also libksi 3.4.0.x is required to build rsyslog if ksi support
is enabled. New libpackages have been build as well. - KSI utilities: Added option to ser publication url.
Since libksi 3.4.0.x, there is no default publication url anymore.
The publication url has to be set using the –publications-server
Parameter, otherwise the ksi signature cannot be verified. UserID
and UserKey can also be set by parameter now.
Closes https://github.com/rsyslog/rsyslog/issues/581 - KSI Lib: Fixed wrong TLV container for KSI signatures from 0905 to 0906.
closes https://github.com/rsyslog/rsyslog/issues/587 - KSI/GT Lib: Fixed multiple issues found using static analyzer
- performance improvement for configs with heavy use of JSON variables
Depending on the config, this can be a very big gain in performance. - added pmpanngfw: contributed module for translating Palo Alto Networks logs.
see also: https://github.com/rsyslog/rsyslog/pull/573
Thanks to Luigi Mori for the contribution. - testbench: Changed valgrind option for imtcp-tls-basic-vg.sh
For details see: https://github.com/rsyslog/rsyslog/pull/569 - pmciscoios: support for asterisk before timestamp added
thanks to github user c0by for the patch
see also: https://github.com/rsyslog/rsyslog/pull/583 - solr external output plugin much enhanced
see also: https://github.com/rsyslog/rsyslog/pull/529
Thanks to Radu Gheorghe for the patch. - omrabbitmq: improvements
thanks to Luigi Mori for the patch
see also: https://github.com/rsyslog/rsyslog/pull/580 - add support for libfastjson (as a replacement for json-c)
- KSI utilities: somewhat improved error messages
Thanks to Henri Lakk for the patch.
see also: https://github.com/rsyslog/rsyslog/pull/588 - pmciscoios: support for some format variations
Thanks to github user c0by for the patch - support grok via new contributed module mmgrok
Thanks to 饶琛琳 (github user chenryn) for the contribution. - omkafka: new statistics counter “maxoutqsize”
Thanks to 饶琛琳 (github user chenryn) for the contribution. - improvments for 0mq modules:
- omczmq – suspend / Retry handling – the output plugin can now recover
from some error states due to issues with plugin startup or message sending - omczmq – refactored topic handling code for ZMQ_PUB output to be a little
more efficient - omczmq – added ability to set a timeout for sends
- omczmq – set topics can be in separate frame (default) or part of message
frame (configurable) - omcmzq – code cleanup
- imczmq – code cleanup
- imczmq – fixed a couple of cases where vars could be used uninitialized
- imczmq – ZMQ_ROUTER support
- imczmq – Fix small memory leak from not freeing sockets when done with them
- allow creation of on demand ephemeral CurveZMQ certs for encryption.
Clients may specify clientcertpath=”*” to indicate they want an on
demand generated cert.
Thanks to Brian Knox for the contributions.
- omczmq – suspend / Retry handling – the output plugin can now recover
- cleanup on code to unset a variable
under extreme cases (very, very unlikely), the old code could also lead
to errornous processing - omelasticsearch: build on FreeBSD
Thanks to github user c0by for the patch - pmciscoios: fix some small issues clang static analyzer detected
- testbench: many improvements and some new tests
note that there still is a number of tests which are somewhat racy - overall code improvements thanks to clang static analyzer
- gnutls fix: Added possible fix for gnutls issue #575
see also: https://github.com/rsyslog/rsyslog/issues/575
Thanks to Charles Southerland for the patch - bugfix omkafka: restore ability to build on all platforms
Undo commit aea09800643343ab8b6aa205b0f10a4be676643b
because that lead to build failures on various important platforms.
This means it currently is not possible to configure the location
of librdkafka, but that will affect far fewer people.
closes: https://github.com/rsyslog/rsyslog/issues/596 - bugfix omkafka: fix potentially negative partition number
Thanks to Tait Clarridge for providing a patch. - bugfix: solve potential race in creation of additional action workers
Under extreme circumstances, this could lead to segfault. Note that we
detected this problem thanks to ASAN address sanitzier in combination
with a very exterme testbench test. We do not think that this issue
was ever reported in practice. - bugfix: potential memory leak in config parsing
Thanks to github user linmujia for the patch - bugfix: small memory leak in loading template config
This happened when a plugin was used inside the template. Then, the
memory for the template name was never freed.
Thanks to github user xushengping for the fix. - bugfix: fix extra whitespace in property expansions
Address off-by-one issues introduced in f3bd7a2 resulting in extra
whitespace in property expansions
Thanks to Matthew Gabeler-Lee for the patch. - bugfix: mmfields leaked memory if very large messages were processed
detected by clang static analyzer - bugfix: mmfields could add garbagge data to field
this happened when very large fields were to be processed.
Thanks to Peter Portante for reporting this. - bugfix: omhttpfs now also compiles with older json-c lib
- bugfix: memory leak in (contributed) module omhttpfs
Thanks to git hub user c6226 for the patch. - bugfix: parameter mismatch in error message for wrap() function
- bugfix: parameter mismatch in error message for random() function
- bugfix: divide by zero if max() function was provided zero
- bugfix: invalid mutex handling in omfile async write mode
could lead to segfault, even though highly unlikely (caught by
testbench on a single platform) - bugfix: fix inconsistent number processing
Unfortunately, previous versions of the rule engine tried to
support oct and hex, but that wasn’t really the case.
Everything based on JSON was just dec-converted. As this was/is
the norm, we fix that inconsistency by always using dec.
Luckly, oct and hex support was never documented and could
probably only have been activated by constant numbers. - bugfix: timezone() object: fix NULL pointer dereference
This happened during startup when the offset or id parameter was not
given. Could lead to a segfault at startup.
Detected by clang static analyzer. - bugfix omfile: memory addressing error if very long outchannel name used
Thanks to github user c6226 for the patch.