Changelog for 8.7.0 (v8-stable)
Version 8.7.0 [v8-stable] 2015-01-13
- add message metadata “system” to msg object
this permits to store metadata alongside the message - imfile: add support for “filename” metadata
this is useful in cases where wildcards are used - imptcp: make stats counter names consistent with what imudp, imtcp uses
- added new module “omkafka” to support writing to Apache Kafka
- omfwd: add new “udp.senddelay” parameter
- mmnormalize enhancements
Thanks to Janmejay Singh for the patch. - RainerScript “foreach” iterator and array reading support
Thanks to Janmejay Singh for the patch. - now requires liblognorm >= 1.0.2
- add support for systemd >= 209 library names
- BSD “ntp” facility (value 12) is now also supported in filter
Thanks to Douglas K. Rand of Iteris, Inc. for the patch.
Note: this patch was released under ASL 2.0 (see email-conversation). - bugfix: global(localHostName=”xxx”) was not respected in all modules
- bugfix: emit correct error message on config-file-not-found
closes https://github.com/rsyslog/rsyslog/issues/173 - bugfix: impstats emitted invalid JSON format (if JSON was selected)
- bugfix: (small) memory leak in omfile’s outchannel code
Thanks to Koral Ilgun for reporting this issue. - bugfix: imuxsock did not deactivate some code not supported by platform
Among potential other problemns, this caused build failure under Solaris.
Note that this build problem just made a broader problem appear that so
far always existed but was not visible.
closes https://github.com/rsyslog/rsyslog/issues/185
rsyslog -devel packages are being removed soon
If you use rsyslog’s devel packages on your system, you will receive errors soon. Be sure to read the complete posting to avoid trouble!
As part of rsyslog’s new release schedule and version naming, devel releases will no longer be named according to the “normal” numbering scheme. This also means that the previous “devel” branches will disappear, as git master branch now is the always-current devel version.
Keep on your mind that we previously had a release cycle of 3 to 9 month for a new feature to appear in a stable version. That was because new feature releases were only done when a complete devel turnaround was done, and relatively many new features were added. For this reason, some people opted to run devel versions in production, and thus needed specific tarballs (and packages) for them.
With the new six week release cycle, we get new features rather quickly into the stable builds. So it usually should be no problem to wait for the next stable to use that recently-implemented new feature. As such, there is no need any longer for special devel releases, and thus no need for devel tarballs and packages.
Well… almost. One thing we would like to have is a “daily devel version”. The idea is that if the testbench runs are OK, a new tarball and a set of packages is generated automatically and posted to a special archive. In general, that archive should receive an update once a day. So people really interested in the [b]leading edge can simply install from that daily package archive — and report bugs quickly, so helping the development process. Unfortunately, time is precious and we don’t know when and if we can setup the required automation. Most probably not before January 2015, and how it works out then needs to be seen.
In the interim, we will begin to delete the -devel packages. The old -devel tarballs will remain available, at least for the time being. The problem with -devel packages is that folks may have set their system to use the -devel repro. If we would just keep it as is, those systems would never again receive any updates, neither security-releated nor others, simply because -devel versions no longer exist in the way they were. That would pose a potentially big security risk. As such, we will delete the -devel content, and begin to do so early next week. If you use the -devel packages, be sure to switch the v8-stable instead.
Changelog for 8.6.0 (v8-stable)
Version 8.6.0 [v8-stable] 2014-12-02
NOTE: This version also incorporates all changes and enhancements made for
v8.5.0, but in a stable release. For details see immediately below.
- configuration-setting rsyslogd command line options deprecated
For most of them, there are now proper configuration objects. Some few will be completely dropped if nobody insists on them. Additional info at
http://blog.gerhards.net/2014/11/phasing-out-legacy-command- line-options.html - new and enhanced plugins for 0mq. These are currently experimantal.
Thanks to Brian Knox who contributed the modules and is their author. - empty rulesets have been permitted. They no longer raise a syntax error.
- add parameter -N3 to enable config check of partial config file
Use for config include files. Disables checking if any action exists at
all. - rsyslogd -e option has finally been removed
It is deprectated since many years. - testbench improvements
Testbench is now more robust and has additional tests. - testbench is now by default disabled
To enable it, use –enable-testbench. This was done as the testbench now does better checking if required modules are present and this in turn would lead to configure error messages where non previously were if we would leave –enable-testbench on by default. Thus we have turned it off. This should not be an issue for those few testbench users. - add new RainerScript functions warp() and replace()
Thanks to Singh Janmejay for the patch. - mmnormalize can now also work on a variable
Thanks to Singh Janmejay for the patch. - new property date options for day ordinal and week number
Thanks to github user arrjay for the patch - remove –enable-zlib configure option, we always require it
It’s hard to envision a system without zlib, so we turn this off
closes https://github.com/rsyslog/rsyslog/issues/76 - slight source-tree restructuring: contributed modules are now in their own ./contrib directory. The idea is to make it clearer to the end user which plugins are supported by the rsyslog project (those in ./plugins).
- bugfix: imudp makes rsyslog hang on shutdown when more than 1 thread used
closes https://github.com/rsyslog/rsyslog/issues/126 - bugfix: not all files closed on auto-backgrounding startup
This could happen when not running under systemd. Some low-numbered fds were not closed in that case. - bugfix: typo in queue configuration parameter made parameter unusable
Thanks to Bojan Smojver for the patch. - bugfix: unitialized buffer off-by-one error in hostname generation
The DNS cache used uninitialized memory, which could lead to invalid hostname generation.
Thanks to Jarrod Sayers for alerting us and provinding analysis and patch recommendations. - bugfix imuxsock: possible segfault when SysSock.Use=”off”
Thanks to alexjfisher for reporting this issue.
closes https://github.com/rsyslog/rsyslog/issues/140 - bugfix: RainerScript: invalid ruleset names were accepted during ruleset defintion, but could of course not be used when e.g. calling a ruleset.
IMPORTANT: this may cause existing configurations to error out on start, as they invalid names could also be used e.g. when assigning rulesets. - bugfix: some module entry points were not called for all modules callbacks like endCnfLoad() were primarily being called for input modules. This has been corrected. Note that this bugfix has some regression potential.
- bugfix omlibdbi: connection was taken down in wrong thread
This could have consequences depending on the driver being used. In general, it looks more like a cosmetic issue. For example, with MySQL it lead to a small memory but also an annoying message about a thread not properly torn down. - imttcp was removed because it was an incompleted experimental module
- pmrfc3164sd because it was a custom module nobody used
We used to keep this as a sample inside the tree, but whoever wants to look at it can check in older versions inside git - omoracle was removed because it was orphaned and did not build/work for quite some years and nobody was interested in fixing it
Changelog for 8.5.0 (v8-devel)
Version 8.5.0 [v8-stable] 2014-10-24
- imfile greatly refactored and support for wildcards added
- PRI-handling code refactored for more clarity and robustness
- ommail: add support for RainerScript config system [action() object]
This finally adds support for the new config style. Also, we now permit to set a constant subject text without the need to create a template for it. - refactored the auto-backgrounding method
The code is now more robust and also offers possibilities for enhanced error reporting in the future. This is also assumed to fix some races where a system startup script hang due to “hanging” rsyslogd. - make gntls tcp syslog driver emit more error messages
Messages previously emitted only to the debug log are now emitted as syslog error messages. It has shown that they contain information helpful to the user for troubleshooting config issues. Note that this change is a bit experimental, as we are not sure if there are situations where large amounts of error messages may be emitted. - bugfix: imfile did not complain if configured file did not exist
closes https://github.com/rsyslog/rsyslog/issues/137 - bugfix: build failure on systems which don’t have json_tokener_errors
Older versions of json-c need to use a different API (which don’t exists on newer versions, unfortunately…)
Thanks to Thomas D. for reporting this problem. - imgssapi: log remote peer address in some error messages
Thanks to Bodik for the patch.
Changelog for 8.4.1 (v8-stable)
Version 8.4.1 [v8-stable] 2014-09-30
- imudp: add for bracketing mode, which makes parsing stats easier
- permit at-sign in variable names
closes: https://github.com/rsyslog/rsyslog/issues/110 - bugfix: fix syntax error in anon_cc_numbers.py script
Thanks to github user anthcourtney for the patch.
closes: https://github.com/rsyslog/rsyslog/issues/109 - bugfix: ompgsql: don’t loose uncomitted data on retry
Thanks to Jared Johnson and Axel Rau for the patch. - bugfix: imfile: if a state file for a different file name was set, that different file (name) was monitored instead of the configured one. Now, the state file is deleted and the correct file monitored.
closes: https://github.com/rsyslog/rsyslog/issues/103 - bugfix: omudpspoof: source port was invalid
Thanks to Pavel Levshin for the patch - bugfix: build failure on systems which don’t have json_tokener_errors
Older versions of json-c need to use a different API (which don’t exists on newer versions, unfortunately…)
Thanks to Thomas D. for reporting this problem. - bugfix: omelasticsearch does not work with broken/changed ES 1.0+ API
closes: https://github.com/rsyslog/rsyslog/issues/104 - bugfix: mmanon did not properly anonymize IP addresses starting with ‘9’
Thanks to defa-at-so36.net for reporting this problem.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=529 - bugfix: build problems on SuSe Linux
Thanks Andreas Stieger for the patch - bugfix: omelasticsearch error file did not work correctly on ES 1.0+ due to a breaking change in the ElasticSearch API.
see also: https://github.com/rsyslog/rsyslog/issues/104 - bugfix: potential abort when a message with PRI > 191 was processed if the “pri-text” property was used in active templates, this could be abused to a remote denial of service from permitted senders
see also: CVE-2014-3634
rsyslog 8.3.5 (v8-devel) released
We have just released 8.3.5 of the v8-devel branch.
This is basically a maintenance release. It adds improved error output to omprog and a couple of patches were imported from the latest v7-stable (7.6.4).
ChangeLog:
http://www.rsyslog.com/changelog-for-8-3-5-v8-devel/
Download:
http://www.rsyslog.com/download-v8-devel/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.3.5 (v8-devel)
Version 8.3.5 [v8-devel] 2014-08-05omprog:
- emit error message via syslog() if loading binary fails
This happens after forking, so omprog has no longer access to rsyslog’s regular error reporting functions. Previously, this meant any error message was lost. Now it is emitted via regular syslog (which may end up in a different instance, if multiple instances run…) - couple of patches imported from v7-stable (7.6.4)
Masking data in logs and RSYSLOG
As a mobile payments company, we at SumUp are obligated to follow many industry regulations, one of them being PCI DSS. Restricting access to credit card numbers is a clear need and this implies ensuring they are not part of the logs which are used for various purposes and have bigger audience, not restricted to the authorized list of employees who have access to sensitive data.
PAN or primary account number is used by card issuers as card number which is unique and brings information about the issuer and also in majority of the cases can be validated with Luhn algorithm. This is the number on you credit or debit card and it should be kept secret by us.
One widely used approach is to have quality assurance of the logs all over the development and deployment cycle. This is a needed and valuable attitude however first it takes a lot of human resources and second it is kind of reactive approach in terms of dealing with production systems. So we want something better, something mandatory which can leave us on the safe side if we got human error somewhere in the chain. This is very important in our case where we need to put logging management system out of PCI scope. From four ways which are offered by PCI DSS, Requirement 3.4:
3.4 Render PAN unreadable anywhere it is stored (including on portable digital media, backup media, and in logs)
by using any of the following approaches:
- One-way hashes based on strong cryptography, (hash must be of the entire PAN).
- Truncation (hashing cannot be used to replace the truncated segment of PAN).
- Index tokens and pads (pads must be securely stored).
- Strong cryptography with associated key-management processes and procedures.
our natural choice for log messages is truncation. We want to truncate PAN data if it’s present in the logs for some reason in example in situation when temporary the log level is increased for investigation. While we have centralized log storage which is in PCI scope we want to transfer the logs in real-time in some external location, accessible for developers and BI where they can find and use the information they need.
Since we are using rsyslog for logging daemon our next step was to get in touch with Adiscon – the company behind this brilliant piece of software. They were very interested when I explained the idea and the work started. A little bit later we got new message modification module called mmexternal. It sends the message to some external binary and expects an input. More on the implementation here.
Let me give you an example with a code snippet from rsyslog config and an example of python script which is doing a regular expression to catch and replace i.e. VISA, MasterCard and AMEX cards. You may find a lot of useful regular expressions here:
rsyslog.conf
module(load=”mmexternal”)
action(type=”mmexternal”
binary=”/usr/local/bin/external_python_cards_replace.py”
interface.input=”msg” )
external_python_cards_replace.py
Please note that the above snippets are only examples. With using regular expressions you are going to have many false positives but in general this won’t be an issue. Also note that you can modify completely different parts of the logs and also you are not limited to any language or technique for doing so.
With the following example we have negligible resource consumption on the server where log modification is done. Synthetic test which not claim for accuracy shows around 5% CPU usage on single core 2.5GHz virtual CPU for 100 messages/s.
This is how we are doing it. All comments and suggestions are welcome!
rsyslog 8.2.2 (v8-stable) released
We have just released 8.2.2 of the v8-stable branch.
This release allows to build the missing (contributed) modules under v8. Please note, that most of these modules did not yet receive real testing as we don’t have the necessary environments. We depend on users submitting error reports and helping out to iron out any issues that may arise.
Additionaly, the separated documentation is available as a tarball download on the download page.
http://www.rsyslog.com/changelog-for-8-2-2-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.2.2 (v8-stable)
Version 8.2.2 [v8-stable] 2014-06-02
- made the missing (contributed) modules build under v8
Note that we could do this to the stable, because there is NO regression chance at all: only the modules themselves were changed, and they did NOT work at all previously. Please also note that most of these modules did not yet receive real testing. As we don’t have the necessary environments (easily enough available), we depend on users submitting error reports and helping to iron out any issues that may arise.
Modules:- mmrfc5424addhmac
- omrabbitmq
- omgssapi
- omhdfs
- omzmq3