remote syslog PRI vulnerability – CVE: CVE-2014-3634
===============================
CVE: CVE-2014-3634
Status of this report
———————
FINAL
Reporter
——-
Rainer Gerhards, rsyslog project lead
Affected
——–
– rsyslog, most probably all versions (checked 5.8.6+)
– sysklogd (checked most recent versions)
– potentially others (see root cause)
Root Cause
———-
Note: rsyslogd was forked from sysklogd, and the root cause applies to
both. For simplicity, here I use sysklogd as this is the base code.
The system header file /usr/include/*/syslog.h contains the following definitions
#define LOG_NFACILITIES 24 /* current number of facilities */
#define LOG_FACMASK 0x03f8 /* mask to extract facility part */
/* facility of pri */
#define LOG_FAC(p) (((p) & LOG_FACMASK) >> 3)
[This is from Ubuntu 12.04LTS, but can be found similarly in most, if
not all, distributions].
The define LOG_NFACILITIES is used by sysklogd to size arrays for facility
processing. In sysklogd, an array for selector matching is using this. Rsyslog
has additional array. The macro LOG_FAC() is used to extract the facility from
a syslog PRI [RFC3164, RFC 5424]. Its result is used to address the arrays.
Unfortunately, the LOG_FACMASK permits PRI values up to 0x3f8 (1016 dec). This
translates to 128 facilities. Consequently, for PRI values above 191 the
LOG_NFACILITIES arrays are overrun.
Other applications may have similar problems, as LOG_NFACILITES “sounds” like
the max value that LOG_FAC() can return. It would probably make sense to
check why there is a difference between LOG_NFACILITES and LOG_FACMASK, and
if this really needs to stay. A proper fix would probably be to make LOG_FAC
return a valid (maybe special) facility if an invalid one is provided. This
is the route taken in rsyslog patches.
Effect in Practice
——————
General
~~~~~~~
Almost all distributions do ship rsyslog without remote reception by
default and almost all distros also put firewall rules into place that
prevent reception of syslog messages from remote hosts (even if rsyslog
would be listening). With these defaults, it is impossible to trigger
the vulnerability in v7 and v8. Older versions may still be vulnerable
if a malicious user writes to the local log socket.
Even when configured to receive remote message (on a central server),
it is good practice to protect such syslog servers to accept only
messages from trusted peers, e.g. within the relay chain. If setup in
such a way, a trusted peer must be compromised to send malfromed
messages. This further limits the magnitude of the vulnerability.
If, however, any system is permitted to send data unconditionally to
the syslogd, a remote attack is possible.
sysklogd
~~~~~~~~
Sysklogd is mildly affected. Having a quick look at the current git master
branch, the wrong action may be applied to messages with invalid facility.
A segfault seems unlikely, as the maximum misadressing is 104 bytes of the
f_pmask table, which is always within properly allocated memory (albeit to
wrong data items). This can lead to triggering invalid selector lines and
thus wrongly writing to files or wrongly forwarding to other hosts.
rsyslogd
~~~~~~~~
Rsyslogd experiences the same problem as sysklogd.
However, more severe effects can occur, BUT NOT WITH THE DEFAULT CONFIGURATION.
The most likely and thus important attack is a remote DoS. Some
of the additional tables are writable and can cause considerable misadressing.
This is especially true for versions 7 and 8. In those versions, remote code
injection may also be possible by a carefully crafted package. It sounds hard
to do, but it cannot be totally outruled [we did not check this in depth].
A segfault (and thus Dos) has the following preconditions:
– the rsyslog property “pri-text” must be used, either in
* templates
* conditional statements (RainerScript and property-based filters)
– the property must actually be accessed
With traditional selector lines, this depends on the facility causing
a misadressing that leads to reading a 1 from the misaccessed location.
When the preconditions are met, misadressing happens. The code uses a string
table and a table of string lengths. Depending on memory layout at time of
misadressing and depending on the actual invalid PRI value, the lookup to
the string table can lead to a much to long length, which is the used in
buffer copy calculations. High PRI values close to the max of 1016 potentially
cause most problems, but we have also seen segfaults with very low invalid
PRI values.
Note that, as usual in such situations, a segfault may not happen immediately.
Instad some data structures may be damaged (e.g. from the memory allocator)
which will later on result in a segfault.
In v5 and below, a segfault is very unlikely, as snprintf() is used to generate
the pri-text property. As such, no write overrun can happen (but still garbagge
be contained inside the property). A segfault could theoretically happen if the
name lookup table indices cause out-of-process misadressing. We could not manage
to produce a segfault with v5.
Versions 7.6.3 and 7.6.4 already have partial fixes for the issue and will not
be vulnerable to a segfault (but the mild other issues described).
All other versions 7 and 8 are vulnerable. Version 6 was not checked as it seems
no longer be used in practice (it was an interim version). No patch for version 6
will be provided.
Note that a segfault of rsyslog can cause message loss. There are multiple
scenarios for this, but likely ones are:
– reception via UDP, where all messages arriving during downtime are lost
– corruption of disk queue structures, which can lead to loss of all disk
queue contents (manual recovery is possible).
This list does not try to be complete. Note that disk queue corruption is likely
to occur in default settings, because the important queue information file (.qi)
is only written on successful shutdown. Without a valid .qi file, queue message
files cannot be processed.
How to Exploit
————–
A syslog message with an invalid PRI value needs to be sent. It is sufficient
to send just the PRI as in this example
“<201>”
Any message starting with “<PRI>” where PRI is an integer greater than 191
can trigger the problem. The maximum offset that can be generated is with
PRI equal to 1016, as this is the modulus used due to LOG_FACMASK.
Note that messages with
– PRI > 191 and
– PRI modulus 1016 <= 191
will not lead to misadressing but go into the wrong bin.
Messsages with
– PRI > 191
– PRI modulus 1016 > 191
will go into the wrong bin and lead to misadressing.
Severity
——–
Given the triggering scenarios, the fact that multiple changes must be made to
default system configurations and potential problems we classify the severity
of this vulnerability as
MEDIUM
Note that the probability of a successful attack is LOW. However, the risk of
message loss is HIGH in those rare instances where an attack is successful. As
mentioned above, it cannot totally be outruled that remote code injection is
possible using this vulnerability.
This vulnerability is at least not publicly know. Based on (no) bug reports, it
seems unlikely that it is being exploited, but that’s obviously hard to know for
sure.
Patches
——-
Patches are available for versions known to be in wide-spread use.
Version 8.4.1 is not vulnerable. Version 7.6.6, while no longer being project
supported received a patch and is also not vulnerable.
New patches for the article available http://www.rsyslog.com/remote-syslog-pri-vulnerability-cve-2014-3683/
rsyslog 8.4.1 (v8-stable) released
We have just released 8.4.1 of the v8-stable branch.
This is primarily a bug-fixing release, but provides one small enhancement, the so-called “bracketing mode” of impstats. It tells impstats to emit begin and end message before and after emitting stats.
Please note that this releases also fixes a potential remote DoS, which may happen for some (non-default) configurations. As such, users are highly encouraged to upgrade to this version.
http://www.rsyslog.com/changelog-for-8-4-1-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.4.1 (v8-stable)
Version 8.4.1 [v8-stable] 2014-09-30
- imudp: add for bracketing mode, which makes parsing stats easier
- permit at-sign in variable names
closes: https://github.com/rsyslog/rsyslog/issues/110 - bugfix: fix syntax error in anon_cc_numbers.py script
Thanks to github user anthcourtney for the patch.
closes: https://github.com/rsyslog/rsyslog/issues/109 - bugfix: ompgsql: don’t loose uncomitted data on retry
Thanks to Jared Johnson and Axel Rau for the patch. - bugfix: imfile: if a state file for a different file name was set, that different file (name) was monitored instead of the configured one. Now, the state file is deleted and the correct file monitored.
closes: https://github.com/rsyslog/rsyslog/issues/103 - bugfix: omudpspoof: source port was invalid
Thanks to Pavel Levshin for the patch - bugfix: build failure on systems which don’t have json_tokener_errors
Older versions of json-c need to use a different API (which don’t exists on newer versions, unfortunately…)
Thanks to Thomas D. for reporting this problem. - bugfix: omelasticsearch does not work with broken/changed ES 1.0+ API
closes: https://github.com/rsyslog/rsyslog/issues/104 - bugfix: mmanon did not properly anonymize IP addresses starting with ‘9’
Thanks to defa-at-so36.net for reporting this problem.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=529 - bugfix: build problems on SuSe Linux
Thanks Andreas Stieger for the patch - bugfix: omelasticsearch error file did not work correctly on ES 1.0+ due to a breaking change in the ElasticSearch API.
see also: https://github.com/rsyslog/rsyslog/issues/104 - bugfix: potential abort when a message with PRI > 191 was processed if the “pri-text” property was used in active templates, this could be abused to a remote denial of service from permitted senders
see also: CVE-2014-3634
rsyslog 7.6.6 (v7-stable) released
This version adresses some bugs and a security issue. Please note that it fixes a potential remote DoS, which may happen for some (non-default) configurations. As such, users are highly encouraged to upgrade to this version.
http://www.rsyslog.com/changelog-for-7-6-6-v7-stable/
Download:
http://www.rsyslog.com/downloads/download-v7-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 7.6.6 (v7-stable)
Version 7.6.6 [v7.6-stable] 2014-09-30
- bugfix: potential abort when a message with PRI > 191 was processed
if the “pri-text” property was used in active templates, this could be abused to a remote denial of service from permitted senders see also: CVE-2014-3634 - bugfix: potential segfault on startup on 64 bit systems
This happened immediately on startup during config processing. Once rsyslog got past this stage, it could not happen. - bugfix: build problems on SuSe Linux
Thanks Andreas Stieger for the patch
Guardtime KSI services responding with error 401
Some people may have experienced trouble with log signing through Guardtime. The debug log shows, that the stamper service responds with a 401 error.
The Guardtime KSI service has been upgraded recently to mitigate DOS attacks by adding user authentication. Please contact Guardtime for more information.
The rsyslog testbench
In this video Rainer will give a quick rundown what the rsyslog testbench is and how tests can be added, as well as how single tests can be run without a full “make check”.
Please accept YouTube cookies to play this video. By accepting you will be accessing content from YouTube, a service provided by an external third party.
If you accept this notice, your choice will be saved and the page will refresh.
rsyslog 7.6.5 (v7-stable) released
This released provides an important regression fix, which rendered 7.6.4 unusable, as selector line evaluation was incorrect. Users of 7.6.4 are highly advised to upgrade to this version.
http://www.rsyslog.com/changelog-for-7-6-5-v7-stable/
Download:
http://www.rsyslog.com/downloads/download-v7-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
rsyslog 7.6.4 (v7-stable) released
This is a new release for the v7-stable branch. It contains a lot of bug fixes and patches. Several issues have been fixed, thus ensuring better stability and reliability. This is a recommended update for all v7 users.
http://www.rsyslog.com/changelog-for-7-6-4-v7-stable/
Download:
http://www.rsyslog.com/downloads/download-v7-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 7.6.4 (v7-stable)
Version 7.6.4 [v7.6-stable] 2014-09-12
- add –enable-generate-man-pages configure switch (default: enabled)
This forces generation of man pages, even if cached ones exists. This “fixes” a typical release tarball nit. While it is hackish, the benefit is clear given the history of failed tarball releases since we changed the cached man page handling. It was just too easy to get that wrong. - removed obsolete –disable-fsstnd configure option
Thanks to Thomas D. for alerting us.
Closes: https://github.com/rsyslog/rsyslog/issues/72 - permits to build against json-c 0.12
Unfortunately, json-c had an ABI breakage, so this is necessary. Note that versions prior to 0.12 had security issues (CVE-2013-6370, CVE-2013-6371) and so it is desirable to link against the new version.
Thanks to Thomas D. for the patch. Note that at least some distros have fixed the security issue in older versions of json-c, so this seems to apply mostly when building from sources. - new omfile default module parameters
- filecreatemode
- fileowner
- fileownernum
- filegroup
- filegroupnum
- dirowner
- dirownernum
- dirgroup
- dirgroupnum
Thanks to Karol Jurak for the patch.
- bugfix: memory leak in TCP TLS mode
- bugfix: imfile: if a state file for a different file name was set, that different file (name) was monitored instead of the configured one. Now, the state file is deleted and the correct file monitored.
closes: https://github.com/rsyslog/rsyslog/issues/103 - bugfix: using UUID property could cause segfault
- bugfix: mmutf8fix did not detect two invalid sequences
Thanks to Axel Rau for the patch. - bugfix: file descriptor leak with Guardtime signatures
When a .gtstate file is opened it is never closed. This is especially bad when dynafiles frequently get evicted from dynafile cache and be re-opened again. - bugfix: busy loop in tcp listener when running out of file descriptors
Thanks to Susant Sahani for the patch. - bugfix: mishandling of input modules not supporting new input instances
If they did not support this, accidently the output module part of the module union was written, leading to unpredictable results. Note: all core modules do support this interface, but some contributed or very old ones do not. - bugfix: double-free when ruleset() parser parameters were used
While unlikely, this could cause stability issues even after the config phase. - bugfix: output modules with parameters with multiple passing modes could caused strange behaviour including aborts
This was due to the fact that the action module only preserved and processed the last set passing mode. Note that this was not a problem for the plugins provided by the rsyslog git: none of them uses different passing modes.
Thanks to Tomas Heinrich for providing a very detailled bug report. - various fixes after coverty scan
These do not address issues seen in practice but those seen by the tool. Some of them may affect practical deployments.
Thanks to Tomas Heinrich for the patches. - bugfix imuxsock: “Last message repeated…” was not emitted at shutdown
The “Last message repeated…” notice didn’t get printed if rsyslog was shut down before the repetition was broken.
Thanks to Tomas Heinrich for the patch. - bugfix: make dist failed when GUARDTIME or LIBGCRYPT feature was disabled
- bugfix: mmjsonparse did not build with json-c < 0.10
This was a regression introduced some time in the past in order to support API changes in json-c. Now we check for the version and use proper code. - bugfix: mmanon did not properly anonymize IP addresses starting with ‘9’
Thanks to defa-at-so36.net for reporting this problem.
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=529
