Rsyslog Windows Agent Released
The rsyslog Windows agent enables customers to integrate Windows Event Logs into their central rsyslog-based logging system. Even more, it supports acting as a syslog relay as well. This enables small branch offices only runnig Windows to provide local syslog sources to the central server as well.
We have released the first full-featured version of this product today. It is based on Adiscon’s mature suite of Windows-to-syslog tools. Most importantly, it provides the ability to fully extract Windows Event Log data, including local resolution of user IDs, SIDs, GUIDs and similiar objects. It not only supports the current Windows Event Log format but also the legacy (pre Windows-2008/Vista) event log system. The technology behind this tool is already in use at thousands of customer sites for many years (in the form of, for example, EventReporter and other Adiscon products).
The rsyslog Agent for Windows has been highly optimized for use with rsyslog at the back end. For example, it supports mutually-authenticated, TLS encrypted syslog as well as the RELP protocol for reliable delivery. Also, it provides an initial implementation of cee-enhanced syslog, and is the world’s first-ever solution to do so (details can be found in the configuration guide for cee-enhanced syslog).
By purchasing rsyslog Agent for Windows customers not only get first class event log integration, they also help fund further improving the rsyslog solution as whole. The package even contains limited rsyslog professional services.
A trial version can be downloaded from the rsyslog Windows agent page.
TCP syslog / RFC5424 log4j appender
As part of a training project, we created a small log4j appender supporting TCP syslog and RFC5424. Most importantly, it is capable of formatting stack traces as a single syslog messages (NOT the usual bunch of multiple malformed messages). The work is based on the syslog4j implementation, which did not work for us (our fault? ;)) and so we extended this framework. Note that the code has never been used in practice by us, so there may be some risk associated – but that should be fairly easy to asses, given that the source is pretty small.
The download of the RFC5424 syslog tcp appender contains it all: doc, jar files as well as the source.
Feedback and usage reports are appreciated, please send mail to ugerhards@adiscon.com (if you run into troubles, you can also use the same email address to discuss the problem!).
rsyslog statistics counter
Rsyslog supports statistic counters via the impstats module. It is important to know that impstats and friends only provides an infrastructure where core components and plugins can register statistics counter. This FAQ entry tries to describe all counters available, but please keep in mind that there may exist that we do not know about.
When interpreting rsyslog statistics, please keep in mind that statistics records are processed as regular syslog messages. As such, the statistics messages themselves increment counters when they are emitted via the regular syslog stream, which is the default (and so counters keep slowly increasing even if there is absolutely no other traffic). Also keep in mind that a busy rsyslog system is very dynamic. Most importantly, this means that the counters may not be 100% consistent, but some slight differences may exist. Avoiding such inconsistencies would be possible only at the price of a very tight locking discipline, which would cause serious performance bottlenecks. Thus, this is not done. Finally, though extremely unlikely, some counters may experience an overflow and restart at 0 for that reasons. However, most counters are 64-bit, so this is extremely unlikely. Those which are not 64 bit are typically taken from some internal data structure that uses lower bits for performance reasons and guards against overflow.
The listing starts with the core component or plugin that creates the counters and than specifies various counters that exist for the sub-entities. The listing below is extended as new counters are added. Some counters probably do not exist in older releases of rsyslog.
Below you can find all available core components and plugins. Please note that every core component or plugin are linked to a information site.
Queue
Actions
PLUGINS
LogTools 0.1.0 Released
We are happy to announce the initial public release of LogTools. This package aims at providing useful tools for log file manipulation and processing. The initial release focuses on a core ingredient, the tamper-proof log store. Besides that, the well-documented format is designed for long-term archiving. Both features together make it very useful for storing both valuable audit information as well as log files potentially meant as evidence in court.
Both the source tarball and an experimental Debian package are available via the LogTools download page.
The log store is tamper-proof by using a chain of hashes. This provides tamper protection if the last hash of the hash chain is regularly saved. In upcoming releases, the hash chain will become protected by digital signatures using strong cryptography. With that, it is no longer necessary (but still beneficial) to keep hashes in a separate location (that would guard even against a compromised key).
The log store is long-term-archival-friendly because its documented format is designed to remain easy to read and interpret even in many years: it is text-based, clearly documented (in the man page or, for example, here) and easily extensible. To support extensibility, it supports different record types and a cryptographic field that can be extended by type specifiers as well (this is not used initially because it is not needed – but the capability is already present). It is our pledge that this format will remain readable while it is being extended in the future.
So LogStore format is a perfect solution for tamper-protected texttual data that must last. And note that we say “textual data”, not “syslog messages” or even “log messages”. No matter what it is, if it is in text format, LogStore can protect it!
LogTools are also a good Linux citizen, following the Unix tradition of small tools that do one job and do it well. With the initial release, two filter-type programs (logreader and logwriter) have been released. They permit to create and read log stores (including a functionality to extract the last used hash for easy transmission). These tools can perfectly be used inside processing pipes. That, among others, means that LogStore data is perfectly “greppable”. Integration with other tools is simple. For example, to make rsyslog write LogStore format, rsyslog’s omprog output module can be utilized. All tools that can emit data to stdout can also be used to write LogStore format. So immediately there is a myriad of options available.
LogTools is an active project that will be further extend with new tools with the same philosophy. One of the next steps planned is to integrate liblognorm techology for easy normalization of incoming semi-structured data. We also appreciate suggestions, so please make yourself heard.
What are “Trusted Properties”?
Rsyslog can annotate messages from system log sockets (via imuxsock) with so-called “Trusted Properties”. These are message properties not provided by the logging client application itself, but rather obtained from the system. As such, they can not be faked by the user application and are trusted in this sense. This bases on the similar idea introduced in systemd.
The following trusted properties are available:
| _UID | The userid under which the logging process is being executed. |
| _GID | The group id under which the logging process is being executed. |
| _PID | The PID of the logging process. Note that this PID, if configured, is also put into the syslog tag. |
| _EXE | Path to the binary that is logging |
| _COMM | The name (as visible by top) of the logging process. |
| _EXE | The full command line of the logging process. Note that this string can contain spaces, thus it is always provided in quoted form. |
Property names are deliberately based on property names in the systemd proposal. Additional properties can be added upon request. User- and Group-IDs as well as the PID are obtained via SCM_CREDENTIALS and as such should be always available on a relatively recent Linux system. The other properties are obtained from the /proc virtual file system. Note that this can be somewhat racy. Most importantly, these properties can not be obtained if the process has already been terminated when the message is being processed by rsyslogd. This can occur for very quick processes (logger being a prime example) and has happened in practice (for the same reason, it is theoretical possible that invalid information is reported, if the PID are reused extremely quickly – this is so unlikely we did not care about this case).
All annotations are currently (Nov 2011) placed at the end of the message, starting with ” @[” (the first character being a space). The annotation is ended with “]”. This is done to keep consistent with legacy syslog implementations. However, the format will probably changed and is intended to be moved over to a RFC5424 structured data item. Also, the property names were currently chosen for best fit with systemd. As systemd does not follow the relevant standards, we may at later time change to standard names, once they become fully available. So please be prepared for some future change if you begin to use this feature. You may guess that we will preserve legacy format when doing changes, but we do not promise that ;)
A sample of a message with trusted properties looks as follows:
2011-11-29T16:57:13.634852+01:00 testhost su: pam_unix(su-l:session): session opened for user root by examp(uid=677) @[_PID=5933 _UID=0 _GID=677 _COMM=su _EXE=/bin/su _CMDLINE="su - "]
The annotation is given in boldface. Note that this is all on one line. It is broken across several lines just for readability.
This feature is available starting with version 5.9.4 in the v5 rsyslog branches.
rsyslog client for Windows
As it currently looks, Adiscon will most probably create a specialised Windows client for rsyslog. This will be based on Adiscon’s MonitorWare technology and provide excellent and high speed integration of Windows clients into a rsyslog infrastructure. While the idea has somewhat matured, we are currently thinking about the details. Expect more information as discussions progress!
In the mean time, you may want to have a look at Adiscon’s EventReporter, which provides excellent Windows-to-rsyslog event log forwarding.
Sending messages with tags larger than 32 characters
The relevant syslog RFCs 3164 and 5424 limit the syslog tag to 32 characters max. Messages with larger tag length are malformed and may be discarded by receivers. Anyhow, some folks sometimes need to send tags longer than permitted.
To do so, a new template must be created and used when sending. The simplest way is to start with the standard forwarding template. The standard templates are hardcoded inside rsyslog. Thus they do not show up in your configuration file (but you can obtain them from the source, of course). In 5.8.6, the forwarding template is defined as follows:
template (name="ForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%
%syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%")NOTE: all templates are on one line in rsyslog.conf. They are broken here for readability.
This template is RFC-compliant. Now look at the part in red. It specifies the tag. Note that, via the property replacer, it is restricted to 32 characters (from position 1 to position 32 inclusive). This is what you need to change. To remove the limit … just remove it ;-) This leads to a template like this:
template (name="LongTagForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%")
Note that I have renamed the template in order to avoid conflicts with build-in templates. As it is a custom template, it is not hardcoded, so you need to actually configure it in your rsyslog.conf. Then, you need to use that template if you want to send messages to a remote host. This can be done via the usual way. Let’s assume you use legacy plain TCP syslog. Then the line looks as follows:
action(type="omfwd"
Target="server.example.net"
Port="10514"
Protocol="tcp"
Template="LongTagForwardFormat"
)This will bind the forwarding action to the newly defined template. Now tags of any size will be forwarded. Please keep in mind that receivers may have problems with large tags and may truncate them or drop the whole message. So check twice that the receiver handles long tags well.
Rsyslog supports tags to a build-defined maximum. The current (5.8.6) default is 511 characters, but this may be different if you install from a package, use a newer version of rsyslog or use sources obtained from someone else. So double-check.
syslog classification with rsyslog
Starting with rsyslog v6, we support log normalization via liblognorm. This is already very cool, because message content can be extrated and normalized via simple so-called rulebases of message samples. Yesterday a new version Continue reading “syslog classification with rsyslog”
CentOS 5: how to compile rsyslog from git
If you compile rsyslog from git on CentOS 5 you run into the trouble that
autoreconf -fvi
fails, telling you that autotools 2.59 provided by CentOS is too old.
The solution to this problem is to install version 2.61 of autotools. You will probably not want to overwrite the default CentOS package, so you should install it in some alternate location (e.g. /opt) and include that location in the topmost spot of your path. all this can be done as follows:
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.61.tar.bz2
tar xjf autoconf-2.61.tar.bz2
cd autoconf-2.61
./configure --prefix=/opt
make
sudo make install
Then, you need to change your path, for example as follows (this is obviously a temporary alteration!):
export PATH=/opt/bin:$PATH
After these changes, you should be able to run autoreconf for rsyslog.
Sending Messages to a Remote Syslog Server
In this recipe, we forward messages from one system to another one. Typical use cases are:
- the local system does not store any messages (e.g. has not sufficient space to do so)
- there is a (e.g. legal) requirement to consolidate all logs on a single system
- the server may run some advanced alerting rules, and needs to have a full picture or network activity to work well
- you want to get the logs to a different system in a different security domain (to prevent attackers from hiding their tracks)
- and many more …
In our case, we forward all messages to the remote system. Note that by applying different filters, you may only forward select entries to the remote system. Also note that you can include as many forwarding actions as you like. For example, if you need to have a backup central server, you can simply forward to both of them, using two different forwarding actions.
To learn how to configure the remote server, see recipe Receiving Messages from a Remote System.
Config Statements
# this is the simplest forwarding action:
*.* action(type="omfwd" target="192.0.2.1" port="10514" protocol="tcp")
# it is equivalent to the following obsolete legacy format line:
*.* @@192.0.2.1:10514 # do NOT use this any longer!
# Note: if the remote system is unreachable, processing will
# block here and discard messages after a while
# so a better use is
*.* action(type="omfwd" target="192.0.2.2" port="10514" protocol="tcp"
action.resumeRetryCount="100"
queue.type="linkedList" queue.size="10000")
# this will de-couple the sending from the other logging actions,
# and prevent delays when the remote system is not reachable. Also,
# it will try to connect 100 times before it discards messages as
# undeliverable.
# the rest below is more or less a plain vanilla rsyslog.conf as
# many distros ship it - it's more for your reference...
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.logThings to think about
You need to select the protocol best suitable for your use case. If in doubt, TCP is a decent choice. This recipe uses TCP for that reason.
TCP forwarding is a build-in capability and always present. As such, no plugin needs to be loaded. The target can be specified by DNS name or IP address. Use IP addresses for most robust operations. If you use a DNS name and name resolution fails, forwarding may be disabled for some time. DNS resolution typically fails on the DNS server itself during system startup.
In this example, we forward to port 10514. We could as well remove the port=”…” parameter from the configuration, which would result in the default port being used. However, you need to specify the port address on the server in any case. So it is strongly advised to use an explicit port number to make sure that client and server configuration match each other (if they used different ports, the message transfer would not work.