signature

Error message when verifying signed logs

rsyslog provides the ability to sign logs through GuardTime. You can verify the logs with a provided tool called rsgtutil. We have identified two reasons why the signing process could fail and thus verifying the log signature is not possible.

Note on legacy API: The old API libgt would create files with the endings .gtsig and .gtstate. These are different from the current and recommended API libksi.

When signing logs with rsyslog and GuardTime, three files will be created.

logfile # holds the log messages
logfile.ksisig # holds the hash chain data
logfile.ksistate # holds GuardTime signature hash

When verifying the logs through rsgtutil, the .ksistate file is needed because it holds the signature hash from GuardTime. Without this file, verification is not possible. Now there are two reasons why this file might not be present and thus a verification fails with:

# ./rsgtutil --verify --show-verified /var/log/logfile
error 1 (i/o error) processing file /var/log/logfile

1. rsyslog is still running

The .ksistate file is created when shutting down rsyslog. At shutdown, a signature request will be sent to GuardTime and will then be returned with the signature hash, which will be stored into this file. If rsyslog is still running, the file will not be created, resulting in an unsuccesful verification.

Even if the .ksistate file is present, it might not be current. So please make sure to stop rsyslog before trying to verify the logs.

2. Timeout on shutdown

On Ubuntu we had the problem, that upstart automatically kills the process after 5 seconds if the SIGTERM signal wasn’t fully executed until then. We now realized, that requesting the signature through GuardTime can take some time, depending on the system, thus exceeding the 5 second timeout and rsyslog getting killed before writing the .ksistate file. Again this resulted in unverifyable logs. This is also a problem for other parts of rsyslog (e.g. DA queues not being completely written to disk).

Upon changing the timeout value in the upstart script for rsyslog, we could circumvent the default timeout and rsyslog could fully complete the signature request and write the .ksistate file. This is at least valid for Ubuntu and may also be valid for other distributions.

To change the timeout go to this file:

/etc/init/rsyslog.conf

Now add

kill timeout 30

right after

expect fork

This should result in the following:

description     “system logging daemon”

start on filesystem
stop on runlevel [06]

expect fork
kill timeout 30
respawn

pre-start script
/lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script

script
. /etc/default/rsyslog
exec rsyslogd $RSYSLOGD_OPTIONS
end script

With the upstart script changed like this, rsyslog should not suffer from a too strict shutdown timeout and thus being able to complete the shutdown successfully. This change will also go in the startup and upstart scripts of the upcoming RPMs and Repo’s.

Side note

In the current versions, rsgtutil gives out a inappropriate error message. It currently states it is an I/O error. In reality it is an EOF error. In the upcoming versions (7.4.8, 7.5.8 and 8.1.4) this will be fixed.

rsyslog 7.3.12 (v7-devel) released

We have just released v 7.3.12 of the rsyslog development branch.

This is a bugfixing release. It adresses several bugs for the solaris platform, omelasticsearch, a potential segfault in the guardtime signature provider and more. For more details review the changelog.

ChangeLog:

http://www.rsyslog.com/changelog-for-7-3-12-v7-devel/

Download:

http://www.rsyslog.com/rsyslog-7-3-12-v7-devel/

As always, feedback is appreciated.

Best regards,
Florian Riedl

[deprecated] How to sign log messages through signature provider Guardtime

Please note: This method is deprecated. Please refer to the new log signing method with KSI.

With rsyslog v7.3.9 we introduced the possibility to sign log messages through Guardtime, a signature provider. The process to enable this is relativey easy. And in the end you have your log files signed with a keyless signature that relies on hash functions through Guardtime. The signature functionality will be automatically loaded by omfile if so requested. It just requires that the signature provider itself is installed. For our RPMs and Ubuntu packages, it is available in the base packe. In the signature process a second file to your logfile will be created that has “.gtsig” as ending. This pair of files will later be needed to prove the integrity of your logfile.

In addition to rsyslog 7.3.9 or above you need “libgt”. The library is either available from Guardtime directly or from our git. If you installed rsyslog from our packages, libgt will be installed automatically.

When installing manually, you need to enable the signature function. The most basic configure command looks like this:

./configure --prefix=/usr --enable-guardtime

When rsyslog is installed, you can use the Guardtime signatures easily with a few additional configuration directives. For detailed information about the configuration directives, please review the manual. The correct action would look like this:

action(type="omfile" file="/var/log/logfile"
                sig.provider="gt"
                sig.timestampService="http://user:password@stamper.guardtime.net/gt-signingservice"
                # Please contact Guardtime for authentication details
                sig.keepTreeHashes="on" 
                sig.keepRecordHashes="on")

The directive sig.provider determines the provider that will be used. Currently, only Guardtime (gt) is available, but other providers might be added in the future. The other two options control the granularity of signature hashes at the cost of disk space. Though, when trying to detect a security breach, it might come in handy as it enables you to spot the location of the security breach. You will receive two files, that share the same name, but have a different extension.

/var/log/logfile
/var/log/logfile.gtsig

When having rsyslog installed you get a new tool called “rsgtutil”. This will help you check the integrity of your logfile in conjunction with the signature file. By issuing

tools/rsgtutil --verify --show-verified /var/log/logfile

you can make an easy check if the logfile is matching the stored hash. If the check was successful you will see it directly. If not, you will be notified as well and further investigation will be necessary.

Please note:

The Guardtime KSI service has been upgraded to mitigate DOS attacks by adding user authentication. Please contact Guardtime for more information.

Scroll to top