rsyslog provides the ability to sign logs through GuardTime. You can verify the logs with a provided tool called rsgtutil. We have identified two reasons why the signing process could fail and thus verifying the log signature is not possible.

Note on legacy API: The old API libgt would create files with the endings .gtsig and .gtstate. These are different from the current and recommended API libksi.

When signing logs with rsyslog and GuardTime, three files will be created.

logfile # holds the log messages
logfile.ksisig # holds the hash chain data
logfile.ksistate # holds GuardTime signature hash

When verifying the logs through rsgtutil, the .ksistate file is needed because it holds the signature hash from GuardTime. Without this file, verification is not possible. Now there are two reasons why this file might not be present and thus a verification fails with:

# ./rsgtutil --verify --show-verified /var/log/logfile
error 1 (i/o error) processing file /var/log/logfile

1. rsyslog is still running

The .ksistate file is created when shutting down rsyslog. At shutdown, a signature request will be sent to GuardTime and will then be returned with the signature hash, which will be stored into this file. If rsyslog is still running, the file will not be created, resulting in an unsuccesful verification.

Even if the .ksistate file is present, it might not be current. So please make sure to stop rsyslog before trying to verify the logs.

2. Timeout on shutdown

On Ubuntu we had the problem, that upstart automatically kills the process after 5 seconds if the SIGTERM signal wasn’t fully executed until then. We now realized, that requesting the signature through GuardTime can take some time, depending on the system, thus exceeding the 5 second timeout and rsyslog getting killed before writing the .ksistate file. Again this resulted in unverifyable logs. This is also a problem for other parts of rsyslog (e.g. DA queues not being completely written to disk).

Upon changing the timeout value in the upstart script for rsyslog, we could circumvent the default timeout and rsyslog could fully complete the signature request and write the .ksistate file. This is at least valid for Ubuntu and may also be valid for other distributions.

To change the timeout go to this file:

/etc/init/rsyslog.conf

Now add

kill timeout 30

right after

expect fork

This should result in the following:

description     “system logging daemon”

start on filesystem
stop on runlevel [06]

expect fork
kill timeout 30
respawn

pre-start script
/lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script

script
. /etc/default/rsyslog
exec rsyslogd $RSYSLOGD_OPTIONS
end script

With the upstart script changed like this, rsyslog should not suffer from a too strict shutdown timeout and thus being able to complete the shutdown successfully. This change will also go in the startup and upstart scripts of the upcoming RPMs and Repo’s.

Side note

In the current versions, rsgtutil gives out a inappropriate error message. It currently states it is an I/O error. In reality it is an EOF error. In the upcoming versions (7.4.8, 7.5.8 and 8.1.4) this will be fixed.