rsyslog

The rocket-fast system for log processing pipelines

rsyslog

The rocket-fast system for log processing pipelines

Search Results for: error 0

rsyslog error 2006

adisconteamUncategorizedNo Comments on rsyslog error 2006

Action has disabled itself.

This is a follow-up error. Additional error messages should appear in front of this message.

What it means is that an action had a failure that caused it to disable itself. If an action is disabled, it will no longer be executed. This event will most probably happen when the configuration file is being processed, right before actual message processing begins. However, the 2006 status code may be issued at any time.

When and why it is issued is purely up to the output plugin in question. In general, an action should disable itself only if there is no chance (not even a remote one) that action processing can correctly continue. For example, omsnmp disables itself if it can not initialize the net-snmp library, in which case no further processing is possible.

A disable action will remain disabled until either rsyslogd is restarted or (for older releases) a restart-type HUP is issued (which mimics a restart and thus is the same thing).

This is a stub entry: If you have questions please post a comment or visit the github issue tracker.

rsyslog error 2007

Rainer GerhardsUncategorized4 Comments on rsyslog error 2007

What does it mean?

This is a generic error message that unfortunately can happen in a number of cases.

In practice, it is often associated with suspension of actions. Then it comes with a text like

action “action 17” suspended

The number behind action changes.

How to solve it?

A frequent case for this error message on Debian-based distributions (like raspbian) is that rsyslog.conf contains the instruction to write to the xconsole pipe, but this pipe is never read. If so, you can simply delete these lines to remove the error message. These lines are usually found at the end of rsyslog.conf.

For other error message, it probably is a good idea to check rsyslog’s issue tracker at github and file a new issue if you can’t find a related case.

Note: we try to keep this page update if we see other frequent causes of this error.

rsyslog error reporting improved

jgerhardsNews, Top News,

Rsyslog provides many up-to-the point error messages for config file and operational problems. These immensly helps when troubleshooting issues. Unfortunately, many users never see them. The prime reason is that most distros do never log syslog.* messages and so they are just throw away and invisible to the user. While we have been trying to make distros change their defaults, this has not been very successful. The result is a lot of user frustration and fruitless support work for the community — many things can very simple be resolved if only the error message is seen and acted on.

We have now changed our approach to this. Starting with v8.21, rsyslog now by default logs its messages via the syslog API instead of processing them internally. This is a big plus especially on systems running systemd journal: messages from rsyslogd will now show up when giving

$ systemctl status rsyslog.service

This is the place where nowadays error messages are expected and this is definitely a place where the typical administrator will see them. So while this change causes the need for some config adjustment on few exotic installations (more below), we expect this to be something that will generally improve the rsyslog user experience.

Along the same lines, we will also work on some better error reporting especially for TLS and queue-related issues, which turn out high in rsyslog suport discussions.

Some fine details on the change of behaviour:

Note: you can usually skip reading the rest of this post if you run only a single instance of rsyslog and do so with more or less default configuration.

The new behaviour was actually available for longer, It needed to be explicitly turned on in rsyslog.conf via

global(processInternalMessages="off")

Of course, distros didn’t do that by default. Also, it required rsyslog to be build with liblogging-stdlog, what many distros do not do. While our intent when we introduced this capability was to provide the better error logging we now have, it simply did not turn out in practice. The original approach was that it was less intrusive. The new method uses the native syslog() API if liblogging-stdlog is not available, so the setting always works (we even consider moving away from liblogging-stdlog, as we see this wasn’t really adopted). In essence, we have primarily changed the default setting for the “processInternalMessages” parameter. This means that by default, internal messages are no longer logged via the internal bridge to rsyslog but via the syslog() API call [either directly or
via liblogging). For the typical single-rsyslogd-instance installation this is mostly unnoticable (except for some additional latency). If multiple instances are run, only the “main” (the one processing system log messages) will see all messages. To return to the old behaviour, do either of those two:

  1. add in rsyslog.conf:
    global(processInternalMessages="on")
  2. export the environment variable RSYSLOG_DFLT_LOG_INTERNAL=1
    This will set a new default – the value can still be overwritten via rsyslog.conf (method 1). Note that the environment variable must be set in your startup script (which one is depending on your init system or systemd configuration).

Note that in most cases even in multiple-instance-setups rsyslog error messages were thrown away. So even in this case the behaviour is superior to the previous state – at least errors are now properly being recorded. This also means that even in multiple-instance-setups it often makes sense to keep the new default!

Error message when verifying signed logs

Adiscon SupportFAQ, , , ,

rsyslog provides the ability to sign logs through GuardTime. You can verify the logs with a provided tool called rsgtutil. We have identified two reasons why the signing process could fail and thus verifying the log signature is not possible.

Note on legacy API: The old API libgt would create files with the endings .gtsig and .gtstate. These are different from the current and recommended API libksi.

When signing logs with rsyslog and GuardTime, three files will be created.

logfile # holds the log messages
logfile.ksisig # holds the hash chain data
logfile.ksistate # holds GuardTime signature hash

When verifying the logs through rsgtutil, the .ksistate file is needed because it holds the signature hash from GuardTime. Without this file, verification is not possible. Now there are two reasons why this file might not be present and thus a verification fails with:

# ./rsgtutil --verify --show-verified /var/log/logfile
error 1 (i/o error) processing file /var/log/logfile

1. rsyslog is still running

The .ksistate file is created when shutting down rsyslog. At shutdown, a signature request will be sent to GuardTime and will then be returned with the signature hash, which will be stored into this file. If rsyslog is still running, the file will not be created, resulting in an unsuccesful verification.

Even if the .ksistate file is present, it might not be current. So please make sure to stop rsyslog before trying to verify the logs.

2. Timeout on shutdown

On Ubuntu we had the problem, that upstart automatically kills the process after 5 seconds if the SIGTERM signal wasn’t fully executed until then. We now realized, that requesting the signature through GuardTime can take some time, depending on the system, thus exceeding the 5 second timeout and rsyslog getting killed before writing the .ksistate file. Again this resulted in unverifyable logs. This is also a problem for other parts of rsyslog (e.g. DA queues not being completely written to disk).

Upon changing the timeout value in the upstart script for rsyslog, we could circumvent the default timeout and rsyslog could fully complete the signature request and write the .ksistate file. This is at least valid for Ubuntu and may also be valid for other distributions.

To change the timeout go to this file:

/etc/init/rsyslog.conf

Now add

kill timeout 30

right after

expect fork

This should result in the following:

description     “system logging daemon”

start on filesystem
stop on runlevel [06]

expect fork
kill timeout 30
respawn

pre-start script
/lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script

script
. /etc/default/rsyslog
exec rsyslogd $RSYSLOGD_OPTIONS
end script

With the upstart script changed like this, rsyslog should not suffer from a too strict shutdown timeout and thus being able to complete the shutdown successfully. This change will also go in the startup and upstart scripts of the upcoming RPMs and Repo’s.

Side note

In the current versions, rsgtutil gives out a inappropriate error message. It currently states it is an I/O error. In reality it is an EOF error. In the upcoming versions (7.4.8, 7.5.8 and 8.1.4) this will be fixed.

Scroll to top