imjournal: Systemd Journal Input Module¶

Module Name:	imjournal
Author:	Jiri Vymazal <jvymazal@redhat.com> (This module is not project-supported)
Available since:	7.3.11

Purpose¶

Provides the ability to import structured log messages from systemd journal to syslog.

Note that this module reads the journal database, what is considered a relatively performance-intense operation. As such, the performance of a configuration utilizing this module may be notably slower than when using imuxsock. The journal provides imuxsock with a copy of all “classical” syslog messages, however, it does not provide structured data. Only if that structured data is needed, imjournal must be used. Otherwise, imjournal may simply be replaced by imuxsock, and we highly suggest doing so.

We suggest to check out our short presentation on rsyslog journal integration to learn more details of anticipated use cases.

Warning: Some versions of systemd journal have problems with database corruption, which leads to the journal to return the same data endlessly in a tight loop. This results in massive message duplication inside rsyslog probably resulting in a denial-of-service when the system resources get exhausted. This can be somewhat mitigated by using proper rate-limiters, but even then there are spikes of old data which are endlessly repeated. By default, ratelimiting is activated and permits to process 20,000 messages within 10 minutes, what should be well enough for most use cases. If insufficient, use the parameters described below to adjust the permitted volume. It is strongly recommended to use this plugin only if there is hard need to do so.

Notable Features¶

statistics counters

Module Parameters¶

Note

Parameter names are case-insensitive; CamelCase is recommended for readability.

Parameter	Summary
PersistStateInterval	Saves the journal cursor after every N messages.
FileCreateMode	Sets the octal permission mode for the state file.
StateFile	Specifies path to the state file holding the journal cursor.
Ratelimit.Interval	Time window in seconds for rate limiting; 0 disables the limit.
Ratelimit.Burst	Maximum messages accepted within each rate-limit window.
IgnorePreviousMessages	Starts reading only new journal entries when no state file exists.
DefaultSeverity	Fallback severity used if a journal entry lacks `SYSLOG_PRIORITY`.
DefaultFacility	Fallback facility used if a journal entry lacks `SYSLOG_FACILITY`.
UsePidFromSystem	Use `_PID` instead of `SYSLOG_PID` as the process identifier.
UsePid	Selects which journal PID field to use: `syslog`, `system`, or `both`.
IgnoreNonValidStatefile	Ignores corrupt state files and restarts reading from the beginning.
WorkAroundJournalBug	Legacy flag with no current effect; retained for compatibility.
FSync	Force `fsync()` on the state file to guard against crash corruption.
Remote	Also read journal files from remote sources.
defaultTag	Provides a default tag when both `SYSLOG_IDENTIFIER` and `_COMM` are missing.

Input Parameters¶

Parameters specific to the input module.

Parameter	Summary
Main	Runs the input’s ruleset on the main thread and stops reading if outputs block.

Statistic Counter¶

This plugin maintains statistics for each listener and for each worker thread. The listener statistic is named “imjournal”.

The following properties are maintained for each listener:

read - total number of message read from journal since startup.
submitted - total number of messages submitted to main queue after reading from journal for processing since startup. All records may not be submitted due to rate-limiting.
discarded - total number of messages that were read but not submitted to main queue due to rate-limiting.
failed - total number of failures to read messages from journal.
poll_failed - total number of journal poll failures.
rotations - total number of journal file rotations.
recovery_attempts - total number of recovery attempts by imjournal after unknown errors by closing and re-opening journal.
ratelimit_discarded_in_interval - number of messages discarded due to rate-limiting within configured rate-limiting interval.
disk_usage_bytes - total size of journal obtained from sd_journal_get_usage().

Here is an example output of corresponding imjournal impstat message, which is produced by loading imjournal with default rate-limit interval and burst and running a docker container with log-driver as journald that spews lots of logs to stdout:

Jun 13 15:02:48 app1-1.example.com rsyslogd-pstats: imjournal: origin=imjournal submitted=20000 read=216557
discarded=196557 failed=0 poll_failed=0 rotations=6 recovery_attempts=0 ratelimit_discarded_in_interval=196557
disk_usage_bytes=106610688

Although these counters provide insight into imjournal end message submissions to main queue as well as losses due to rate-limiting or other problems to extract messages from journal, they don’t offer full visibility into journal end issues. While these counters measure journal rotations and disk usage, they do not offer visibility into message loss due to journal rate-limiting. sd_journal_* API does not provide any visibility into messages that are discarded by the journal due to rate-limiting. Journald does emit a syslog message when log messages cannot make it into the journal due to rate-limiting:

Jun 13 15:50:32 app1-1.example.com systemd-journal[333]: Suppressed 102 messages from /system.slice/docker.service

Such messages can be processed after they are read through imjournal to get a signal for message loss due to journal end rate-limiting using a dynamic statistics counter for such log lines with a rule like this:

dyn_stats(name="journal" resettable="off")
if $programname == 'journal' and $msg contains 'Suppressed' and $msg contains 'messages from' then {
        set $.inc = dyn_inc("journal", "suppressed_count");
}

Caveats/Known Bugs:¶

As stated above, a corrupted systemd journal database can cause major problems, depending on what the corruption results in. This is beyond the control of the rsyslog team.
imjournal does not check if messages received actually originated from rsyslog itself (via omjournal or other means). Depending on configuration, this can also lead to a loop. With imuxsock, this problem does not exist.

Build Requirements:¶

Development headers for systemd, version >= 197.

Example 1¶

The following example shows pulling structured imjournal messages and saving them into /var/log/ceelog.

module(load="imjournal" PersistStateInterval="100"
       StateFile="/path/to/file") #load imjournal module
module(load="mmjsonparse") #load mmjsonparse module for structured logs

template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%\n" ) #template for messages

action(type="mmjsonparse")
action(type="omfile" file="/var/log/ceelog" template="CEETemplate")

Example 2¶

The following example is the same as Example 1, but with the input module.

ruleset(name="imjournal-example" queue.type="direct"){
 action(type="mmjsonparse")
 action(type="omfile" file="/var/log/ceelog" template="CEETemplate")
}

input(
 type="imjournal"
 ruleset="imjournal-example"
 main="on"
)

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project