imfile: Text File Input Module¶

Module Name:	imfile
Author:	Rainer Gerhards <rgerhards@adiscon.com>

Purpose¶

This module provides the ability to convert any standard text file into a syslog message. A standard text file is a file consisting of printable characters with lines being delimited by LF.

The file is read line-by-line and any line read is passed to rsyslog’s rule engine. The rule engine applies filter conditions and selects which actions needs to be carried out. Empty lines are not processed, as they would result in empty syslog records. They are simply ignored.

As new lines are written they are taken from the file and processed. Depending on the selected mode, this happens via inotify or based on a polling interval. Especially in polling mode, file reading doesn’t happen immediately. But there are also slight delays (due to process scheduling and internal processing) in inotify mode.

The file monitor supports file rotation. To fully work, rsyslogd must run while the file is rotated. Then, any remaining lines from the old file are read and processed and when done with that, the new file is being processed from the beginning. If rsyslogd is stopped during rotation, the new file is read, but any not-yet-reported lines from the previous file can no longer be obtained.

When rsyslogd is stopped while monitoring a text file, it records the last processed location and continues to work from there upon restart. So no data is lost during a restart (except, as noted above, if the file is rotated just in this very moment).

Notable Features¶

Metadata
State Files
WildCards
presentation on using wildcards with imfile

Configuration¶

Note

Parameter names are case-insensitive; CamelCase is recommended for: readability.

Module Parameters¶

Parameter	Summary
Mode	Selects inotify, polling, or fen mode for file monitoring.
readTimeout	Sets the multi-line read timeout in seconds; module value provides the default.
deleteStateOnFileMove	Removes the state file when a monitored file is moved or renamed.
timeoutGranularity	Interval in seconds for checking multi-line read timeouts.
sortFiles	Processes files in sorted order when enabled.
PollingInterval	Seconds between file scans in polling mode; default `10`.
statefile.directory	Sets a dedicated directory for imfile state files.

Input Parameters¶

Parameter	Summary
File	Specifies the absolute file path to monitor; wildcards are allowed.
Tag	Assigns a tag to messages read from the file; include a colon if desired.
Facility	Sets the syslog facility for messages read from this file.
Severity	Sets the syslog severity for lines read from the file.
PersistStateInterval	Controls how often the state file is written based on processed line count.
startmsg.regex	Regex that marks the beginning of a new message for multiline processing.
endmsg.regex	Regex that identifies the end of a multi-line message.
readTimeout	Sets the multi-line read timeout in seconds; module value provides the default.
readMode	Selects a built-in method for detecting multi-line messages (0–2).
escapeLF	Escapes embedded LF characters in multi-line messages as `#012` when enabled.
escapeLF.replacement	Overrides the default LF escape sequence when `escapeLF` is `on`.
MaxLinesAtOnce	Legacy polling-mode limit on how many lines are read before switching files.
MaxSubmitAtOnce	Sets the maximum batch size that imfile submits at once.
deleteStateOnFileDelete	Removes the state file when the monitored file is deleted.
deleteStateOnFileMove	Removes the state file when a monitored file is moved or renamed.
Ruleset	Binds the input instance to a specific ruleset.
addMetadata	Turns metadata addition on or off; enabled by default when wildcards are used.
addCeeTag	Adds the `@cee:` cookie to the message when enabled.
reopenOnTruncate	Experimental: reopen the input file if it was truncated without changing its inode.
MaxLinesPerMinute	Limits how many lines per minute are enqueued as messages; excess lines are discarded.
MaxBytesPerMinute	Drops messages when the total bytes per minute exceed the specified limit.
trimLineOverBytes	Truncates lines longer than the specified number of bytes.
freshStartTail	On first start, seeks to the end of existing files and processes only new lines.
discardTruncatedMsg	Discards the truncated part of an overlong message instead of processing it.
msgDiscardingError	Controls whether an error is logged when a message is truncated.
needParse	Forces messages to pass through configured parser modules.
persistStateAfterSubmission	Persists state file information after each batch submission for robustness.
ignoreOlderThan	Ignores newly discovered files older than the specified number of seconds.
stateFile	Deprecated; sets a fixed state file name for this input.

Metadata¶

The imfile module supports message metadata. It supports the following data items:

filename

Name of the file where the message originated from. This is most useful when using wildcards inside file monitors, because it then is the only way to know which file the message originated from. The value can be accessed using the %$!metadata!filename% property. Note: For symlink-ed files this does not contain name of the actual file (source of the data) but name of the symlink (file which matched configured input).
fileoffset

Offset of the file in bytes at the time the message was read. The offset reported is from the start of the line. This information can be useful when recreating multi-line files that may have been accessed or transmitted non-sequentially. The value can be accessed using the %$!metadata!fileoffset% property.

Metadata is only present if enabled. By default it is enabled for input() statements that contain wildcards. For all others, it is disabled by default. It can explicitly be turned on or off via the addMetadata input() parameter, which always overrides the default.

State Files¶

Rsyslog must keep track of which parts of the monitored file are already processed. This is done in so-called “state files” that are created in the rsyslog working directory and are read on startup to resume monitoring after a shutdown. The location of the rsyslog working directory is configurable via the global(workDirectory) advanced format parameter.

Note: The PersistStateInterval parameter must be set, otherwise state files will NOT be created.

Rsyslog automatically generates state file names. These state file names will begin with the string imfile-state: and be followed by some suffix rsyslog generates.

There is intentionally no more precise description of when state file naming, as this is an implementation detail and may change as needed.

Note that it is possible to set a fixed state file name via the deprecated stateFile parameter. It is suggested to avoid this, as the user must take care of name clashes. Most importantly, if “stateFile” is set for file monitors with wildcards, the same state file is used for all occurrences of these files. In short, this will usually not work and cause confusion. Upon startup, rsyslog tries to detect these cases and emit warning messages. However, the detection simply checks for the presence of “*” and as such it will not cover more complex cases.

Note that when the global(workDirectory) advanced format parameter is points to a non-writable location, the state file will not be generated. In those cases, the file content will always be completely re-sent by imfile, because the module does not know that it already processed parts of that file. if The parameter is not set to all, it defaults to the file system root, which may or may not be writable by the rsyslog process.

WildCards¶

As of rsyslog version 8.25.0 and later, wildcards are fully supported in both directory names and filenames. This allows for flexible file monitoring configurations.

Examples of supported wildcard usage:

/var/log/*.log (all .log files in /var/log)
/var/log/*/*.log (all .log files in immediate subdirectories of /var/log)
/var/log/app*/*/*.log (all .log files in subdirectories of directories starting with “app” in /var/log)

All matching files in all matching subfolders will be monitored.

Note

Using complex wildcard patterns, especially those that match many directories and files, may have an impact on performance. It’s advisable to use the most specific patterns possible.

Historically, versions prior to 8.25.0 had limitations, supporting wildcards only in the filename part and not in directory paths.

Caveats/Known Bugs¶

symlink may not always be properly processed

Configuration Examples¶

The following sample monitors two files. If you need just one, remove the second one. If you need more, add them according to the sample ;). This code must be placed in /etc/rsyslog.conf (or wherever your distro puts rsyslog’s config files). Note that only commands actually needed need to be specified. The second file uses less commands and uses defaults instead.

module(load="imfile" PollingInterval="10") #needs to be done just once

# File 1
input(type="imfile"
      File="/path/to/file1"
      Tag="tag1"
      Severity="error"
      Facility="local7")

# File 2
input(type="imfile"
      File="/path/to/file2"
      Tag="tag2")

# ... and so on ... #

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project