omfile: File Output Module¶

Module Name:	omfile
Author:	Rainer Gerhards <rgerhards@adiscon.com>

Purpose¶

The omfile plug-in provides the core functionality of writing messages to files residing inside the local file system (which may actually be remote if methods like NFS are used). Both files named with static names as well files with names based on message content are supported by this module.

Notable Features¶

Statistic Counter

Configuration Parameters¶

Omfile is a built-in module that does not need to be loaded. In order to specify module parameters, use

module(load="builtin:omfile" ...parameters...)

Note that legacy parameters do not affect new-style RainerScript configuration objects. See basic configuration structure doc to learn about different configuration languages in use by rsyslog.

Note

Parameter names are case-insensitive; CamelCase is recommended for readability.

General Notes¶

As can be seen in the parameters below, owner and groups can be set either by name or by direct id (uid, gid). While using a name is more convenient, using the id is more robust. There may be some situations where the OS is not able to do the name-to-id resolution, and these cases the owner information will be set to the process default. This seems to be uncommon and depends on the authentication provider and service start order. In general, using names is fine.

Module Parameters¶

Parameter	Summary
Template	Sets the template to be used for this action.
dirCreateMode	Sets the creation mode for directories that are automatically generated.
fileCreateMode	Specifies the file system permissions for newly created log files.
fileOwner	Set the file owner for files newly created.
fileOwnerNum	Set the file owner for files newly created.
fileGroup	Set the group for files newly created.
fileGroupNum	Set the group for files newly created.
dirOwner	Set the file owner for directories newly created.
dirOwnerNum	Set the file owner for directories newly created.
dirGroup	Set the group for directories newly created.
dirGroupNum	Set the group for directories newly created.
dynafile.donotsuspend	This permits SUSPENDing dynafile actions.
compression.driver	For compressed operation (“zlib mode”), this permits to set the compression driver to be used.
compression.zstd.workers	In zstd mode, this enables to configure zstd-internal compression worker threads.

Action Parameters¶

Note that one of the parameters file or dynaFile must be specified. This selects whether a static or dynamic file (name) shall be written to.

Parameter	Summary
Template	Sets the template to be used for this action.
fileOwner	Set the file owner for files newly created.
fileOwnerNum	Set the file owner for files newly created.
fileGroup	Set the group for files newly created.
fileGroupNum	Set the group for files newly created.
dirOwner	Set the file owner for directories newly created.
dirOwnerNum	Set the file owner for directories newly created.
dirGroup	Set the group for directories newly created.
dirGroupNum	Set the group for directories newly created.
File	This creates a static file output, always writing into the same file.
dynaFile	For each message, the file name is generated based on the given template.
closeTimeout	Specifies after how many minutes of inactivity a file is automatically closed.
dynaFileCacheSize	This parameter specifies the maximum size of the cache for dynamically-generated file names (dynafile= parameter).
zipLevel	If greater than 0, turns on gzip compression of the output file.
veryRobustZip	If zipLevel is greater than 0, then this setting controls if extra headers are written to make the resulting file extra hardened against malfunction.
flushInterval	Defines, in seconds, the interval after which unwritten data is flushed.
asyncWriting	If turned on, the files will be written in asynchronous mode via a separate thread.
flushOnTXEnd	Omfile has the capability to write output using a buffered writer.
ioBufferSize	Size of the buffer used to writing output data.
fileCreateMode	Specifies the file system permissions for newly created log files.
dirCreateMode	Sets the creation mode for directories that are automatically generated.
failOnChOwnFailure	This option modifies behaviour of file creation.
createDirs	Create directories on an as-needed basis.
sync	Enables file syncing capability of omfile.
sig.Provider	Selects a signature provider for log signing.
cry.Provider	Selects a crypto provider for log encryption.
rotation.sizeLimit	This permits to set a size limit on the output file.
rotation.sizeLimitCommand	Configures the script invoked when rotation.sizeLimit is exceeded.

Statistic Counter¶

This plugin maintains statistics for each dynafile cache. Dynafile cache performance is critical for overall system performance, so reviewing these counters on a busy system (especially one experiencing performance problems) is advisable. The statistic is named “dynafile cache”, followed by the template name used for this dynafile action.

The following properties are maintained for each dynafile:

request - total number of requests made to obtain a dynafile
level0 - requests for the current active file, so no real cache lookup needed to be done. These are extremely good.
missed - cache misses, where the required file did not reside in cache. Even with a perfect cache, there will be at least one miss per file. That happens when the file is being accessed for the first time and brought into cache. So “missed” will always be at least as large as the number of different files processed.
evicted - the number of times a file needed to be evicted from the cache as it ran out of space. These can simply happen when date-based files are used, and the previous date files are being removed from the cache as time progresses. It is better, though, to set an appropriate “closeTimeout” (counter described below), so that files are removed from the cache after they become no longer accessed. It is bad if active files need to be evicted from the cache. This is a very costly operation as an evict requires to close the file (thus a full flush, no matter of its buffer state) and a later access requires a re-open – and the eviction of another file, as the cache obviously has run out of free entries. If this happens frequently, it can severely affect performance. So a high eviction rate is a sign that the dynafile cache size should be increased. If it is already very high, it is recommended to re-think about the design of the file store, at least if the eviction process causes real performance problems.
maxused - the maximum number of cache entries ever used. This can be used to trim the cache down to a value that’s actually useful but does not waste resources. Note that when date-based files are used and rsyslog is run for an extended period of time, the cache gradually fills up to the max configured value as older files are migrated out of it. This will make “maxused” questionable after some time. Frequently enough purging the cache can prevent this (usually, once a day is sufficient).
closetimeouts - available since 8.3.3 – tells how often a file was closed due to timeout settings (“closeTimeout” action parameter). These are cases where dynafiles or static files have been closed by rsyslog due to inactivity. Note that if no “closeTimeout” is specified for the action, this counter always is zero. A high or low number in itself doesn’t mean anything good or bad. It totally depends on the use case, so no general advise can be given.

Caveats/Known Bugs¶

people often report problems that dynafiles are not properly created. The common cause for this problem is SELinux rules, which do not permit the create of those files (check generated file names and paths!). The same happens for generic permission issues (this is often a problem under Ubuntu where permissions are dropped by default)
One needs to be careful with log rotation if signatures and/or encryption are being used. These create side-files, which form a set and must be kept together. For signatures, the “.sigstate” file must NOT be rotated away if signature chains are to be build across multiple files. This is because .sigstate contains just global information for the whole file set. However, all other files need to be rotated together. The proper sequence is to
1. move all files inside the file set
2. only AFTER this is completely done, HUP rsyslog
This sequence will ensure that all files inside the set are atomically closed and in sync. HUPing only after a subset of files have been moved results in inconsistencies and will most probably render the file set unusable.
If zipLevel is greater than 0 and veryRobustZip is set to off, data appended to previously unclean closed files will not be accessible with gunzip if rsyslog writes again in the same file. Nonetheless, data is still there and can be correctly accessed with other tools like gztool (v>=1.1) with: gztool -p.

Examples¶

Example 1¶

The following command writes all syslog messages into a file.

action(type="omfile" dirCreateMode="0700" FileCreateMode="0644"
       File="/var/log/messages")

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project