omprog: Program integration Output module
Module Name: | omprog |
Author: |
Purpose
This module permits to integrate arbitrary external programs into rsyslog’s logging. It is similar to the “execute program (^)” action, but offers better security and much higher performance. While “execute program (^)” can be a useful tool for executing programs if rare events occur, omprog can be used to provide massive amounts of log data to a program.
Executes the configured program and feeds log messages to that binary via stdin. The binary is free to do whatever it wants with the supplied data. If the program terminates, it is re-started. If rsyslog terminates, the program’s stdin will see EOF. The program must then terminate. The message format passed to the program can, as usual, be modified by defining rsyslog templates.
Note that in order to execute the given program, rsyslog needs to have sufficient permissions on the binary file. This is especially true if not running as root. Also, keep in mind that default SELinux policies most probably do not permit rsyslogd to execute arbitrary binaries. As such, permissions must be appropriately added. Note that SELinux restrictions also apply if rsyslogd runs under root. To check if a problem is SELinux-related, you can temporarily disable SELinux and retry. If it then works, you know for sure you have a SELinux issue.
Starting with 8.4.0, rsyslogd emits an error message via the syslog() API call when there is a problem executing the binary. This can be
extremely valuable in troubleshooting. For those technically savvy:
when we execute a binary, we need to fork, and we do not have
full access to rsyslog’s usual error-reporting capabilities after the
fork. As the actual execution must happen after the fork, we cannot
use the default error logger to emit the error message. As such,
we use syslog(). In most cases, there is no real difference
between both methods. However, if you run multiple rsyslog instances,
the message shows up in that instance that processes the default
log socket, which may be different from the one where the error occurred.
Also, if you redirected the log destination, that redirection may
not work as expected.
Configuration Parameters
Note
Parameter names are case-insensitive.
Action Parameters
template
type | default | mandatory |
|
|---|---|---|---|
word | RSYSLOG_FileFormat | no | none |
Name of the template to use to format the log messages passed to the external program.
binary
type | default | mandatory |
|
|---|---|---|---|
string | yes |
|
Full path and command line parameters of the external program to execute. Arbitrary external programs should be placed under the /usr/libexec/rsyslog directory. That is, the binaries put in this namespaced directory are meant for the consumption of rsyslog, and are not intended to be executed by users. In legacy config, it is not possible to specify command line parameters.
confirmMessages
type | default | mandatory |
|
|---|---|---|---|
binary | off | no | none |
Added in version 8.31.0.
Specifies whether the external program provides feedback to rsyslog via stdout. When this switch is set to “on”, rsyslog will wait for the program to confirm each received message. This feature facilitates error handling: instead of having to implement a retry logic, the external program can rely on the rsyslog queueing capabilities.
To confirm a message, the program must write a line with the word OK to its
standard output. If it writes a line containing anything else, rsyslog considers
that the message could not be processed, keeps it in the action queue, and
re-sends it to the program later (after the period specified by the action.resumeInterval parameter).
In addition, when a new instance of the program is started, rsyslog will also wait for the program to confirm it is ready to start consuming logs. This prevents rsyslog from starting to send logs to a program that could not complete its initialization properly.
confirmTimeout
type | default | mandatory |
|
|---|---|---|---|
integer | 10000 | no | none |
Added in version 8.38.0.
Specifies how long rsyslog must wait for the external program to confirm each message when confirmMessages is set to “on”. If the program does not send a response within this timeout, it will be restarted (see signalOnClose, closeTimeout and killUnresponsive for details on the cleanup sequence). The value must be expressed in milliseconds and must be greater than zero.
reportFailures
type | default | mandatory |
|
|---|---|---|---|
binary | off | no | none |
Added in version 8.38.0.
Specifies whether rsyslog must internally log a warning message whenever the program returns an error when confirming a message. The logged message will include the error line returned by the program. This parameter is ignored when confirmMessages is set to “off”.
Enabling this flag can be useful to log the problems detected by the program. However, the information that can be logged is limited to a short error line, and the logs will be tagged as originated by the ‘syslog’ facility (like the rest of rsyslog logs). To avoid these shortcomings, consider the use of the output parameter to capture the stderr of the program.
useTransactions
type | default | mandatory |
|
|---|---|---|---|
binary | off | no | none |
Added in version 8.31.0.
Specifies whether the external program processes the messages in batches (transactions). When this switch is enabled, the logs sent to the program are grouped in transactions. At the start of a transaction, rsyslog sends a special mark message to the program (see beginTransactionMark). At the end of the transaction, rsyslog sends another mark message (see commitTransactionMark).
If confirmMessages is also set to “on”, the program must confirm both the
mark messages and the logs within the transaction. The mark messages must be
confirmed by returning OK, and the individual messages by returning DEFER_COMMIT (instead of OK). Refer to the link below for details.
Warning
This feature is currently experimental. It could change in future releases
without keeping backwards compatibility with existing configurations or the
specified interface. There is also a known issue with the use of
transactions together with confirmMessages=on.
beginTransactionMark
type | default | mandatory |
|
|---|---|---|---|
string | BEGIN TRANSACTION | no | none |
Added in version 8.31.0.
Allows specifying the mark message that rsyslog will send to the external program to indicate the start of a transaction (batch). This parameter is ignored if useTransactions is disabled.
commitTransactionMark
type | default | mandatory |
|
|---|---|---|---|
string | COMMIT TRANSACTION | no | none |
Added in version 8.31.0.
Allows specifying the mark message that rsyslog will send to the external program to indicate the end of a transaction (batch). This parameter is ignored if useTransactions is disabled.
output
type | default | mandatory |
|
|---|---|---|---|
string | none | no | none |
Added in version v8.1.6.
Full path of a file where the output of the external program will be saved. If the file already exists, the output is appended to it. If the file does not exist, it is created with the permissions specified by fileCreateMode.
If confirmMessages is set to “off” (the default), both the stdout and stderr of the child process are written to the specified file.
If confirmMessages is set to “on”, only the stderr of the child is written to the specified file (since stdout is used for confirming the messages).
Rsyslog will reopen the file whenever it receives a HUP signal. This allows the file to be externally rotated (using a tool like logrotate): after each rotation of the file, make sure a HUP signal is sent to rsyslogd.
If the omprog action is configured to use multiple worker threads (queue.workerThreads is set to a value greater than 1), the lines written by the various program instances will not appear intermingled in the output file, as long as the lines do not exceed a certain length and the program writes them to stdout/stderr in line-buffered mode. For details, refer to Interface between rsyslog and external output plugins.
If this parameter is not specified, the output of the program will be
redirected to /dev/null.
Note
Before version v8.38.0, this parameter was intended for debugging purposes only. Since v8.38.0, the parameter can be used for production.
fileCreateMode
type | default | mandatory |
|
|---|---|---|---|
string | 0600 | no | none |
Added in version v8.38.0.
Permissions the output file will be created with, in case the file does not
exist. The value must be a 4-digit octal number, with the initial digit being
zero. Please note that the actual permission depends on the rsyslogd process
umask. If in doubt, use $umask 0000 right at the beginning of the
configuration file to remove any restrictions.
hup.signal
type | default | mandatory |
|
|---|---|---|---|
word | none | no | none |
Added in version 8.9.0.
Specifies which signal, if any, is to be forwarded to the external program when rsyslog receives a HUP signal. Currently, HUP, USR1, USR2, INT, and TERM are supported. If unset, no signal is sent on HUP. This is the default and what pre 8.9.0 versions did.
signalOnClose
type | default | mandatory |
|
|---|---|---|---|
binary | off | no | none |
Added in version 8.23.0.
Specifies whether a TERM signal must be sent to the external program before closing it (when either the worker thread has been unscheduled, a restart of the program is being forced, or rsyslog is about to shutdown).
If this switch is set to “on”, rsyslog will send a TERM signal to the child process before closing the pipe. That is, the process will first receive a TERM signal, and then an EOF on stdin.
No signal is issued if this switch is set to “off” (default). The child process can still detect it must terminate because reading from stdin will return EOF.
See the killUnresponsive parameter for more details.
closeTimeout
type | default | mandatory |
|
|---|---|---|---|
integer | 5000 | no | none |
Added in version 8.35.0.
Specifies how long rsyslog must wait for the external program to terminate (when either the worker thread has been unscheduled, a restart of the program is being forced, or rsyslog is about to shutdown) after closing the pipe, that is, after sending EOF to the stdin of the child process. The value must be expressed in milliseconds and must be greater than or equal to zero.
See the killUnresponsive parameter for more details.
killUnresponsive
type | default | mandatory |
|
|---|---|---|---|
binary | the value of ‘signalOnClose’ | no | none |
Added in version 8.35.0.
Specifies whether a KILL signal must be sent to the external program in case it does not terminate within the timeout indicated by closeTimeout (when either the worker thread has been unscheduled, a restart of the program is being forced, or rsyslog is about to shutdown).
If signalOnClose is set to “on”, the default value of killUnresponsive is also “on”. In this case, the cleanup sequence of the child process is as
follows: (1) a TERM signal is sent to the child, (2) the pipe with the child
process is closed (the child will receive EOF on stdin), (3) rsyslog waits
for the child process to terminate during closeTimeout, (4) if the child
has not terminated within the timeout, a KILL signal is sent to it.
If signalOnClose is set to “off”, the default value of killUnresponsive is also “off”. In this case, the child cleanup sequence is as follows: (1) the
pipe with the child process is closed (the child will receive EOF on stdin),
(2) rsyslog waits for the child process to terminate during closeTimeout,
(3) if the child has not terminated within the timeout, rsyslog ignores it.
This parameter can be set to a different value than signalOnClose, obtaining the corresponding variations of cleanup sequences described above.
forceSingleInstance
type | default | mandatory |
|
|---|---|---|---|
binary | off | no | none |
Added in version v8.1.6.
By default, the omprog action will start an instance (process) of the external program per worker thread (the maximum number of worker threads can be specified with the queue.workerThreads parameter). Moreover, if the action is associated to a disk-assisted queue, an additional instance will be started when the queue is persisted, to process the items stored on disk.
If you want to force a single instance of the program to be executed, regardless of the number of worker threads or the queue type, set this flag to “on”. This is useful when the external program uses or accesses some kind of shared resource that does not allow concurrent access from multiple processes.
Note
Before version v8.38.0, this parameter had no effect.
Examples
Example: command line arguments
In the following example, logs will be sent to a program log.sh located
in /usr/libexec/rsyslog. The program will receive the command line arguments p1, p2 and --param3="value 3".
module(load="omprog")
action(type="omprog"
binary="/usr/libexec/rsyslog/log.sh p1 p2 --param3=\"value 3\""
template="RSYSLOG_TraditionalFileFormat")
Example: external program that writes logs to a database
In this example, logs are sent to the stdin of a Python program that
(let’s assume) writes them to a database. A dedicated disk-assisted
queue with (a maximum of) 5 worker threads is used, to avoid affecting
other log destinations in moments of high load. The confirmMessages flag is enabled, which tells rsyslog to wait for the program to confirm
its initialization and each message received. The purpose of this setup
is preventing logs from being lost because of database connection
failures.
If the program cannot write a log to the database, it will return a
negative confirmation to rsyslog via stdout. Rsyslog will then keep the
failed log in the queue, and send it again to the program after 5
seconds. The program can also write error details to stderr, which will
be captured by rsyslog and written to /var/log/db_forward.log. If
no response is received from the program within a 30-second timeout,
rsyslog will kill and restart it.
module(load="omprog")
action(type="omprog"
name="db_forward"
binary="/usr/libexec/rsyslog/db_forward.py"
confirmMessages="on"
confirmTimeout="30000"
queue.type="LinkedList"
queue.saveOnShutdown="on"
queue.workerThreads="5"
action.resumeInterval="5"
killUnresponsive="on"
output="/var/log/db_forward.log")
Note that the useTransactions flag is not used in this example. The
program stores and confirms each log individually.
obsolete legacy directives
$ActionOMProgBinary <binary> The binary program to be executed.
See also
Help with configuring/using Rsyslog:
Mailing list - best route for general questions
GitHub: rsyslog source project - detailed questions, reporting issues that are believed to be bugs with
Rsyslog
See also
Contributing to Rsyslog:
Source project: rsyslog project README.
Documentation: rsyslog-doc project README
Copyright 2008-2023 Rainer Gerhards (Großrinderfeld), and Others.