rsyslog

The rocket-fast system for log processing

TwitterFacebookGoogle

omprog: Program integration Output module

Module Name: omprog
Author: Rainer Gerhards <rgerhards@adiscon.com>

Purpose

This module permits to integrate arbitrary external programs into rsyslog’s logging. It is similar to the “execute program (^)” action, but offers better security and much higher performance. While “execute program (^)” can be a useful tool for executing programs if rare events occur, omprog can be used to provide massive amounts of log data to a program.

Executes the configured program and feeds log messages to that binary via stdin. The binary is free to do whatever it wants with the supplied data. If the program terminates, it is re-started. If rsyslog terminates, the program’s stdin will see EOF. The program must then terminate. The message format passed to the program can, as usual, be modified by defining rsyslog templates.

Note that each time an omprog action is defined, the corresponding program is invoked. A single instance is not being re-used. There are arguments pro and con for re-using existing binaries. For the time being, it simply is not done. In the future, we may add an option for such pooling, provided that some demand for that is voiced. You can also mimic the same effect by defining multiple rulesets and including them.

Note that in order to execute the given program, rsyslog needs to have sufficient permissions on the binary file. This is especially true if not running as root. Also, keep in mind that default SELinux policies most probably do not permit rsyslogd to execute arbitrary binaries. As such, permissions must be appropriately added. Note that SELinux restrictions also apply if rsyslogd runs under root. To check if a problem is SELinux-related, you can temporarily disable SELinux and retry. If it then works, you know for sure you have a SELinux issue.

Starting with 8.4.0, rsyslogd emits an error message via the syslog() API call when there is a problem executing the binary. This can be extremely valuable in troubleshooting. For those technically savy: when we execute a binary, we need to fork, and we do not have full access to rsyslog’s usual error-reporting capabilities after the fork. As the actual execution must happen after the fork, we cannot use the default error logger to emit the error message. As such, we use syslog(). In most cases, there is no real difference between both methods. However, if you run multiple rsyslog instances, the message shows up in that instance that processes the default log socket, which may be different from the one where the error occured. Also, if you redirected the log destination, that redirection may not work as expected.

Configuration Parameters

Note

Parameter names are case-insensitive.

Action Parameters

template

type default mandatory obsolete legacy directive
word RSYSLOG_FileFormat no none

Name of the template to use to format the log messages passed to the external program.

binary

type default mandatory obsolete legacy directive
string   yes $ActionOMProgBinary

Full path and command line parameters of the external program to execute.

In legacy config, it is not possible to specify command line parameters.

confirmMessages

type default mandatory obsolete legacy directive
binary off no none

New in version 8.31.0.

Specifies whether the external program provides feedback to rsyslog via stdout. When this switch is set to “on”, rsyslog will wait for the program to confirm each received message. This feature facilitates error handling: instead of having to implement a retry logic, the external program can rely on the rsyslog queueing capabilities.

To confirm a message, the program must write a line with the word OK to its standard output. If it writes a line containing anything else, rsyslog considers that the message could not be processed, keeps it in the action queue, and re-sends it to the program later (after the period specified by the action.resumeInterval parameter).

In addition, when a new instance of the program is started, rsyslog will also wait for the program to confirm it is ready to start consuming logs. This prevents rsyslog from starting to send logs to a program that could not complete its initialization properly.

useTransactions

type default mandatory obsolete legacy directive
binary off no none

New in version 8.31.0.

Specifies whether the external program processes the messages in batches (transactions). When this switch is enabled, the logs sent to the program are grouped in transactions. At the start of a transaction, rsyslog sends a special mark message to the program (see beginTransactionMark). At the end of the transaction, rsyslog sends another mark message (see commitTransactionMark).

If confirmMessages is also set to “on”, the program must confirm both the mark messages and the logs within the transaction. The mark messages must be confirmed by returning OK, and the individual messages by returning DEFER_COMMIT (instead of OK). Refer to the link below for details.

Note

There is currently a known issue with the use of transactions together with confirmMessages=on.

beginTransactionMark

type default mandatory obsolete legacy directive
string BEGIN TRANSACTION no none

New in version 8.31.0.

Allows specifying the mark message that rsyslog will send to the external program to indicate the start of a transaction (batch). This parameter is ignored if useTransactions is disabled.

commitTransactionMark

type default mandatory obsolete legacy directive
string COMMIT TRANSACTION no none

New in version 8.31.0.

Allows specifying the mark message that rsyslog will send to the external program to indicate the end of a transaction (batch). This parameter is ignored if useTransactions is disabled.

output

type default mandatory obsolete legacy directive
string none no none

New in version v8.1.6.

Full path of a file where the output of the external program must be saved, for debugging purposes.

Note that if the action has multiple worker threads (queue.workerThreads is set to a value greater than 1), all threads will write to the file at the same time, which will cause the output of the multiple child processes to be mixed. When using this parameter, use a single worker thread.

If confirmMessages is set to “off” (the default), both the stdout and stderr of the child process are written to the specified file.

If confirmMessages is set to “on”, only the stderr of the child is written to the specified file (since stdout is used for confirming the messages).

hup.signal

type default mandatory obsolete legacy directive
word none no none

New in version 8.9.0.

Specifies which signal, if any, is to be forwarded to the external program when rsyslog receives a HUP signal. Currently, HUP, USR1, USR2, INT, and TERM are supported. If unset, no signal is sent on HUP. This is the default and what pre 8.9.0 versions did.

signalOnClose

type default mandatory obsolete legacy directive
binary off no none

New in version 8.23.0.

Specifies whether a TERM signal must be sent to the external program before closing it (either because the worker thread has been unscheduled, or rsyslog is about to shutdown).

If this switch is set to “on”, rsyslog will send a TERM signal to the child process before closing the pipe. That is, the process will first receive a TERM signal, and then an EOF on stdin.

No signal is issued if this switch is set to “off” (default). The child process can still detect it must terminate because reading from stdin will return EOF.

See the killUnresponsive parameter for more details.

closeTimeout

type default mandatory obsolete legacy directive
integer 5000 no none

New in version 8.35.0.

Specifies how long rsyslog must wait for the external program to terminate after closing the pipe (that is, sending EOF to the stdin of the child process). The value must be expressed in milliseconds and must be greater than or equal to zero.

See the killUnresponsive parameter for more details.

killUnresponsive

type default mandatory obsolete legacy directive
binary the value of ‘signalOnClose’ no none

New in version 8.35.0.

Specifies whether a KILL signal must be sent to the external program in case it does not terminate within the timeout indicated by closeTimeout (when rsyslog is shutting down or the worker thread is being unscheduled).

If signalOnClose is set to “on”, the default value of killUnresponsive is also “on”. In this case, the cleanup sequence of the child process is as follows: (1) a TERM signal is sent to the child, (2) the pipe with the child process is closed (the child will receive EOF on stdin), (3) rsyslog waits for the child process to terminate during closeTimeout, (4) if the child has not terminated within the timeout, a KILL signal is sent to it.

If signalOnClose is set to “off”, the default value of killUnresponsive is also “off”. In this case, the child cleanup sequence is as follows: (1) the pipe with the child process is closed (the child will receive EOF on stdin), (2) rsyslog waits for the child process to terminate during closeTimeout, (3) if the child has not terminated within the timeout, rsyslog ignores it and continues with the shutdown (or the unschedule of the worker thread).

This parameter can be set to a different value than signalOnClose, obtaining the corresponding variations of cleanup sequences described above.

Examples

Example: command line arguments

In the following example, logs will be sent to a program log.sh located in /path/to. The program will receive the command line arguments -p="value 1" and --param2="value2".

module(load="omprog")

action(type="omprog"
       binary="/path/to/log.sh -p=\"value 1\" --param2=\"value2\""
       template="RSYSLOG_TraditionalFileFormat")

Example: external program that writes logs to a database

In this example, logs are sent to the stdin of a Python program that (let’s assume) writes them to a database. A dedicated disk-assisted queue with (a maximum of) 5 worker threads is used, to avoid affecting other log destinations in moments of high load. The confirmMessages flag is enabled, which tells rsyslog to wait for the program to confirm its initialization and each message received. The purpose of this setup is preventing logs from being lost because of database connection failures.

If the program cannot write a log to the database, it will return a negative confirmation to rsyslog via stdout. Rsyslog will then keep the failed log in the queue, and send it again to the program after 5 seconds, with infinite retries.

module(load="omprog")

action(type="omprog"
       name="db_forward"
       binary="/usr/share/logging/db_forward.py"
       confirmMessages="on"
       queue.type="LinkedList"
       queue.saveOnShutdown="on"
       queue.workerThreads="5"
       action.resumeInterval="5"
       action.resumeRetryCount="-1")

Note that the useTransactions flag is not used in this example. The program stores and confirms each log individually.

obsolete legacy directives

  • $ActionOMProgBinary <binary> The binary program to be executed.

Deprecated parameters

Note: While these parameters are still accepted, they should no longer be used for newly created configurations.

forceSingleInstance

type default mandatory obsolete legacy directive
binary off no none

New in version v8.1.6.

If set to “on”, this switch prevents the action’s worker threads from concurrently sending logs to the running instances (child processes) of the external program.

Note that enabling this switch, despite its name, will NOT force a single instance of the program to be executed. If you want this behavior, set the queue.workerThreads parameter to 1 (which is the default value). This will cause only one worker thread to be scheduled for the action.

Besides, when queue.workerThreads is greater than 1, enabling this switch will NOT prevent the child processes from concurrently process the received logs, since the processes run asynchronously with respect to rsyslog because of the pipe buffering (unless the feedback mode is used; see the confirmMessages parameter). In general, if the external program uses or accesses some kind of shared resource that does not allow concurrent access from multiple processes, it is recommended to set queue.workerThreads to 1.

This parameter is deprecated. If you are using it with a value of “off” (the default), you can safely remove it. If you are using it with a value of “on”, consider setting queue.workerThreads to 1 instead, for the reasons explained above.

See also

Help with configuring/using Rsyslog:

  • Mailing list - best route for general questions
  • GitHub: rsyslog source project - detailed questions, reporting issues that are believed to be bugs with Rsyslog
  • Stack Exchange (View, Ask) - experimental support from rsyslog community

See also

Contributing to Rsyslog:

© 2008-2017, Rainer Gerhards and Others. This site uses the “better” theme for Sphinx.