Templates ========= Templates are a key feature of rsyslog. They allow to specify any format a user might want. They are also used for dynamic file name generation. Every output in rsyslog uses templates - this holds true for files, user messages and so on. The database writer expects its template to be a proper SQL statement - so this is highly customizable too. You might ask how does all of this work when no templates at all are specified. Good question ;). The answer is simple, though. Templates are compatible with the stock syslogd formats which are hardcoded into rsyslogd. So if no template is specified, we use one of those hardcoded templates. Search for "template\_" in rsconf.c and you will find the hardcoded ones. Templates are specified by template() statements. They can also be specified via $template legacy statements. **Note: key elements of templates are rsyslog properties.** See the :doc:`rsyslog properties reference ` for a list of which are available. Template processing ------------------- Due to lack of standardization regarding logs formats, when a template is specified it's supposed to include HEADER, as defined in `RFC5424 `_ It's very important to have this in mind, and also how to understand how `rsyslog parsing `_ works For example, if MSG field is set to "this:is a message" and no HOSTNAME, neither TAG are specified, outgoing parser will split the message as: :: TAG:this: MSG:is a message The template() statement ------------------------ The template() statement is used to define templates. Note that it is a **static** statement, that means all templates are defined when rsyslog reads the config file. As such, templates are not affected by if-statements or config nesting. The basic structure of the template statement is as follows: :: template(parameters) In addition to this simpler syntax, list templates (to be described below) support an extended syntax: :: template(parameters) { list-descriptions } Each template has a parameter **name**, which specifies the template name, and a parameter **type**, which specifies the template type. The name parameter must be unique, and behaviour is unpredictable if it is not. The **type** parameter specifies different template types. Different types simply enable different ways to specify the template content. The template type **does not** affect what an (output) plugin can do with it. So use the type that best fits your needs (from a config writing point of view!). The following types are available: - list - subtree - string - plugin The various types are described below. list ~~~~ In this case, the template is generated by a list of constant and variable statements. These follow the template spec in curly braces. This type is also primarily meant for use with structure-aware outputs, like ommongodb. However, it also works perfectly with text-based outputs. We recommend to use this mode if more complex property substitutions need to be done. In that case, the list-based template syntax is much clearer than the simple string-based one. The list template contains the template header (with **type="list"**) and is followed by **constant** and **property** statements, given in curly braces to signify the template statement they belong to. As the name says, **constant** statements describe constant text and **property** describes property access. There are many options to **property**, described further below. Most of these options are used to extract only partial property contents or to modify the text obtained (like to change its case to upper or lower case, only). To grasp the idea, an actual sample is: :: template(name="tpl1" type="list") { constant(value="Syslog MSG is: '") property(name="msg") constant(value="', ") property(name="timereported" dateFormat="rfc3339" caseConversion="lower") constant(value="\n") } This sample is probably primarily targeted at the usual file-based output. constant statement ^^^^^^^^^^^^^^^^^^ This provides a way to specify constant text. The text is used literally. It is primarily intended for text-based output, so that some constant text can be included. For example, if a complex template is built for file output, one usually needs to finish it by a newline, which can be introduced by a constant statement. Here is an actual sample of that use case from the rsylsog testbench: :: template(name="outfmt" type="list") { property(name="$!usr!msgnum") constant(value="\n") } The following escape sequences are recognized inside the constant text: - \\\\ - single backslash - \\n - LF - \\ooo - (three octal digits) - represents character with this numerical value (e.g. \\101 equals "A"). Note that three octal digits must be given (in contrast to some languages, where between one and three are valid). While we support octal notation, we recommend to use hex notation as this is better known. - \\xhh - (where h is a hex digit) - represents character with this numerical value (e.g. \\x41 equals "A"). Note that two hexadecimal digits must be given (in contrast to some languages where one or two are valid). - ... some others ... list needs to be extended Note: if an unsupported character follows a backslash, this is treated as an error. Behaviour is unpredictable in this case. To aid usage of the same template both for text-based outputs and structured ones, constant text without an "outname" parameter will be ignored when creating the name/value tree for structured outputs. So if you want to supply some constant text e.g. to mongodb, you must include an outname, as can be seen here: :: template(name="outfmt" type="list") { property(name="$!usr!msgnum") constant(value="\n" outname="IWantThisInMyDB") } The "constant" statement supports the following parameters: - value - the constant value to use - outname - output field name (for structured outputs) property statement ^^^^^^^^^^^^^^^^^^ This statement is used to include property text. It can access all properties. Also, options permit to specify picking only part of a property or modifying it. It supports the following parameters: - name - the name of the property to access - outname - output field name (for structured outputs) - dateformat - date format to use (only for date-related properties) - date.inUTC - date shall be shown in UTC (please note that this requires a bit more performance due to the necessary conversions) Available since 8.18.0. - caseconversion - permits to convert case of the text. Supported values are "lower" and "upper" - controlcharacters - specifies how to handle control characters. Supported values are "escape", which escapes them, "space", which replaces them by a single space, and "drop", which simply removes them from the string. - securepath - used for creating pathnames suitable for use in dynafile templates - format - specify format on a field basis. Supported values are: - "`csv `_\ " for use when csv-data is generated - "`json `_\ " which formats proper json content (but without a field header) - "`jsonf `_\ " which formats as a complete json field - "`jsonr `_\ " which avoids double escaping the value but makes it safe for a json field - "`jsonfr `_\ " which is the combination of "jsonf" and "jsonr". - position.from - obtain substring starting from this position (1 is the first position) - position.to - obtain substring up to this position - position.relativeToEnd - the from and to position is relative to the end of the string instead of the usual start of string. (available since rsyslog v7.3.10) - fixedwidth - changes behaviour of position.to so that it pads the source string with spaces up to the value of position.to if the source string is shorter. "on" or "off" (default) (available since rsyslog v8.13.0) - compressspace - compresses multiple spaces (US-ASCII SP character) inside the string to a single one. This compression happens at a very late stage in processing. Most importantly, it happens after substring extraction, so the **position.from** and **position.to** positions are **NOT** affected by this option. (available since v8.18.0). - field.number - obtain this field match - field.delimiter - decimal value of delimiter character for field extraction - regex.expression - expression to use - regex.type - either ERE or BRE - regex.nomatchmode - what to do if we have no match - regex.match - match to use - regex.submatch - submatch to use - droplastlf - drop a trailing LF, if it is present - mandatory - signifies a field as mandatory. If set to "on", this field will always be present in data passed to structured outputs, even if it is empty. If "off" (the default) empty fields will not be passed to structured outputs. This is especially useful for outputs that support dynamic schemas (like ommongodb). - spifno1stsp - expert options for RFC3164 template processing subtree ~~~~~~~ Available since rsyslog 7.1.4 In this case, the template is generated based on a complete (CEE) subtree. This type of template is most useful for outputs that know how to process hierarchical structure, like ommongodb. With that type, the parameter **subtree** must be specified, which tells which subtree to use. For example template(name="tpl1" type="subtree" subtree="$!") includes all CEE data, while template(name="tpl2" type="subtree" subtree="$!usr!tpl2") includes only the subtree starting at $!usr!tpl2. The core idea when using this type of template is that the actual data is prefabricated via set and unset script statements, and the resulting structure is then used inside the template. This method MUST be used if a complete subtree needs to be placed *directly* into the object's root. With all other template types, only subcontainers can be generated. Note that subtree type can also be used with text-based outputs, like omfile. HOWEVER, you do not have any capability to specify constant text, and as such cannot include line breaks. As a consequence, using this template type for text outputs is usually only useful for debugging or very special cases (e.g. where the text is interpreted by a JSON parser later on). Use case ^^^^^^^^ A typical use case is to first create a custom subtree and then include it into the template, like in this small example: :: set $!usr!tpl2!msg = $msg; set $!usr!tpl2!dataflow = field($msg, 58, 2); template(name="tpl2" type="subtree" subtree="$!usr!tpl2") Here, we assume that $msg contains various fields, and the data from a field is to be extracted and stored - together with the message - as field content. string ~~~~~~ This closely resembles the legacy template statement. It has a mandatory parameter **string**, which holds the template string to be applied. A template string is a mix of constant text and replacement variables (see property replacer). These variables are taken from message or other dynamic content when the final string to be passed to a plugin is generated. String-based templates are a great way to specify textual content, especially if no complex manipulation to properties is necessary. This is a sample for a string-based template: :: template(name="tpl3" type="string" string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" ) The text between percent signs ('%') is interpreted by the rsyslog :doc:`property replacer `. In a nutshell, it contains the property to use as well as options for formatting and further processing. This is very similar to what the ``property`` object in list templates does (it actually is just a different language to express most of the same things). Everything outside of the percent signs is constant text. In the above case, we have mostly spaces between the property values. At the end of the string, an escape sequence is used. Escape sequences permit to specify nonprintable characters. They work very similar to escape sequences in C and many other languages. They are initiated by the backslash characters and followed by one or more characters that specify the actual character. For example \\7 is the US-ASCII BEL character and \\n is a newline. The set is similar to what C and perl support, but a bit more limited. plugin ~~~~~~ In this case, the template is generated by a plugin (which is then called a "strgen" or "string generator"). The format is fixed as it is coded. While this is inflexible, it provides superior performance, and is often used for that reason (not that "regular" templates are slow - but in very demanding environments that "last bit" can make a difference). Refer to the plugin's documentation for further details. For this type, the parameter **plugin** must be specified and must contain the name of the plugin as it identifies itself. Note that the plugin must be loaded prior to being used inside a template. Config example: ``template(name="tpl4" type="plugin" plugin="mystrgen")`` options ~~~~~~~ The part is optional. It carries options influencing the template as a whole and is a part of the template parameters. See details below. Be sure NOT to mistake template options with property options - the latter ones are processed by the property replacer and apply to a SINGLE property, only (and not the whole template). Template options are case-insensitive. Currently defined are: **option.sql** - format the string suitable for a SQL statement in MySQL format. This will replace single quotes ("'") and the backslash character by their backslash-escaped counterpart ("\\'" and "\\\\") inside each field. Please note that in MySQL configuration, the ``NO_BACKSLASH_ESCAPES`` mode must be turned off for this format to work (this is the default). **option.stdsql** - format the string suitable for a SQL statement that is to be sent to a standards-compliant sql server. This will replace single quotes ("'") by two single quotes ("''") inside each field. You must use stdsql together with MySQL if in MySQL configuration the ``NO_BACKSLASH_ESCAPES`` is turned on. **option.json** - format the string suitable for a json statement. This will replace single quotes ("'") by two single quotes ("''") inside each field. **option.casesensitive** - treat property name references as case sensitive. The default is "off", where all property name references are first converted to lowercase during template definition. With this option turned "on", property names are looked up as defined in the template. Use this option if you have JSON (``$!*``), local (``!.*``), or global (``$!\\*``) properties which contain uppercase letters. The normal Rsyslog properties are case-insensitive, so this option is not needed for properly referencing those properties. Use of the options **option.sql**, **option.stdsql**, and **option.json** are mutually exclusive. Using more than one at the same time can cause unpredictable behaviour. Either the **sql** or **stdsql** option **must** be specified when a template is used for writing to a database, otherwise injection might occur. Please note that due to the unfortunate fact that several vendors have violated the sql standard and introduced their own escape methods, it is impossible to have a single option doing all the work.  So you yourself must make sure you are using the right format. **If you choose the wrong one, you are still vulnerable to sql injection.** Please note that the database writer *checks* that the sql option is present in the template. If it is not present, the write database action is disabled. This is to guard you against accidentally forgetting it and then becoming vulnerable to SQL injection. The sql option can also be useful with files - especially if you want to import them into a database on another machine for performance reasons. However, do NOT use it if you do not have a real need for it - among others, it takes some toll on the processing time. Not much, but on a really busy system you might notice it. The default template for the write to database action has the sql option set. As we currently support only MySQL and the sql option matches the default MySQL configuration, this is a good choice. However, if you have turned on ``NO_BACKSLASH_ESCAPES`` in your MySQL config, you need to supply a template with the stdsql option. Otherwise you will become vulnerable to SQL injection. :: template (name="TraditionalFormat" type="string" string="%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n" Examples ~~~~~~~~ Standard Template for Writing to Files ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: template(name="FileFormat" type="list") { property(name="timestamp" dateFormat="rfc3339") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag") property(name="msg" spifno1stsp="on" ) property(name="msg" droplastlf="on" ) constant(value="\n") } The equivalent string template looks like this: :: template(name="FileFormat" type="string" string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" ) Note that the template string itself must be on a single line. Standard Template for Forwarding to a Remote Host (RFC3164 mode) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: template(name="ForwardFormat" type="list") { constant(value="<") property(name="pri") constant(value=">") property(name="timestamp" dateFormat="rfc3339") constant(value=" ") property(name="hostname") constant(value=" ") property(name="syslogtag" position.from="1" position.to="32") property(name="msg" spifno1stsp="on" ) property(name="msg") } The equivalent string template looks like this: :: template(name="forwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" ) Note that the template string itself must be on a single line. Standard Template for writing to the MySQL database ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: template(name="StdSQLformat" type="list" option.sql="on") { constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)") constant(value=" values ('") property(name="msg") constant(value="', ") property(name="syslogfacility") constant(value=", '") property(name="hostname") constant(value="', ") property(name="syslogpriority") constant(value=", '") property(name="timereported" dateFormat="mysql") constant(value="', '") property(name="timegenerated" dateFormat="mysql") constant(value="', ") property(name="iut") constant(value=", '") property(name="syslogtag") constant(value="')") } The equivalent string template looks like this: :: template(name="stdSQLformat" type="string" option.sql="on" string="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')" ) Note that the template string itself must be on a single line. Creating Dynamic File Names for omfile ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Templates can be used to generate actions with dynamic file names. For example, if you would like to split syslog messages from different hosts to different files (one per host), you can define the following template: :: template (name="DynFile" type="string" string="/var/log/system-%HOSTNAME%.log") Legacy example: :: $template DynFile,"/var/log/system-%HOSTNAME%.log" This template can then be used when defining an action. It will result in something like "/var/log/system-localhost.log" legacy format ------------- In pre v6-versions of rsyslog, you need to use the ``$template`` statement to configure templates. They provide the equivalent to string- and plugin-based templates. The legacy syntax continuous to work in v7, however we recommend to avoid legacy format for newly written config files. Legacy and current config statements can coexist within the same config file. The general format is :: $template name,param[,options] where "name" is the template name and "param" is a single parameter that specifies template content. The optional "options" part is used to set template options. string ~~~~~~ The parameter is the same string that with the current-style format you specify in the **string** parameter, for example: :: $template strtpl,"PRI: %pri%, MSG: %msg%\n" Note that list templates are not available in legacy format, so you need to use complex property replacer constructs to do complex things. plugin ~~~~~~ This is equivalent to the "plugin"-type template directive. Here, the parameter is the plugin name, with an equal sign prepended. An example is: :: $template plugintpl,=myplugin Reserved Template Names ----------------------- Template names beginning with "RSYSLOG\_" are reserved for rsyslog use. Do NOT use them if, otherwise you may receive a conflict in the future (and quite unpredictable behaviour). There is a small set of pre-defined templates that you can use without the need to define it: - **RSYSLOG\_TraditionalFileFormat** - the "old style" default log file format with low-precision timestamps - **RSYSLOG\_FileFormat** - a modern-style logfile format similar to TraditionalFileFormat, both with high-precision timestamps and timezone information - **RSYSLOG\_TraditionalForwardFormat** - the traditional forwarding format with low-precision timestamps. Most useful if you send messages to other syslogd's or rsyslogd below version 3.12.5. - **RSYSLOG\_SysklogdFileFormat** - sysklogd compatible log file format. If used with options: ``$SpaceLFOnReceive on``, ``$EscapeControlCharactersOnReceive off``, ``$DropTrailingLFOnReception off``, the log format will conform to sysklogd log format. - **RSYSLOG\_ForwardFormat** - a new high-precision forwarding format very similar to the traditional one, but with high-precision timestamps and timezone information. Recommended to be used when sending messages to rsyslog 3.12.5 or above. - **RSYSLOG\_SyslogProtocol23Format** - the format specified in IETF's internet-draft ietf-syslog-protocol-23, which is very close to the actual syslog standard `RFC5424 `_ (we couldn't update this template as things were in production for quite some time when RFC5424 was finally approved). This format includes several improvements. You may use this format with all relatively recent versions of rsyslog or syslogd. - **RSYSLOG\_DebugFormat** - a special format used for troubleshooting property problems. This format is meant to be written to a log file. Do **not** use for production or remote forwarding. Legacy String-based Template Samples ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section provides some default templates in legacy format, as used in rsyslog previous to version 6. Note that this format is still supported, so there is no hard need to upgrade existing configurations. However, it is strongly recommended that the legacy constructs are not used when crafting new templates. Note that each $template statement is on a **single** line, but probably broken across several lines for display purposes by your browsers. Lines are separated by empty lines. Keep in mind, that line breaks are important in legacy format. :: $template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" $template TraditionalFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" $template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" $template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%" $template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL`` See Also -------- - `How to bind a template `_ - `Adding the BOM to a message `_ - `How to separate log files by host name of the sending device `_