rsyslog

The rocket-fast system for log processing

Using a different log Format for all Files

Rsyslog comes with a limited set of log file formats. These resemble the default format that people (and log analyzers) usually expect. However, for some reason or another, it may be required to change the log format. In this recipe, we define a new format and use it as the default format for all log files.

Config Statements

$template myFormat,"%rawmsg%\n"
$ActionFileDefaultTemplate myFormat
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* /var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log

Things to think about

The template and ActionFileDefaultTemplate statements must be made at the top of the configuration file, before any of the files are specified.

How it works

The Template-statement defines the new format. It consist of fields to be written, potential modifications as well as literal text. In the sample config statement, "rawmsg" ist a property that contains the syslog message as it was received by rsyslogd ("received" from any source, for example a remote system or the local log socket). The string "\n" is a line feed (ASCII LF), a constant being added to the string. Usually, log line templates need to end with "\n", because without that, all log records would be written into a single line. Note that there are many fields and options for these fields that you can specify. The system is very flexible. But getting into the detail of all of that is beyond the scope of this cookbook-style book. Please see the "property replacer" official documentation for more details.

The $ActionFileDefaultTemplate then makes the newly defined template the default for all file actions. This saves you from specifying it with any single action line. But otherwise, it is equivalent to

$template myFormat,"%rawmsg%\n"
# The authpriv file has restricted access.
authpriv.*      /var/log/secure;myFormat
# Log all the mail messages in one place.
mail.*          /var/log/maillog;myFormat
# Log cron stuff
cron.*          /var/log/cron;myFormat
# Everybody gets emergency messages
*.emerg                                       *
# Save news errors of level crit and higher in a special file.
uucp,news.crit  /var/log/spooler;myFormat
# Save boot messages also to boot.log
local7.*        /var/log/boot.log;myFormat

8 thoughts on “Using a different log Format for all Files

  1. And…

    "But getting into the detail of all of that is beyond the scope of this cookbook-style book."

    Very funny! Perhaps you have read a cookbook where you need some spices or ingredients, but the author just assumes that everyone knows what is "frimminjamzen" and tells you you need 100g, but you don’t know what this is. A little google and you know that it is anchovies. This situation is not so bad… it is just a translation.

    Now imagine that the same author tells you to "arnzarnz" the potatoes and leeks, but only his family uses this word. Now google becomes less useful. maybe you will be lucky and another family uses it, enough that a search engine can help you eventually discover that it means to dice the mixture into 1cm cubes.
    The situation is worse, but not hopeless.

    Finally, imagine that a software author tells us how to apply a special to a system logger. The author does not tell us how to make the special format template, or where to put it so that the software will find it. He simply tells us we can use one and give it whatever name we like…

    It is like the spanish recipes for a love potion… one needs a unicorn horn and a tooth from a chimera, although everything else is easy to find. The recipes NEVER explain where one can FIND a unicorn (yes, yes, we know that you must have a virgin, but then what?) let alone find the "unicorn horn" or where one can find the tooth of a chimera.

    In our case, we are missing "see this document or url for more info" or "the filepath to the templates is xxx/xxxx" or "here is an example

    There is nothing here explaining:

    – where is the default template specified?

    – where can one find the format for the template?

  2. PS – the cross reference is for:

    http://rsyslog.com/doc/rsyslog_conf_templates.html

    This describes the format. I think we can just put them in the main config file, but again, no simple instructions exist.

    For example, there is no explanation of the standard debug template output.

    I just want to see facility and level displayed on each line. I found the syntax from a ~version 1.7 document, but maybe it will not work.

    I think the software is very good, but the documentation is full but unstructured perhaps.
    I know it is hard to find time for such cleanup. danke!

  3. Well, how about doing some of that work yourself? I am far from opposing doc contributions. I know the doc is not well, but there is so much to do. The open source way is collaboration, not a single person doing all work ;)

  4. Hiya, m. Gerhards!
    Funny enough, I had a surprise…
    I have returned almost a year after I wrote the above.
    In fact, I didn’t realize it was me until after once again trying to figure out how to config rsyslogd on a newer debian version!

    Now I want to do something also simple, to clean up the log lines and use a format like:
    YYYYMMDDHHMMSS F L blah blah blah from the syslog message.

    Again, I am struck by no one making such docs available.
    In fact, I start to believe the "default template" must be embedded in your source code.
    I must once again read through your website for an hour and give up after finding nothing.
    But it could be worse! I have now seen your response from the last trip, and I agree with your thoughts…

    On the old days, people would say RTFM, but the FM was weak – and now we are spoiled by modern methods…
    But I understand how easy it is to become tired.
    So, I will look for your source code and see if I can cut off the gorgon’s head with my trusty comb!

    Thank you for your hard work behind the scenes.

  5. Hello again!

    I have spent too much time with modern scripting languages. Even though I once did assembly language and C, it was many years ago… I am no longer man enough to wade into battle for this – the learning curve is too high!

    I am giving up (for the second time) but it would be very nice if we could create a template in a text file and then import it, or perhaps post-process it into a string(?) that your application would recognize… just for the default template. Indeed, one must read the source (and understand it, along with enough GCC to grok the codebase) in order to have any understanding.

    I will simply live with the ugly default logs, as beggars cannot be choosers!!! :)
    I toast a beer to you, and acknowledge my quick defeat by simply browsing the source code….

    Best Wishes, and thanks again for concentrating on the functional bugs rather than such minor details.
    I read your commits, they are certainly more important than date formats!

    Good night, perhaps I will see you in another year!

  6. For anyone seeking to alter the formats, perhaps I have found a clue:

    OLD TEMPLATE FIRST LINES:
    ===================================================
    ###########################
    #### GLOBAL DIRECTIVES ####
    ###########################

    #
    # Use traditional timestamp format.
    # To enable high precision timestamps, comment out the following line.
    #
    $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

    ===================================================

    The above line refers to a template which has been compiled into the application.
    We seek to replace this template. I think we then define a new template.

    Good news is that it is possible, but the Bad news is that there is no large page
    of examples unless you want to google for many snippets and use trial and error.

    In a perfect world, we would write something easy to grasp like:

    default_template{ [RFC-3389] [HOST] [FACILITY] [LEVEL] [MESSAGE] }

    We would have other tokens, and maybe other things. However, I believe that Ranier’s method is similar,
    even if it isn’t shown clearly. See the following, taken from elsewhere on this site:

    $ActionFileDefaultTemplate %TIMESTAMP:1:24:date-rfc3339%,

    So, it appears that the tokens are white space delimited, and possibly they use
    % for use as substitution tokens. Once you get this far, you can see that the comma
    is apparently just a marker for the template key.

    From here, you can see (sort of) what he tries to do if you consult the page:
    http://www.rsyslog.com/doc/rsyslog_conf_templates.html and read the
    "THIS IS LEGACY DOCUMENTATION" section.

    It’s a mess, but I can see the light at the end of the tunnel from here.
    One day there will be a table structure for each example directive,
    and more examples of "this pattern" gives "this result".

    Of course, this is the traditional open source documentation method, and I had forgotten what it was like – now I am spoiled by php, apache and python documentation. Gerhard’s work is quite difficult, and I was wrong to bring up such complaints about making rsyslogd user friendly – hardly anyone ever changes log file formats in any case.

    $template myFormat,"%rawmsg%\n"

    $ActionFileDefaultTemplate myFormat

  7. @old user:

    Loved your comments! Just spent a day trying to get 3 lines of rsyslog conf to work :) Cookbook examples helps a lot.

Comments are closed.