rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Author : adisconteam

rsyslog 5.9.7 (v5-beta) released

adisconteamNews, Release Announcement, , , , , ,

This is primarily a bug-fixing release. It also introduces the ability to extract substrings from within property fields. This is a recommended update for all users of the v5-beta branch. It will probably turn into the next v5-stable soon, so everyone is encouraged to give this version a try in order to iron out any remaining bugs.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-9-7-v5-beta/

Download:

http://www.rsyslog.com/rsyslog-5-9-7-beta/

As always, feedback is appreciated.

Best regards,
Tim Eifler

How to bind a template

adisconteamFAQ, , ,

This little FAQ describe how to bind a template.
First with the new template format “list” and then with the old “legacy” format.

First off all you have to define a template for example for specify output.

Here is an example template in the list format:

template(name=”FileFormat” type=”list”) {
property(name=”timestamp” dateFormat=”rfc3339″)
constant(value=” “)
property(name=”hostname”)
constant(value=” “)
property(name=”syslogtag”)
constant(value=” “)
property(name=”msg” spifno1stsp=”on” )
property(name=”msg” droplastlf=”on” )
constant(value=”\n”)
}

Then you have to bind the template to an action. You do this via the “template” parameter.

action(… template=”name-of-template”)

Here is an example action with a example-template:

*.* action(type=”omfile” file=”/var/log/all-messages.log” template=”Name-of-your-template”)

In the configuration it should looks like this:

template(name=”FileFormat” type=”list”) {
property(name=”timestamp” dateFormat=”rfc3339″)
constant(value=” “)
property(name=”hostname”)
constant(value=” “)
property(name=”syslogtag”)
constant(value=” “)
property(name=”msg” spifno1stsp=”on” )
property(name=”msg” droplastlf=”on” )
constant(value=”\n”)
}
action(type=”omfile” file=”/var/log/all-msgs.log” template=”FileFormat”)

Here is an example for obsolete legacy format

Here is an example template in the legacy format:

$template ExampleFormat,”%timereported:::date-rfc3339% %HOSTNAME% %msg%”

Here is an example action with a example-template:

*.* /var/log/all-messages.log;Your-Template-Name

In the Configuration it looks like this:

$template ExampleFormat,”%timereported:::date-rfc3339% %HOSTNAME% %msg%”
*.* /var/log/all-messages.log;ExampleFormat

Note that it is advised NOT to use obsolete legacy format for anything complex. Assigning templates is at the border of being not recommended.

Action’s with directives

adisconteamSome core recipies, , , ,

This snippet will show, how Action directives need to be applied to work properly. We will show it with the RELP output module. RELP should ensure a safe and loss-free transmission between two machines. But if not configured properly, messages may get lost anyway. This is mainly meant for any client side configuration.

First of all you have to enable the RELP module.

To load the module use this:

$ModLoad omrelp

To make sure, messages will not get dropped in the event the receiver is not available, we basically need the following directives. Additionaly, the queued messages should get saved to the harddrive if the client service needs to shut down. It is followed by a forwarding action via RELP to our remote server.

$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* :omrelp:192.168.152.2:20514

Attention: The directives are only valid for the next configured action! So you have to set the directives each time you use a new action.

Here is an example with two actions.

#first action
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
:syslogtag, isequal, “app1” :omrelp:192.168.152.2:20514

#second action
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
:syslogtag, isequal, “app2” :omrelp:192.168.152.3:20514

As you can see, we have the whole block of directives mulitple times. But this time, we filter the message for the syslogtag and have the diffenrently tagged messages sent to different receivers. Now if the receiver is not available, sending the messages will be retried until it is back up again. If the local rsyslog needs to shut down, all queued messages get written to disk without being lost.

Changelog for 5.8.9 (v5-stable)

adisconteamChangelog, , , , , ,

Version 5.8.9 [V5-stable] 2012-03-15

  • added tool to recover disk queue if .qi file is missing (recover_qi.pl)
    Thanks to Kaiwang Chen for contributing this tool
  • bugfix: stopped DA queue was never processed after a restart due to a
    regression from statistics module
  • added better doc for statsobj interface
    Thanks to Kaiwang Chen for his suggestions and analysis in regard to the
    stats subsystem.

rsyslog 5.8.9 (v5-stable) released

adisconteamNews, Release Announcement, , , , , ,

This is primarily a maintenance release. It also provides a new tool that permits the recovery of disk queues, when the queue information file (.qi) has been lost. Thanks to Kaiwang Chen for contributing this tool.

There was an important flaw in 5.8.8 that caused disk queue content to be lost after a shutdown and restart of rsyslogd. So if you run this version, an upgrade to 5.8.9 is highly suggested.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-8-9-v5-stable/

Download:

http://www.rsyslog.com/rsyslog-5-8-9-v5-stable/

As always, feedback is appreciated.

Best regards,

Tim Eifler

Changelog for 5.9.5 (v5-devel)

adisconteamChangelog, , , , ,

Version 5.9.5 [V5-DEVEL], 2012-01-27

  • improved impstats subsystem, added many new counters
  • enhanced module loader to not rely on PATH_MAX
  • refactored imklog linux driver, now combined with BSD driver
    The Linux driver no longer supports outdated kernel symbol resolution,
    which was disabled by default for very long. Also overall cleanup,
    resulting in much smaller code. Linux and BSD are now covered by a
    single small driver.
  • $IMUXSockRateLimitInterval DEFAULT CHANGED, was 5, now 0
    The new default turns off rate limiting. This was chosen as people
    experienced problems with rate-limiting activated by default. Now it
    needs an explicit opt-in by setting this parameter.
    Thanks to Chris Gaffney for suggesting to make it opt-in; thanks to
    many unnamed others who already had complained at the time Chris made
    the suggestion ;-)

rsyslog 5.9.5 (v5-devel) released

adisconteamNews, Release Announcement, , , , ,

This release brings many additional statistics counters and a couple of bug fixes. Note that the default setting of $IMUxSockRateLimitInterval was changed to 0, as rate limiting seems to have caused more trouble than it was worth. To enable it, simply set it to 200, the previous default.

ChangeLog:

http://www.rsyslog.com/changelog-for-5-9-5-v5-devel/

Download:

http://www.rsyslog.com/rsyslog-5-9-5-devel/

As always, feedback is appreciated.

Best regards,
Tim Eifler

Scroll to top