rsyslog

This summary was contributed by David Caplinger through the mailing list.

First, enable the module with something like:

module(load="impstats" interval="660" severity="7")

This will start generating logs tagged with “rsyslogd-pstats” every 600 seconds. If you like, you can use that tag to filter them into their own file:

if $syslogtag contains 'rsyslogd-pstats' then { 
     action(type="omfile" queue.type="linkedlist" queue.discardmark="980" 
            name="pstats" file="/var/log/pstats") 
     stop 
}

You’ll wind up with several log lines at each interval, all showing current counters (since rsyslog restart). So to determine inter-interval deltas, you’d have to import these into a spreadsheet. (Newer rsyslog can emit just the deltas in the log lines, but that’s in v7.5.x I believe.)

For example, if you want to filter based on some property (such as source IP address) and send the matching logs to both a local file and on to a remote destination, you might use something like:

if $fromhost-ip ==
     [ "1.1.1.1", 
       "2.2.2.2" ] 
then {
     action (type="omfwd" queue.type="linkedlist" queue.discardmark="980" 
             action.resumeretrycount="-1" name="NET.forward" target="10.10.10.10" 
             port="514" protocol="tcp")
     action (type="omfile" queue.type="linkedlist" queue.discardmark="980" 
             name="NET.local" file="/var/log/messages")
     stop
}

Which is a log flow like:

source -> imudp -> main Q -> NET.local (to local files) & NET.forward (to remote)

Here’s an example of a batch of pstats output (re-ordered slightly) from the above config:

Nov 13 14:31:35 loghost rsyslogd-pstats: imudp(*:514): submitted=23035
Nov 13 14:31:35 loghost rsyslogd-pstats: main Q: size=15 enqueued=89624087 full=0 discarded.full=0 discarded.nf=0 maxqsize=444
Nov 13 14:31:35 loghost rsyslogd-pstats: NET.local: size=0 enqueued=11541 full=0 discarded.full=0 discarded.nf=0 maxqsize=7
Nov 13 14:31:35 loghost rsyslogd-pstats: NET.local: processed=11541 failed=0
Nov 13 14:31:35 loghost rsyslogd-pstats: NET.forward: size=0 enqueued=11541 full=0 discarded.full=0 discarded.nf=0 maxqsize=7
Nov 13 14:31:35 loghost rsyslogd-pstats: NET.forward: processed=11541 failed=0
Nov 13 14:31:35 loghost rsyslogd-pstats: pstats: size=0 enqueued=65508 full=0 discarded.full=0 discarded.nf=0 maxqsize=25
Nov 13 14:31:35 loghost rsyslogd-pstats: pstats: processed=65500 failed=0

In this case we have:

1) A UDP input (imudp)

This logs message counts “submitted” to rsyslog via UDP port 514.

2) A main queue (main Q)

This shows messages entering the queue (enqueued), as well as any dropped messages (discarded.full=0, discarded.nf=0). It also shows how many times the queue has become completely full (full=0) and it keeps a running total of the maximum size the queue has ever hit (maxqsize=444). (All these counters are since rsyslog startup.)

3) Two output/action queues (NET.local, NET.forward)

These logs queue stats like above, as well as successfully “processed” (via omfile and omfwd in this case), indicating successful delivery to their final destination (local file or remote TCP receiver, in this case).

4) Another queue to handle pstats output itself (as I described above)

This example doesn’t happen to include DA-mode, which adds another pstats log line for the DA portion of the associated action queue.

If you don’t give your action queues names, you’ll wind up with pstats logs referring to things like “action 2”, and have a hard time figuring out what is going on.

A well-behaved queue will have zero discarded.full and discarded.nf, and a low maxqsize, meaning that everything entering the queue is leaving promptly. In a backlog situation, you’ll see size and maxqsize for an action/output queue increase over time, until maxqsize hits your configured queue.size parameter. Then the main Q will start increasing in size (and maxqsize) until it approaches and exceeds full. Then the discarded.nf and discarded.full counters will start climbing.