The rocket-fast system for log processing

impstats delayed or lost? – cause and cure

Some users report that they do not receive all impstats log records or that these log records are delayed. The common thing about these questions is that those user tend to have very large main message queues.

By default, impstats is run in-band, which means that it’s messages are submitted to the main message queue just like any other messages are.  So if the main queue takes very long to processs, impstats messages get delayed as well. They may be discarded, too, based on queue settings and queue full status. In this scenarios, impstats obviously has problems reporting what is going on.

A simple solution is to run impstats off-band. This is done by simply instructing it to directly write to file. In that mode, the rsyslog engine is not used at all, and output will always be available and happen right on the interval. There is a drawback, though: as the rsyslog core engine is not evolved, things like dynafiles, different templates or forwarding to another host are not possible.

Note that impstats can submit messages both the a file and the regular message stream. This may be an interesting alternative if the main queue causes trouble but usually logs shall be gathered at some central place.

Example for writing to a local file:

module(load="impstats" interval="600" severity="7" log.file="/var/log/impstats")

One thought on “impstats delayed or lost? – cause and cure

  1. Would it be possible to avoid the main queue entirely (and its potential for losing messages) by binding impstats to a specific ruleset and configuring this ruleset to have its own main queue ?


    ruleset(name="handle_impstats" queue.type="FixedArray" queue.size="50000″)
    Your required processing – (no queuing or DA queue)

    Since handle_impstats is on a separate single thread (havent specified queue.workerthreads= which defaults to 1) there, I am right to assume that the order of messages will also be preserved upon entering the ruleset ?

    (obviously, inside this ruleset, if you try to forward to a remote destination with DA queuing, or simply by using UDP order might not be kept futher down the road)

Comments are closed.