rsyslog

High-performance log ingestion and ETL engine

rsyslog

High-performance log ingestion and ETL engine

Author : adisconteam

rsyslog statistic counter plugin imrelp

adisconteamsub-faq, , , ,

Plugin – imrelp

This plugin maintains statistics for each listener. The statistic by default is named “imrelp” , followed by the listener port in parenthesis. For  example, the counter for a listener on port 514 is called “imprelp(514)”. If the input is given a name, that input name is used instead of “imrelp”. This counter is available starting rsyslog 7.5.1

The following properties are maintained for each listener:

  • submitted – total number of messages submitted for processing since startup

Back to statistics counter overview

rsyslog statistic counter Actions

adisconteamsub-faq, , , ,

Actions

  • processed – total number of messages processed by this action. This includes those messages that failed actual execution (so it is a total count of messages ever seen, but not necessarily successfully processed)
  • failed – total number of messages that failed during processing. These are actually lost if they have not been processed by some other action. Most importantly in a failover chain the messages are flagged as “failed” in the failing actions even though they are forwarded to the failover action (the failover action’s “processed” count should equal to failing actions “fail” count in this scenario)
  • suspended (7.5.8+) – total number of times this action suspended itself. Note that this counts the number of times the action transitioned from active to suspended state. The counter is no indication of how long the action was suspended or how often it was retried. This is intentional, as the counter as it currently is permits to tell how often the action ran into a failure condition.
  • suspended.duration (7.5.8+) – the total number of seconds this action was disabled. Note that the second count is incremented as soon as the action is suspended for another interval. As such, the time may be higher than it should be at the reporting point of time. If the pstats interval is shorter than the suspension timeout, the same suspended.duration value may be reported for successive pstats outputs. For a long-running system, this is considered a minimal difference. In general, note that this setting is not totally accurate, especially when running with multiple worker threads. In rsyslog v8, this is the total suspended time for all worker instances of this action.
  • resumed (7.5.8+) – total number of times this action resumed itself. A resumption occurs after the action has detected that a failure condition does no longer exist.

Back to statistics counter overview

rsyslog statistic counter plugin omelasticsearch

adisconteamsub-faq, , , , ,

Plugin – omelasticsearch

This plugin maintains global statistics, which accumulate all action instances. The statistic is named “omelasticsearch”. Parameters are:

  • submitted – number of messages submitted for processing (with both success and error result)
  • fail.httprequests – the number of times a http request failed. Note that a single http request may be used to submit multiple messages, so this number may be (much) lower than fail.http.
  • fail.http – number of message failures due to connection like-problems (things like remote server down, broken link etc)
  • fail.es – number of failures due to elasticsearch error reply; Note that this counter does NOT count the number of failed messages but the number of times a failure occured (a potentially much smaller number). Counting messages would be quite performance-intense and is thus not done.

The fail.httprequests and fail.http counters reflect only failures that omelasticsearch detected. Once it detects problems, it (usually, depends on circumstances) tell the rsyslog core that it wants to be suspended until the situation clears (this is a requirement for rsyslog output modules). Once it is suspended, it does NOT receive any further messages. Depending on the user configuration, messages will be lost during this period. Those lost messages will NOT be counted by impstats (as it does not see them).

Note that some previous (pre 7.4.5) versions of this plugin had different counters. These were experimental and confusing. The only ones really used were “submits”, which were the number of successfully processed messages and “connfail” which were equivalent to “failed.http”.

 

Back to statistics counter overview

rsyslog 7.5.7 (v7-devel) released

adisconteamNews, Release Announcement, , , ,

This is primarily a bug-fixing release, but offers some improvements in worker thread handling (thanks to Pavel Levshin!) as well as usability improvements when working changing queue sizes.

More detailed information is available in the changelog.

ChangeLog:

http://www.rsyslog.com/changelog-for-7-5-7-v7-devel/

Download:

http://www.rsyslog.com/rsyslog-7-5-7-v7-devel/

As always, feedback is appreciated.

Best regards,
Tim Eifler

Changelog for 7.4.6 (v7-stable)

adisconteamChangelog, , , , ,

Version 7.4.6 [v7.4-stable] 2013-10-31

  • bugfix: potential abort during HUP
    This could happen when one of imklog, imzmq3, imkmsg, impstats,
    imjournal, or imuxsock were under heavy load during a HUP.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=489
    Thanks to Guy Rozendorn for reporting the problem and Peval Levhshin for
    his analysis.
  • bugfix: imtcp flowControl parameter incorrectly defaulted to “off”
    This could cause message loss on systems under heavy load and was
    a change-of-behaviour to previous version. This is a regression
    most probably introduced in 5.9.0 (but did not try hard to find the
    exact point of its introduction).
  • now requires libestr 0.1.9 as earlier versions lead to problems with
    number handling in RainerScript
  • bugfix: memory leak in strlen() RainerScript function
    Thanks to Gregoire Seux for reportig this bug.
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=486
  • bugfix: buffer overrun if re_extract function was called for submatch 50
    Thanks to Pavel Levshin for reporting the problem and its location.
  • bugfix: memleak in re_extract() function
    Thanks to Pavel Levshin for reporting this problem.
  • bugfix: potential abort in RainerScript optimizer
    closes: http://bugzilla.adiscon.com/show_bug.cgi?id=488
    Thanks to Thomas Doll for reporting the problem and Pavel Levshin for
    fixing it.
  • bugfix: memory leak in omhiredis
    Thanks to Pavel Levshin for the fix
  • bugfix: segfault if variable was assigned to non-container subtree
    Thanks to Pavel Levshin for the fix

 

Changelog for 7.5.6 (v7-devel)

adisconteamChangelog, , , , ,

Version 7.5.6 [devel] 2013-10-29

  • improved performance of RainerScript variable access
    by refactoring the whole body of variable handling code. This also
    solves some of the anomalies experienced in some versions of rsyslog.
    All variable types are now handled in unified code, including
    access via templates.
  • RainerScript: make use of 64 bit for numbers where available
    Thanks to Pavel Levshin for enhancement.
  • slight performance optimization if GCC is used
    We give branch prediction hints for the frequent RETiRet macro which is
    used for error handling. Some slight performance gain is to be expected
    from that.
  • removed global variable support
    The original idea was not well thought out and global variables, as
    implemented, worked far different from what anybody would expect. As
    such, we consider the current approach as an experiment that did not
    work out and opt to removing it, clearing the way for a better future
    solution. Note: global vars were introduced in 7.5.3 on Sept, 11th 2013.
  • new module mmsequence, primarily used for action load balancing
    Thanks to Pavel Levshin for contributing this module.
  • bugfix: unset statement always worked on message var, even if local
    var was given
  • bugfix: segfault if variable was assigned to non-container subtree
    Thanks to Pavel Levshin for the fix
  • bugfix: imuxsock did not suport addtl sockets if syssock was disabled
    Thanks to Pavel Levshin for the fix
  • bugfix: running imupd on multiple threads lead to segfault if recvmmsg
    is available
  • bugfix: segfault if re_extract() function was used and no match found
  • bugfix: omelasticsearch did not compile on platforms without atomic
    instructions
  • bugfix: potential misadressing on startup if property-filter was used
    This could happen if the property name was longer than 127 chars, a case
    that would not happen in practice.
  • bugfix: invalid property filter was not properly disabled in ruleset
    Note: the cosmetic memory leak introduced with that patch in 7.4.5 is
    now also fixed.
Scroll to top