Coupling with Logstash via Redis
Original post: Recipe: rsyslog + Redis + Logstash by @Sematext
OK, so you want to hook up rsyslog with Logstash. If you don’t remember why you want that, let me give you a few hints:
- Logstash can do lots of things, it’s easy to set up but tends to be too heavy to put on every server
- you have Redis already installed so you can use it as a centralized queue. If you don’t have it yet, it’s worth a try because it’s very light for this kind of workload.
- you have rsyslog on pretty much all your Linux boxes. It’s light and surprisingly capable, so why not make it push to Redis in order to hook it up with Logstash?
In this post, you’ll see how to install and configure the needed components so you can send your local syslog (or tail files with rsyslog) to be buffered in Redis so you can use Logstash to ship them to Elasticsearch, a logging SaaS like Logsene (which exposes the Elasticsearch API for both indexing and searching) so you can search and analyze them with Kibana:
rsyslog 8.13.0 (v8-stable) released
We have released rsyslog 8.13.0.
This release sports a big number of changes. While most are bugfixes, there are also some additions to existing functionality, most notably the enhancements for ZeroMQ and Redis modules.
http://www.rsyslog.com/changelog-for-8-13-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
rsyslog 8.12.0 (v8-stable) released
We have released rsyslog 8.12.0.
http://www.rsyslog.com/changelog-for-8-12-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.12.0 (v8-stable)
Version 8.12.0 [v8-stable] 2015-08-11
- Harmonize resetConfigVariables values and defaults
see also https://github.com/rsyslog/rsyslog/pull/413
Thanks to Tomas Heinrich for the patch. - GT/KSI: fix some issues in signature file format and add conversion tool
The file format is incompatible to previous format, but tools have been upgraded to handle both and also an option been added to convert from old to new format. - bugfix: ommysql did not work when gnutls was enabled
As it turned out, this was due to a check for GnuTLS functions with the side-effect that AC_CHECK_LIB, by default, adds the lib to LIBS, if there is no explicit action, what was the case here. So everything was now linked against GnuTLS, which in turn made ommysql fail.
Thanks to Thomas D. (whissi) for the analysis of the ommysql/gnutls problem and Thomas Heinrich for pointing out that AC_CHECK_LIB might be the culprit. - bugfix omfile: potential memory leak on file close
see also: https://github.com/rsyslog/rsyslog/pull/423
Thanks to Robert Schiele for the patch. - bugfix omfile: potential race in dynafile detection/creation
This could lead to a segfault.
Thanks to Tomas Heinrich for the patch. - bugfix omfile: Fix race-condition detection in path-creation code
The affected code is used to detect a race condition in between testing for the existence of a directory and creating it if it didn’t exist. The variable tracking the number of attempts wasn’t reset for subsequent elements in the path, thus limiting the number of reattempts to one per the whole path, instead of one per each path element.
This solution was provided by Martin Poole. - bugfix parser subsystem: potential misadressing in SanitizeMsg() could lead to a segfault
Thanks to Tomas Heinrich for the patch. - imfile: files moved outside of directory are now (properly) handled
- bugfix: imfile: segfault when using startmsg.regex if first log line doesn’t match
Thanks to Ciprian Hacman for the patch. - bugfix imfile: file table was corrupted when on file deletion
This could happen when a file that was statically configured (not via an wildcard) was deleted. - bugfix ompgsql: transaction were improperly handled
Now transaction support is solidly disabled until we have enough requests to implement it again. Module still works fine in single insert mode.
closes https://github.com/rsyslog/rsyslog/issues/399 - bugfix mmjsonparse: memory leak if non-cee-json message is processed
see also https://github.com/rsyslog/rsyslog/pull/383
Thanks to Anton Matveenko for the patch - testbench: remove raciness from UDP based tests
- testbench: added bash into all scripts makign it mandatory
- bugfix testbench: Fixed problem building syslog_caller util when liblogging-stdlog is not available.
Thanks to Louis Bouchard for the patch - bugfix rscryutil.1: Added fix checking for generate_man_pages condition
Thanks to Radovan Sroka for the patch - bugfix freebsd console: \n (NL) is prepended with \r (CR) in console output on freebsd only. For more details see here:
https://github.com/rsyslog/rsyslog/issues/372
Thanks to AlexandreFenyo for the patch
Packages for newer Ubuntu versions
With the latest release of rsyslog (8.11.0) we have also introduced a new set of packages that we will produce from now on. We have now added rsyslog release packages for Ubuntu Utopic, Vivid and Wily to the list. Thus, you can now use the packages made by Adiscon on newer Ubuntu versions, too, even on the upcoming version.
And, new releases get a package shortly after the official source release, so you can easily stay up to date with rsyslog on a larger variety of Ubuntu distros. For now, we will also keep making the packages for Precise and Trusty.
rsyslog 8.11.0 (v8-stable) released
We have released rsyslog 8.11.0.
http://www.rsyslog.com/changelog-for-8-11-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.11.0 (v8-stable)
Version 8.11.0 [v8-stable] 2015-06-30
- new signature provider for Keyless Signature Infrastructure (KSI) added
- build system: re-enable use of “make distcheck”
- bugfix imfile: regex multiline mode ignored escapeLF option
Thanks to Ciprian Hacman for reporting the problem
closes https://github.com/rsyslog/rsyslog/issues/370 - bugfix omkafka: fixed several concurrency issues, most of them related to dynamic topics.
Thanks to Janmejay Singh for the patch. - bugfix: execonlywhenpreviousissuspende
d did not work correctly
This especially caused problems when an action with this attribute was configured with an action queue. - bugfix core engine: ensured global variable atomicity
This could lead to problems in RainerScript, as well as probably in other areas where global variables are used inside rsyslog. I wouldn’t outrule it could lead to segfaults.
Thanks to Janmejay Singh for the patch. - bugfix imfile: segfault when using startmsg.regex because of empty log line
closes https://github.com/rsyslog/rsyslog/issues/357
Thanks to Ciprian Hacman for the patch. - bugfix: build problem on Solaris
Thanks to Dagobert Michelsen for reporting this and getting us up to
speed on the openCWS build farm. - bugfix: build system strndup was used even if not present now added compatibility function. This came up on Solaris builds.
Thanks to Dagobert Michelsen for reporting the problem.
closes https://github.com/rsyslog/rsyslog/issues/347
rsyslog 8.10.0 (v8-stable) released
We have released rsyslog 8.10.0.
http://www.rsyslog.com/changelog-for-8-10-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.10.0 (v8-stable)
Version 8.10.0 [v8-stable] 2015-05-19
- imfile: add capability to process multi-line messages based on regex input parameter “endmsg.regex” was added for that purpose. The new mode provides much more power in processing different multiline-formats.
- pmrfc3164: add new parameters
- “detect.yearAfterTimestamp”
This supports timestamps as generated e.g. by some Aruba Networks equipment. - “permit.
squareBracesInHostname”
Permits to use “hostnames” in the form of “[127.0.0.1]”; also seen in Aruba Networks equipment, but we strongly assume this can also happen in other cases, especially with IPv6.
- “detect.yearAfterTimestamp”
- supplementary groups are now set when dropping privileges
closes https://github.com/rsyslog/rsyslog/issues/296
Thanks to Zach Lisinski for the patch. - imfile: added brace glob expansion to wildcard
Thanks to Zach Lisinski for the patch. - zmq: add the ability for zeromq input and outputs to advertise their presence on UDP via the zbeacon API.
Thanks to Brian Knox for the contribution. - added omhttpfs: contributed module for writing to HDFS via HTTP
Thanks to sskaje for the contribution. - Configure option “–disable-debug-symbols” added which is disabled per default. If you set the new option, configure won’t set the appropriate compiler flag to generate debug symbols anymore.
- When building from git source we now require rst2man and yacc (or a replacement like bison).
That isn’t any new requirement, we only added missing configure checks. - Configure option “–enable-generate-man-pages” is now disabled for non git source builds per default but enforced when building from git source.
- mmpstrucdata: some code cleanup
removed lots of early development debug outputs - bugfix imuxsock: fix a crash when setting a hostname
Setting a hostname via the legacy directive would lead to a crash during shutdown caused by a double-free.
Thanks to Tomas Heinrich for the patch. - bugfix: memory leak in mmpstrucdata
Thanks to Grégoire Seux for reporting this issue.
closes https://github.com/rsyslog/rsyslog/issues/310 - bugfix (minor): default action name: assigned number was one off
see also https://github.com/rsyslog/rsyslog/pull/340
Thanks to Tomas Heinrich for the patch. - bugfix: memory leak in imfile
A small leak happened each time a new file was monitored based on a wildcard. Depending on the rate of file creation, this could result in a serious memory leak.
rsyslog 8.9.0 (v8-stable) released
We have released rsyslog 8.9.0.
http://www.rsyslog.com/changelog-for-8-9-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl