Changelog for 8.11.0 (v8-stable)
Version 8.11.0 [v8-stable] 2015-06-30
- new signature provider for Keyless Signature Infrastructure (KSI) added
- build system: re-enable use of “make distcheck”
- bugfix imfile: regex multiline mode ignored escapeLF option
Thanks to Ciprian Hacman for reporting the problem
closes https://github.com/rsyslog/rsyslog/issues/370 - bugfix omkafka: fixed several concurrency issues, most of them related to dynamic topics.
Thanks to Janmejay Singh for the patch. - bugfix: execonlywhenpreviousissuspende
d did not work correctly
This especially caused problems when an action with this attribute was configured with an action queue. - bugfix core engine: ensured global variable atomicity
This could lead to problems in RainerScript, as well as probably in other areas where global variables are used inside rsyslog. I wouldn’t outrule it could lead to segfaults.
Thanks to Janmejay Singh for the patch. - bugfix imfile: segfault when using startmsg.regex because of empty log line
closes https://github.com/rsyslog/rsyslog/issues/357
Thanks to Ciprian Hacman for the patch. - bugfix: build problem on Solaris
Thanks to Dagobert Michelsen for reporting this and getting us up to
speed on the openCWS build farm. - bugfix: build system strndup was used even if not present now added compatibility function. This came up on Solaris builds.
Thanks to Dagobert Michelsen for reporting the problem.
closes https://github.com/rsyslog/rsyslog/issues/347
rsyslog 8.10.0 (v8-stable) released
We have released rsyslog 8.10.0.
http://www.rsyslog.com/changelog-for-8-10-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.10.0 (v8-stable)
Version 8.10.0 [v8-stable] 2015-05-19
- imfile: add capability to process multi-line messages based on regex input parameter “endmsg.regex” was added for that purpose. The new mode provides much more power in processing different multiline-formats.
- pmrfc3164: add new parameters
- “detect.yearAfterTimestamp”
This supports timestamps as generated e.g. by some Aruba Networks equipment. - “permit.
squareBracesInHostname”
Permits to use “hostnames” in the form of “[127.0.0.1]”; also seen in Aruba Networks equipment, but we strongly assume this can also happen in other cases, especially with IPv6.
- “detect.yearAfterTimestamp”
- supplementary groups are now set when dropping privileges
closes https://github.com/rsyslog/rsyslog/issues/296
Thanks to Zach Lisinski for the patch. - imfile: added brace glob expansion to wildcard
Thanks to Zach Lisinski for the patch. - zmq: add the ability for zeromq input and outputs to advertise their presence on UDP via the zbeacon API.
Thanks to Brian Knox for the contribution. - added omhttpfs: contributed module for writing to HDFS via HTTP
Thanks to sskaje for the contribution. - Configure option “–disable-debug-symbols” added which is disabled per default. If you set the new option, configure won’t set the appropriate compiler flag to generate debug symbols anymore.
- When building from git source we now require rst2man and yacc (or a replacement like bison).
That isn’t any new requirement, we only added missing configure checks. - Configure option “–enable-generate-man-pages” is now disabled for non git source builds per default but enforced when building from git source.
- mmpstrucdata: some code cleanup
removed lots of early development debug outputs - bugfix imuxsock: fix a crash when setting a hostname
Setting a hostname via the legacy directive would lead to a crash during shutdown caused by a double-free.
Thanks to Tomas Heinrich for the patch. - bugfix: memory leak in mmpstrucdata
Thanks to Grégoire Seux for reporting this issue.
closes https://github.com/rsyslog/rsyslog/issues/310 - bugfix (minor): default action name: assigned number was one off
see also https://github.com/rsyslog/rsyslog/pull/340
Thanks to Tomas Heinrich for the patch. - bugfix: memory leak in imfile
A small leak happened each time a new file was monitored based on a wildcard. Depending on the rate of file creation, this could result in a serious memory leak.
rsyslog 8.9.0 (v8-stable) released
We have released rsyslog 8.9.0.
http://www.rsyslog.com/changelog-for-8-9-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.9.0 (v8-stable)
Version 8.9.0 [v8-stable] 2015-04-07
- omprog: add option “hup.forward” to forwards HUP to external plugins
This was suggested by David Lang so that external plugins (and other
programs) can also do HUP-specific processing. The default is not
to forward HUP, so no change of behavior by default. - imuxsock: added capability to use regular parser chain
Previously, this was a fixed format, that was known to be spoken on
the system log socket. This also adds new parameters:- sysSock.useSpecialParser module parameter
- sysSock.parseHostname module parameter
- useSpecialParser input parameter
- parseHostname input parameter
- 0mq: improvements in input and output modules
See module READMEs, part is to be considered experimental.
Thanks to Brian Knox for the contribution. - imtcp: add support for ip based bind for imtcp -> param “address”
Thanks to github user crackytsi for the patch. - bugfix: MsgDeserialize out of sync with MsgSerialize for StrucData
This lead to failure of disk queue processing when structured data was
present. Thanks to github user adrush for the fix. - bugfix imfile: partial data loss, especially in readMode != 0
closes https://github.com/rsyslog/rsyslog/issues/144 - bugfix: potential large memory consumption with failed actions
see also https://github.com/rsyslog/rsyslog/issues/253 - bugfix: omudpspoof: invalid default send template in RainerScript format
The file format template was used, which obviously does not work for
forwarding. Thanks to Christopher Racky for alerting us.
closes https://github.com/rsyslog/rsyslog/issues/268 - bugfix: size-based legacy config statements did not work properly
on some platforms, they were incorrectly handled, resulting in all
sorts of “interesting” effects (up to segfault on startup) - build system: added option –without-valgrind-testbench
… which provides the capability to either enforce or turn off
valgrind use inside the testbench. Thanks to whissi for the patch. - rsyslogd: fix misleading typos in error messages
Thanks to Ansgar Püster for the fixes.
Using rsyslog and Elasticsearch to Handle Different Types of JSON Logs
Originally posted on the Sematext blog: Using Elasticsearch Mapping Types to Handle Different JSON Logs
By default, Elasticsearch does a good job of figuring the type of data in each field of your logs. But if you like your logs structured like we do, you probably want more control over how they’re indexed: is time_elapsed an integer or a float? Do you want your tags analyzed so you can search for big in big data? Or do you need it not_analyzed, so you can show top tags via the terms aggregation? Or maybe both?
In this post, we’ll look at how to use index templates to manage multiple types of logs across multiple indices. Also, we’ll explain how to use rsyslog to handle JSON logging and specify types.
Elasticsearch Mapping and Logs
To control settings for how a field is analyzed in Elasticsearch, you’ll need to define a mapping. This works similarly in Logsene, our log analytics SaaS, because it uses Elasticsearch and exposes its API.
With logs you’ll probably use time-based indices, because they scale better (in Logsene, for instance, you get daily indices). To make sure the mapping you define today applies to the index you create tomorrow, you need to define it in an index template.
Managing Multiple Types
Mappings provide a nice abstraction when you have to deal with multiple types of structured data. Let’s say you have two apps generating logs of different structures: both have a timestamp field, but one recording logins has a user field, and another one recording purchases has an amount field.
To deal with this, you can define the timestamp field in the _default_ mapping which applies to all types. Then, in each type’s own mapping we’ll define fields unique to that mapping. The following snippet is an example that works with Logsene, provided that aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee is your Logsene app token. If you roll your own Elasticsearch, you can use whichever name you want, and make sure the template name applies to matches index pattern (for example, logs-* will work if your indices are in the logs-YYYY-MM-dd format).
curl -XPUT 'logsene-receiver.sematext.com/_template/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate' -d '{ "template" : "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*", "order" : 21, "mappings" : { "_default_" : { "properties" : { "timestamp" : { "type" : "date" } } }, "firstapp" : { "properties" : { "user" : { "type" : "string" } } }, "secondapp" : { "properties" : { "amount" : { "type" : "long" } } } } }'
Sending JSON Logs to Specific Types
When you send a document to Elasticsearch by using the API, you have to provide an index and a type. You can use an Elasticsearch client for your preferred language to log directly to Elasticsearch or Logsene this way. But I wouldn’t recommend this, because then you’d have to manage things like buffering if the destination is unreachable.
Instead, I’d keep my logging simple and use a specialized logging tool, such as rsyslog, to do the hard work for me. Logging to a file is usually the easiest option. It’s local, and you can have your logging tool tail the file and send contents over the network. I usually prefer sockets (like syslog) because they let me configure rsyslog to:
– write events in a human format to a local file I can tail if I need to (usually in development)
– forward logs without hitting disk if I need to (usually in production)
Whatever you prefer, I think writing to local files or sockets is better than sending logs over the network from your application. Unless you’re willing to do a reliability trade-off and use UDP, which gets rid of most complexities.
Opinions aside, if you want to send JSON over syslog, there’s the JSON-over-syslog (CEE) format that we detailed in a previous post. You can use rsyslog’s JSON parser module to take your structured logs and forward them to Logsene:
module(load="imuxsock") # can listen to local syslog socket module(load="omelasticsearch") # can forward to Elasticsearch module(load="mmjsonparse") # can parse JSON action(type="mmjsonparse") # parse CEE-formatted messages template(name="syslog-cee" type="list") { # Elasticsearch documents will contain property(name="$!all-json") # all JSON fields that were parsed } action( type="omelasticsearch" template="syslog-cee" # use the template defined earlier server="logsene-receiver.sematext.com" serverport="80" searchType="syslogapp" searchIndex="LOGSENE-APP-TOKEN-GOES-HERE" bulkmode="on" # send logs in batches queue.dequeuebatchsize="1000" # of up to 1000 action.resumeretrycount="-1" # retry indefinitely (buffer) if destination is unreachable )
To send a CEE-formatted syslog, you can run logger ‘@cee: {“amount”: 50}’ for example. Rsyslog would forward this JSON to Elasticsearch or Logsene via HTTP. Note that Logsene also supports CEE-formatted JSON over syslog out of the box if you want to use a syslog protocol instead of the Elasticsearch API.
Filtering by Type
Once your logs are in, you can filter them by type (via the _type field) in Kibana:
However, if you want more refined filtering by source, we suggest using a separate field for storing the application name. This can be useful when you have different applications using the same logging format. For example, both crond and postfix use plain syslog.
rsyslog daily builds and tarballs
The past days, we have worked on making rsyslog daily builds and tarballs a reality. We hope this will enable users to rapidly deploy the latest features as well as make it easier to help with testing the current development system. Daily builds are what the scheduled v8-devel builds were under the previous release paradigm. Consequently, the archives are named v8-devel.
Right now, builds are only supported for Ubuntu. Users of other platforms are advised to use the daily tarballs to build from source. Depending on feedback on and success of the daily builds, we will make them available for more platforms.
A daily build is based on the latest git master version. So it really is at the [b]leading edge of technology. So why create them?
A top reason is that we often fix a bug for someone, and that someone then is unable to build from source. In the end result, we have a bugfix, but there is no external confirmation that it really fixed the bug when we merge it into the next release. We hope that now those users can simply pick the daily build and check if that solves their problem.
Also, in general we hope that some users will use the daily tarballs to get not only the latest and greatest but contribute to the project by doing some testing.
Finally, and quite important, with daily builds we will see build problems as early as possible. In the past, we often saw problems only after source release (or very close to it), which was obviously problematic. Now, this should no longer happen. For obvious reasons, the final release build is now more or less a copy of a daily build.
As a technical side-note, daily builds are identified by the git master branch head hash that was used to build them. As a forth version component, they have the first 12 digits of that hash (an example is “8.8.0.35e7f12a2c04”). This enables us to track error reports to the right version. The packages have a different version name, based on the build date. The reason is that the hash does not increment and so newer versions (with lower hash values) are considered as “old” by Launchpad. We avoid this by using an always incrementing package version. Also note that the package changelog just contains a “daily build” entry — anything else makes limited sense.
We hope you enjoy this new feature! Feedback is appreciated.
rsyslog 8.7.0 (v8-stable) released
We have released rsyslog 8.7.0.
Version 8.7.0 contains various improvements and additions to a wide array of modules, like imfile, imptcp, improvements to RainerScript and mmnormalize (thanks to Singh Janmejay) and a couple of other improvements. But, the biggest addition is the new omkafka module that now allows direct writing to Apache Kafka.
This release also contains important bug fixes.
This is a recommended upgrade for all users.
http://www.rsyslog.com/changelog-for-8-7-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl
Changelog for 8.7.0 (v8-stable)
Version 8.7.0 [v8-stable] 2015-01-13
- add message metadata “system” to msg object
this permits to store metadata alongside the message - imfile: add support for “filename” metadata
this is useful in cases where wildcards are used - imptcp: make stats counter names consistent with what imudp, imtcp uses
- added new module “omkafka” to support writing to Apache Kafka
- omfwd: add new “udp.senddelay” parameter
- mmnormalize enhancements
Thanks to Janmejay Singh for the patch. - RainerScript “foreach” iterator and array reading support
Thanks to Janmejay Singh for the patch. - now requires liblognorm >= 1.0.2
- add support for systemd >= 209 library names
- BSD “ntp” facility (value 12) is now also supported in filter
Thanks to Douglas K. Rand of Iteris, Inc. for the patch.
Note: this patch was released under ASL 2.0 (see email-conversation). - bugfix: global(localHostName=”xxx”) was not respected in all modules
- bugfix: emit correct error message on config-file-not-found
closes https://github.com/rsyslog/rsyslog/issues/173 - bugfix: impstats emitted invalid JSON format (if JSON was selected)
- bugfix: (small) memory leak in omfile’s outchannel code
Thanks to Koral Ilgun for reporting this issue. - bugfix: imuxsock did not deactivate some code not supported by platform
Among potential other problemns, this caused build failure under Solaris.
Note that this build problem just made a broader problem appear that so
far always existed but was not visible.
closes https://github.com/rsyslog/rsyslog/issues/185
rsyslog 8.6.0 (v8-stable) released
We have released rsyslog 8.6.0.
This is the first stable release under a new release cycle and versioning scheme. This new scheme is important news in itself. For more details, please have a look here:
http://www.rsyslog.com/rsyslogs-new-release-cycle-and-versioning-scheme/
Version 8.6.0 contains important new features like the ability to monitor files via wildcards in imfile. It also contains new, experimental zero message queue modules (special thanks to team member Brian Knox), improvements to RainerScript and mmnormalize (thanks to Singh Janmejay) and a couple of other improvements.
This release also contains important bug fixes.
This is a recommended upgrade for all users.
http://www.rsyslog.com/changelog-for-8-6-0-v8-stable/
Download:
http://www.rsyslog.com/downloads/download-v8-stable/
As always, feedback is appreciated.
Best regards,
Florian Riedl