rsyslog Goes AI First — A New Chapter Begins
After 24 months of focused evaluation and careful experimentation, we’re excited to announce a major shift in the evolution of rsyslog: we’re going AI First.
This marks the beginning of a strategic transformation in how we design, develop, and support rsyslog and its ecosystem. While today’s post is just a short announcement, it lays the groundwork for a series of updates to follow — including deep dives into what we’re doing, why, and how it benefits you.
Revitalizing Rsyslog with Docker: A New Era of Log Management
We’re excited to announce a renewed focus on the rsyslog Docker project, bringing you robust, flexible, and easy-to-use containerized solutions for your logging needs. This isn’t just a refresh; it’s a reimagining of how rsyslog can integrate into modern, containerized environments.
Slightly Changed rsyslog Stable Release Cycle
For the past couple of years, rsyslog made scheduled releases every 6 weeks. We now changed this slightly to make version numbers easier to understand.
Remember, rsyslog versions are called 8.<yy><mm>.0, so the April 2020 release is 8.2004.0. When we release very six weeks, we get odd and even month numbers and, even more confusing, we sometimes seem to “skip” a month while at other times it looks like we craft a scheduled stable “every month”. To avoid this type of confusion, we have now decided to release every two month, and do that on even month.
We will usually try to release in the second half of the given month. However, we will no longer tell the exact target date. We need some flexibility here to avoid targeting “bad release periods”. As a concrete example, we will probably never do a December release during the holiday period. As such, December releases are more likely to happen in the first half of the month, which should give admins also some time to do all of their internal testing work ahead of the holidays.
We originally used the six week schedule to provide a balance between frequent bug fixes and not too frequent releases. With the appearance of daily stable releases a longer release cycle is no more a real concern. Everybody in need of a fix not yet present in the scheduled stable can just switch to the daily stable as needed. Remember that both are stable versions. The daily stable is often more stable as it contains the latest fixes.
rsyslog version numbering change
Rsyslog used a version number scheme of
8.<real-version>.0
where we incremented <real-version> every 6 weeks with each release. The 8 and 0 are constant (well, the 0 could change to 1 with a very important patch, but in practice we have only done this once).
While this scheme has worked pretty well since we introduced it, we often see people not understanding that there is really a big difference between 8.24 and e.g. 8.40. Followind recent trends in software versioning, we will make more clear how old a version really is. Begining with today’s release, we change the version number slightly to
8.yymm.0
where yy is the two-digit year and mm the two-digit month of the release date. We release every 6 weeks, so we will never have two releases within the same month.
So while you expected 8.41.0, you will now get 8.1901.0. To make things even more clear, rsyslog visible version output will be even more up to the point: rsyslog -v will now report “8.1901.0 (aka 2019.01)“.
Rainer Gerhards’ blog has more details on why we did this change and how we came to the new system.
New Logo Selected
The logo was originally contributed in 2014 by “robert s”, whom unfortunately I am no longer able to contact. While before we never officially adopted it, it went into widespread use and is already often used to represent rsyslog. So in a sense the now-official selection let’s us keep consistent.
We are glad to have the community decision. I am right now implementing the new logo all over rsyslog web spaces. It will also be available via the rsyslog website github project (PR just created).
Many thanks to all who voted. It was a pleasant experience for us. This may have also set stage for future polls on different topics.
What are your thoughts regarding current and potential rsyslog support channels?
Overview
Traditionally the rsyslog community has sought and provided support through three main channels:
- mailing list
- forums
- ticketing system (at one time Bugzilla, now GitHub)
Over the years, the community support options have shifted to the point that we are considering retiring the forums in order to best direct users that post there to other, more current options that better fit their needs. It would appear that aside from specific cases, the time of the web forum has passed.
That said, we would like to get your feedback to best determine the way to move forward. What follows are some initial ideas to get the conversation started. Please feel free to respond here, via Twitter, the mailing list or on GitHub. Thank you for your time.
Potential Support options
The following items are all “whiteboard” topics, listed in no real order in an effort to start discussion. Neither the order or presence in the list indicates that a decision has already been made by the team to pursue the support option. Please feel free to suggest your own entries.
Keep the forums, send notifications of new posts made on the forums to the mailing list
- Note: Attempts to respond to those notifications would not result in the replies being posted to the original topic on the forums.
- Would this truly result in any additional responses to those forum posts than are currently being provided now?
Set forums to read-only, direct visitors to GitHub for support
Set forums to read-only, direct visitors to StackOverflow
It would appear there is already solid participation there for questions tagged with rsyslog:
https://stackoverflow.com/questions/tagged/rsyslog
Official Twitter presence
followers are encouraged to retweet rsyslog related questions, guides, etc to their followers.
This is actually a “trick” entry of sorts! We already have a Twitter account that you can follow and interact with: @rsyslog
- Do you already follow that account?
- Would you retweet content from others?
- Would you respond to help requests that are retweeted
- If links to active GitHub issues are posted, will you take the time to go view them?
Official Facebook presence
Would you participate in discussions and support requests made there?
IRC, XMPP, Slack, …
- Would you participate?
- Do you feel this could replace the forums?
- Would this be more useful to you than the mailing list?
rsyslog error reporting improved
Rsyslog provides many up-to-the point error messages for config file and operational problems. These immensly helps when troubleshooting issues. Unfortunately, many users never see them. The prime reason is that most distros do never log syslog.* messages and so they are just throw away and invisible to the user. While we have been trying to make distros change their defaults, this has not been very successful. The result is a lot of user frustration and fruitless support work for the community — many things can very simple be resolved if only the error message is seen and acted on.
We have now changed our approach to this. Starting with v8.21, rsyslog now by default logs its messages via the syslog API instead of processing them internally. This is a big plus especially on systems running systemd journal: messages from rsyslogd will now show up when giving
$ systemctl status rsyslog.service
This is the place where nowadays error messages are expected and this is definitely a place where the typical administrator will see them. So while this change causes the need for some config adjustment on few exotic installations (more below), we expect this to be something that will generally improve the rsyslog user experience.
Along the same lines, we will also work on some better error reporting especially for TLS and queue-related issues, which turn out high in rsyslog suport discussions.
Some fine details on the change of behaviour:
Note: you can usually skip reading the rest of this post if you run only a single instance of rsyslog and do so with more or less default configuration.
The new behaviour was actually available for longer, It needed to be explicitly turned on in rsyslog.conf via
global(processInternalMessages="off")
Of course, distros didn’t do that by default. Also, it required rsyslog to be build with liblogging-stdlog, what many distros do not do. While our intent when we introduced this capability was to provide the better error logging we now have, it simply did not turn out in practice. The original approach was that it was less intrusive. The new method uses the native syslog() API if liblogging-stdlog is not available, so the setting always works (we even consider moving away from liblogging-stdlog, as we see this wasn’t really adopted). In essence, we have primarily changed the default setting for the “processInternalMessages” parameter. This means that by default, internal messages are no longer logged via the internal bridge to rsyslog but via the syslog() API call [either directly or
via liblogging). For the typical single-rsyslogd-instance installation this is mostly unnoticable (except for some additional latency). If multiple instances are run, only the “main” (the one processing system log messages) will see all messages. To return to the old behaviour, do either of those two:
- add in rsyslog.conf:
global(processInternalMessages="on") - export the environment variable
RSYSLOG_DFLT_LOG_INTERNAL=1
This will set a new default – the value can still be overwritten via rsyslog.conf (method 1). Note that the environment variable must be set in your startup script (which one is depending on your init system or systemd configuration).
Note that in most cases even in multiple-instance-setups rsyslog error messages were thrown away. So even in this case the behaviour is superior to the previous state – at least errors are now properly being recorded. This also means that even in multiple-instance-setups it often makes sense to keep the new default!
rsyslog daily builds and tarballs
The past days, we have worked on making rsyslog daily builds and tarballs a reality. We hope this will enable users to rapidly deploy the latest features as well as make it easier to help with testing the current development system. Daily builds are what the scheduled v8-devel builds were under the previous release paradigm. Consequently, the archives are named v8-devel.
Right now, builds are only supported for Ubuntu. Users of other platforms are advised to use the daily tarballs to build from source. Depending on feedback on and success of the daily builds, we will make them available for more platforms.
A daily build is based on the latest git master version. So it really is at the [b]leading edge of technology. So why create them?
A top reason is that we often fix a bug for someone, and that someone then is unable to build from source. In the end result, we have a bugfix, but there is no external confirmation that it really fixed the bug when we merge it into the next release. We hope that now those users can simply pick the daily build and check if that solves their problem.
Also, in general we hope that some users will use the daily tarballs to get not only the latest and greatest but contribute to the project by doing some testing.
Finally, and quite important, with daily builds we will see build problems as early as possible. In the past, we often saw problems only after source release (or very close to it), which was obviously problematic. Now, this should no longer happen. For obvious reasons, the final release build is now more or less a copy of a daily build.
As a technical side-note, daily builds are identified by the git master branch head hash that was used to build them. As a forth version component, they have the first 12 digits of that hash (an example is “8.8.0.35e7f12a2c04”). This enables us to track error reports to the right version. The packages have a different version name, based on the build date. The reason is that the hash does not increment and so newer versions (with lower hash values) are considered as “old” by Launchpad. We avoid this by using an always incrementing package version. Also note that the package changelog just contains a “daily build” entry — anything else makes limited sense.
We hope you enjoy this new feature! Feedback is appreciated.
rsyslog’s new release cycle and versioning scheme
With today’s release of rsyslog 8.6.0, we start a new release schedule and versioning scheme. In a nutshell, we will be doing stable releases every six weeks now, and devel releases will be distributed via git exclusively.
We have made this move after reflecting the changes in user participation in open source development as well as analyzing what big projects like the Linux kernel, Firefox and Chrome are doing. I am very excited about the new methodology and sincerely hope it will make new features even more readily available to a large user base. Details on the new system are in the embedded presentation.
rsyslog v8 improvements and how to write plugins in any language
In the first part, we will explain the new RSYSLOG v8 engine, its motivation and its benefits. Learn, for example, why writing to Elasticsearch is much faster with the new engine. We will describe the tuning parameters vital for making best use of the new features.
In the second part we will explain how to write RSYSLOG plugins in any language. Traditionally, writing rsyslog plugins has been considered quite hard, with at least C knowledge necessary. In v8, we have introduced new interfaces which make it possible to write plugins in any language – be it Python, Perl or Java. Even bash will do. In essence, this is a great tool for any admin to add special needs with just a bit of scripting. We will proivde concrete instructions on how to write a plugin, point to read-to-copy samples and tell how to integrate this into rsyslog.
NOTE: This is Rainers LinuxTag Berlin 2014 talk.