Announcements :: announcing rsyslog 3.11.0 - first open source disk buffering syslogd :: (552 Reads)
Posted by
rgerhards
on Thursday, January 31, 2008 - 11:49 PM
It is a great pleasure for me to do a fully buzzword-compliant announcement for rsyslog 3.11.0, which is available immediately. This release is a very important milestone: it is the first time ever that the full, completely new queuing engine is in place.
Now rsyslog is the world's first-ever purely open-sourced syslogd supporting disk based buffering individually for each log destination[*] (called "action" in rsyslog speak). Besides, each action can now be executed on its own thread. Together with the multi-threaded main worker pool, rsyslog is more than ready for tomorrow's massively multi-core machines. I think another "world's first" is the ability to automatically discard low priority syslog messages under congestion conditions.
The new engine provides, among others:
reliable storage of messages while a destination is down
persistence of message even when rsyslogd is shut down
automatic retry of failed actions until they succeed, of course without any message loss
unparalleled ability to handle message burst
fully automatic worker thread pools
slow actions (e.g. database writes) are de-coupled from quick local actions (e.g. file writes)
rate-limited action processing
flow control slows senders down when queues get full
ability to discard less important messages in favor of higher important ones when the queue runs out of space
All of this while using minimal system resources.
The system can be set up with few configuration commands, but provides very in-depth commands to manipulate aspects like worker thread startup and shutdown parameters, queue sizes, disk quotas, watermark algorithm settings and many more (sometimes exotic) tweaking knobs.
Rsyslog 3.11.0 is fresh off the "development press". While it is well tested in lab, I expect that there are bugs in it, especially given the magnitude of changes and the massively multithreaded processing flow. As such, I do not yet advise to use it on important production machines.
I would, however, be very interested in feedback from the field, including bug reports. My focus in the next weeks will be on making the new engine rock-solid while at the same time introducing many new features which are quite easy to build on top of the new engine (like more advanced rate limiting algorithms, action execution-time windows and more advanced ways to flow-control inputs). My hope is that within two month or so we will have a very stable version suitable for production use in the most demanding environments.
I hope that this announcement will motivate some people to try rsyslog out and provide feedback to the project. Just be warned that I will not be at my desk next week: I am taking my now hopefully deserved break after a long time of hard coding. I thought to delay the announcement, but have decided it is probably better to get some head start. So please do not be disappointed if I do not reply immediately on comments posted. For comments and questions, I recommend using either the mailing list or the web forum at
