What to do when an External Script does not work?
When a script runs fine interactively but fails in the rsyslog context (i.e., when executed by a background process or as part of a service) it typically indicates differences between the interactive environment and the service environment. Most importantly, it is not rsyslog itself you need to look at. Check the system config and debug your script so that you can obtain maximum information on why and when it fails.
If you know exactly that rsyslog cannot start the script, check that it has sufficient permissions and that all pathes are correct (use absolute pathes!).
Besides that general advice, here are some common reasons why the problem can happen and suggestions for debugging:
Continue reading “What to do when an External Script does not work?”Documentation Improvement and AI
For a long time, I struggled with the daunting task of enhancing the documentation for Rsyslog. My extensive knowledge of Rsyslog technology often made it challenging for me to create user-friendly documentation, especially for individuals with little to no syslog background. Additionally, as a non-native English speaker, I was aware that some of my sentences might be harder to understand than desired. But thanks to the breakthroughs in generative artificial intelligence (AI), the game has changed, and a new era of documentation improvement has begun.
Elevating Syslog Security: RSyslog Introduces DTLS Plugins for UDP
We at the RSyslog project are excited to share our recent advancements in syslog security. We have introduced initial plugins for Datagram Transport Layer Security (DTLS) syslog, namely imdtls (input module) and omdtls (output module). This development, which aligns with RFC 6012, represents a significant enhancement, albeit not a game-changer, in our continuous efforts to improve secure log transmission.
Additional improvements to rsyslog doc and site…
We’re excited to announce significant enhancements to the rsyslog website, designed to make your experience more efficient and enjoyable. Our primary focus has been on the documentation presentation, and we’ve implemented a range of upgrades across the site to reflect this.
Improving the rsyslog documentation…
The current state of rsyslog documentation and its representation on our official website has been a subject of concern within the professional community. We are initiating a comprehensive project aimed at systematically addressing these issues. Over the coming weeks, stakeholders can expect a series of methodical changes, some of which may be significantly transformative.
rsyslog on AWS – Applying Configuration Changes
Once you’ve updated the configuration of the AWS rsyslog application, it’s important to manually apply the new settings as rsyslog doesn’t do this automatically. This is to prevent partial changes from being loaded and potentially causing issues.
The AWS rsyslog AWS application provides a dedicated tool, rsyslogctl, which can be used to check and reload the configuration. During the reload process, rsyslogctl determines the most efficient way to apply the changes. For example, some changes like drop rules can be applied without interrupting message processing, while others require a full restart, causing a brief interruption.
Continue reading “rsyslog on AWS – Applying Configuration Changes”rsyslog on AWS – S3 file structure
The EBS disk included in the product is only used for day-to-day storage of logs. Persistent log storage is kept on an S3 store. This store also contains some other data items which should persist over upgrades of the rsyslog on AWS application.
The following prefixes/folders are used by rsyslog:
- /rsyslog.logstore/ – the actual logstore
This is synced with data from the local EBS disk once a day for the past day (in default settings). - /rsyslog.config/ – config data items.
This contains the user-based config which can be restored from here during an upgrade or on misconfiguration.
The users should select proper S3 policies based on her or his needs. Most importantly, Versioning and Retention Period should be set accordingly.
The S3 store to use can be configured during the cloud formation process and manually via the meta config.
rsyslog on AWS – an Overview
Our team at Adiscon offers a comprehensive paid full-service rsyslog product, available on the AWS Marketplace. As the same team that develops and supports the rsyslog open source project, we’re dedicated to providing exceptional service and ongoing innovation.
By purchasing our AWS Marketplace product, you’re also supporting the continued development of rsyslog. This ensures that the open source project remains robust, reliable, and up-to-date.
Our full-service rsyslog offering is designed specifically for organizations seeking a seamless and hassle-free way to collect syslog data on the cloud. We provide ongoing support and maintenance, along with regular updates to ensure the highest level of performance and security.
In summary, our AWS Marketplace product is the perfect solution for organizations that value simplicity, efficiency, and reliability when it comes to collecting syslog data in the cloud.
Documentation is available (and constantly being improved). Please follow these links:
Slightly Changed rsyslog Stable Release Cycle
For the past couple of years, rsyslog made scheduled releases every 6 weeks. We now changed this slightly to make version numbers easier to understand.
Remember, rsyslog versions are called 8.<yy><mm>.0, so the April 2020 release is 8.2004.0. When we release very six weeks, we get odd and even month numbers and, even more confusing, we sometimes seem to “skip” a month while at other times it looks like we craft a scheduled stable “every month”. To avoid this type of confusion, we have now decided to release every two month, and do that on even month.
We will usually try to release in the second half of the given month. However, we will no longer tell the exact target date. We need some flexibility here to avoid targeting “bad release periods”. As a concrete example, we will probably never do a December release during the holiday period. As such, December releases are more likely to happen in the first half of the month, which should give admins also some time to do all of their internal testing work ahead of the holidays.
We originally used the six week schedule to provide a balance between frequent bug fixes and not too frequent releases. With the appearance of daily stable releases a longer release cycle is no more a real concern. Everybody in need of a fix not yet present in the scheduled stable can just switch to the daily stable as needed. Remember that both are stable versions. The daily stable is often more stable as it contains the latest fixes.
Avoid overly-large in memory queues
Rsyslog provides the “queue.size” parameter to set a limit on the number of messages a queue can keep in memory. This is primarily meant to support peak traffic.
Note that this counter is given in number of messages, not bytes. A frequent mistake is to think in bytes and select very large values (e.g. 7 million frequently seen, maybe due to a web tutorial somewhere). If queues are that large there is a chance the rsyslog will be aborted by out of memory condition when the queue gets fuller and fuller.
An example. You send data to a remote syslog server. You define a very large queue on it. Usually, the queue keeps very slow. But when the system goes offline, the queue fills up. This will lead to sharply increasing memory usage. Depending on all circumstances this may not be a problem – or it may be! The likelihood of becoming problematic, and harder to reproduce, increases with the number of queues defined.
To avoid such misunderstandings, rsyslog starting at 8.1905.0 emits a warning message. It has probably lead you to this page. If the queue size is correct, you can ignore the warning message. You can also filter it out via regular rules, if you like. But if you did not intend to define such a large queue, please reconsider the value.
Note: rsyslog considers queues larger than 500,000 messages to be overly large – there seldom is a good reason to use sizes in excess of that.