rsyslog

The rocket-fast system for log processing

AWS rsyslog – Meta Configuration

The “meta configuration” is provides a simple-to-use way for rsyslog server customization, without the need to manage rsyslog.conf and it’s helpers directly.

The configuration is stored inside the text file “rsyslog-features.env” in “/opr/rsyslog/config“. Each line consists of a keyword, an equal sign, and an associated value. The AMI file contains this file with default settings. The CloudFormation script generates an initial file based on data provided by the user during the CloudFormation process.

This is an example of a meta-config file:

RSYSLOG_PSTATS_ENABLED="on"
RSYSLOG_PSTATS_FILE_ENABLED="on"
RSYSLOG_PSTATS_CLOUDWATCH_ENABLED="off"
RSYSLOG_LOGSTORE_PATH="/var/log/rsyslog.store/"
RSYSLOG_S3_ID="rsyslogfiles"
RSYSLOG_S3_LOGSTORE_PATH="/rsyslog.store/"
RSYSLOG_S3_CONFIG_PATH="/rsyslog.config/"
    
LOGROTATE_COMPRESS="zstd"

Editing the Meta-Config

Currently no specific feature for editing the meta-config exists. We suggest to use the text editor of your choice to modify the text file directly. Note: the file currently is writeable for the “root” user, only.

Configuration Parameters

Note: Parameter values contain either a string or number or the special values “on” and “off” to turn specific features on or off.

Internal Statistic Counters

  • RSYSLOG_PSTATS_ENABLED
    Turns the rsyslog-internal statistics system (impstats) completely on or off. If any other pstats-related settings are to be used, this must be set to “on”.
  • RSYSLOG_PSTATS_FILE_ENABLED
    Turns on or off writing statistics records to a local log file.
  • RSYSLOG_PSTATS_CLOUDWATCH_ENABLED
    Turns on or off pushing select statistics counters to the AWS CloudWatch Dashboard.

Please note that sending statistics to CloudWatch is a priced item in AWS. For that very reason, we also permit to write statistics to local files, which involves no cost but storage fees. File writing and CloudWatch push can be turned on or off independently. Both can be used together.

Storage Options

The storage options permit to assign where persistent storage is located.

  • RSYSLOG_LOGSTORE_PATH
    Full path name where logs incoming from the network should be stored. On this store, a two-level directory structure is automatically be created: first level is the hostname, second level is the syslog tag from the incoming message. For security reasons, slashes inside hostname or tag are replaced by underscores. The log store is automatically rotated when log rotation is turned on.
  • RSYSLOG_S3_ID
    The id (name) of the S3 store to be used for persistent storage. Leave empty to disable automatic synchronizing to S3. The S3 store’s structure is described here.
    Note: automatic data sync to S3 happens after log rotation, which is daily run shortly after midnight.
  • RSYSLOG_S3_LOGSTORE_PATH
    Like the local LOGSTORE_PATH, this is the “path” (prefix) used inside the S3 store for log files.
    Note: if RSYSLOG_S3_ID is empty, this parameter is ignored.
  • RSYSLOG_S3_CONFIG_PATH
    The “home path” for persistent storage of the rsyslog user configuration.
    Note: if RSYSLOG_S3_ID is empty, this parameter is ignored.

Important: The beta version’s default log store is not persistent and will not survive instance termination. To ensure data is not lost, users should configure an external EBS disk for persistent storage. Currently, mount settings must be created manually as there is no automatic setting. However, this feature is planned for future beta versions.

Log Rotation

Our application rotates log files daily shortly after midnight using advanced rsyslog capabilities and our own custom logic. We do not utilize the logrotate tool. As part of the log rotation process, the previous day’s log files are compressed (if the compression feature is enabled) and then synced to an S3 bucket (if the RSYSLOG_S3_ID configuration variable is set). After the log files have been successfully synced, our local retention policy is applied to manage disk space usage.

It’s important to note that our application does not enforce any retention policy for S3 buckets. Therefore, it’s crucial to set an appropriate retention policy for your S3 bucket using AWS tooling to avoid exceeding your desired storage limits or incurring unnecessary costs. Please ensure that you set a retention policy that meets your specific business requirements.

The configuration can be customized via these settings:

  • LOGROTATE_COMPRESS
    Specifies the compression method to be used during rotation. Currently, the available options are “no” and “zstd”. If set to “no”, no compression will be applied.
  • LOGROTATE_LOCAL_RETENTION
    Specifies the number of days that log files should be kept locally. After the specified period, they will be deleted. To delete local log files immediately after they have been synced to S3, set the value to zero..

Applying Configuration Changes

Important: after changing the meta configuration, be sure to check it against any typos or other problems. This can be done via the command sudo rsyslogctl check-config. If this is error-free, you can apply the new config via rsyslogctl reload-config.

For more information on how to apply configuration changes to rsyslog on AWS configuration changes, please refer to this dedicated page.

Back to aws rsyslog overview.

Scroll to top