omazureeventhubs: Microsoft Azure Event Hubs Output Module¶

Module Name:	omazureeventhubs
Author:	Andre Lorbach <alorbach@adiscon.com>
Available since:	v8.2304

Purpose¶

The purpose of the rsyslog output plugin omazureeventhubs is to provide a fast and reliable way to send log data from rsyslog to Microsoft Azure Event Hubs. This plugin uses the Advanced Message Queuing Protocol (AMQP) to securely transmit log data from rsyslog to Microsoft Azure, where it can be centralized, analyzed, and stored. The plugin uses the “Qpid Proton C API” library to implement the AMQP protocol, providing a flexible and efficient solution for sending log data to Microsoft Azure Event Hubs.

AMQP is a reliable and secure binary protocol for exchanging messages between applications, and it is widely used in the cloud and enterprise messaging systems. The use of AMQP in the omazureeventhubs plugin, in combination with the Qpid Proton C API library, ensures that log data is transmitted in a robust and reliable manner, even in the presence of network outages or other disruptions.

The omazureeventhubs plugin supports various configuration options, allowing organizations to customize their log data pipeline to meet their specific requirements. This includes options for specifying the Event Hubs endpoint, port, and authentication credentials. With this plugin, organizations can easily integrate their rsyslog infrastructure with Microsoft Azure Event Hubs, providing a scalable and secure solution for log management. The plugin is designed to work with the latest versions of rsyslog and Microsoft Azure, ensuring compatibility and reliability.

Requirements¶

To output logs from rsyslog to Microsoft Azure Event Hubs, you will need to fulfill the following requirements:

Qpid Proton C Library Version 0.13 or higher including Qpid Proton ProActor
The AMQP Protocol needs to have firewall Ports 5671 and 443 TCP to be open for outgoing connections.

Configuration Parameters¶

Note

Parameter names are case-insensitive; camelCase is recommended for readability.

Warning

Older guidance mentioned a closeTimeout parameter, but the module does not implement it. Configurations that reference closeTimeout will fail to load.

Input Parameters¶

Parameter	Summary
azurehost	Sets the fully qualified Event Hubs namespace host the action connects to.
azureport	Defines the TCP port used when connecting to the Event Hubs namespace.
azure_key_name	Specifies the shared access key name used to authenticate with Event Hubs.
azure_key	Provides the shared access key value used to sign Event Hubs requests.
container	Identifies the Event Hubs instance that receives the formatted log messages.
template	Selects the rsyslog template that formats messages sent to Event Hubs.
amqp_address	Provides a full AMQPS connection string that overrides individual Azure settings.
eventproperties	Adds custom key-value properties to each AMQP message emitted by the action.
statsname	Names the statistics counters that track this action instance.

Statistic Counter¶

This plugin maintains global statistics for omazureeventhubs that accumulate all action instances. The statistic origin is named “omazureeventhubs” with following counters:

submitted - This counter tracks the number of log messages that have been submitted by the rsyslog process to the output module for delivery to Microsoft Azure Event Hubs.
accepted - This counter tracks the number of log messages that have been successfully delivered to Microsoft Azure Event Hubs by the output module.
failures - This counter tracks the number of log messages that have failed to be delivered to Microsoft Azure Event Hubs due to various error conditions, such as network connectivity issues, incorrect configuration settings, or other technical problems. This counter provides important information about any issues that may be affecting the delivery of log data to Microsoft Azure Event Hubs.
failures_other - This counter tracks the number of log messages that have failed to be delivered due to other error conditions, such as incorrect payload format or unexpected data.

These statistics counters are updated in real-time by the rsyslog output module as log data is processed, and they provide valuable information about the performance and operation of the output module.

For multiple actions using statistics callback, there will be one record for each action.

Examples¶

Example 1: Use AMQP URL¶

The following sample does the following:

loads the omazureeventhubs module
outputs all logs to Microsoft Azure Event Hubs with standard template
Uses amqp_address parameter

module(load="omazureeventhubs")
action(type="omazureeventhubs" amqp_address="amqps://<AccessKeyName>:<AccessKey>@<EventHubsNamespace>.servicebus.windows.net/<EventHubsInstance>")

Example 2: RAW Format¶

The following sample does the following:

loads the omazureeventhubs module
outputs all logs to Microsoft Azure Event Hubs with simple custom template
Uses azurehost, azureport, azure_key_name and azure_key parameters instead of amqp_address parameter

module(load="omazureeventhubs")
template(name="outfmt" type="string" string="%msg%\n")

action(type="omazureeventhubs"
     azurehost="<EventHubsNamespace>.servicebus.windows.net"
     azureport="5671"
     azure_key_name="<AccessKeyName>"
     azure_key="<AccessKey>"
     container="<EventHubsInstance>"
     template="outfmt"
)

Example 3: JSON Format¶

The following sample does the following:

loads the omazureeventhubs module
outputs all logs to Microsoft Azure Event Hubs with JSON custom template
Uses azurehost, azureport, azure_key_name and azure_key parameters instead of amqp_address parameter
Uses eventproperties array parameter to set additional message properties

module(load="omazureeventhubs")
template(name="outfmtjson" type="list" option.jsonf="on") {
     property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
     constant(value="\"source\": \"EventHubMessage\", ")
     property(outname="host" name="hostname" format="jsonf")
     property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number")
     property(outname="facility" name="syslogfacility" format="jsonf" datatype="number")
     property(outname="appname" name="syslogtag" format="jsonf")
     property(outname="message" name="msg" format="jsonf" )
     property(outname="etlsource" name="$myhostname" format="jsonf")
}

action(type="omazureeventhubs"
     azurehost="<EventHubsNamespace>.servicebus.windows.net"
     azureport="5671"
     azure_key_name="<AccessKeyName>"
     azure_key="<AccessKey>"
     container="<EventHubsInstance>"
     template="outfmtjson"
     eventproperties=[       "Table=CustomTable",
                             "Format=JSON"]
)

Example 4: High Performance¶

To achieve high performance when sending syslog data to Azure Event Hubs, you should consider configuring your output module to use multiple worker instances. This can be done by setting the “workerthreads” parameter in the configuration file.

The following example is for high performance (Azure Premium Tier) and does the following:

loads the omazureeventhubs module
outputs all logs to Microsoft Azure Event Hubs with JSON custom template
Uses azurehost, azureport, azure_key_name and azure_key parameters instead of amqp_address parameter
Uses eventproperties array parameter to set additional message properties
Uses Linkedlist In-Memory Queue which enables multiple omazureeventhubs workers running at the same time. Using a dequeue size of 2000 and a dequeue timeout of 1000 has shown very good results in performance tests.
Uses 8 workerthreads in this example, which will be spawn automatically if more than 2000 messages are waiting in the Queue. To achieve more performance, the number can be incremented.

module(load="omazureeventhubs")
template(name="outfmtjson" type="list" option.jsonf="on") {
     property(outname="timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
     constant(value="\"source\": \"EventHubMessage\", ")
     property(outname="host" name="hostname" format="jsonf")
     property(outname="severity" name="syslogseverity" caseConversion="upper" format="jsonf" datatype="number")
     property(outname="facility" name="syslogfacility" format="jsonf" datatype="number")
     property(outname="appname" name="syslogtag" format="jsonf")
     property(outname="message" name="msg" format="jsonf" )
     property(outname="etlsource" name="$myhostname" format="jsonf")
}

action(type="omazureeventhubs"
     azurehost="<EventHubsNamespace>.servicebus.windows.net"
     azureport="5671"
     azure_key_name="<AccessKeyName>"
     azure_key="<AccessKey>"
     container="<EventHubsInstance>"
     template="outfmtjson"
     eventproperties=[       "Table=CustomTable",
                             "Format=JSON"]
     queue.type="linkedList"
     queue.size="200000"
     queue.saveonshutdown="on"
     queue.dequeueBatchSize="2000"
     queue.minDequeueBatchSize.timeout="1000"
     queue.workerThreads="8"
     queue.workerThreadMinimumMessages="2000"
     queue.timeoutWorkerthreadShutdown="10000"
     queue.timeoutshutdown="1000"
)

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project