Learn how to push rsyslog events into Vespa.ai’s search platform with omhttp and Vespa’s Document v1 API.

FAQ: Sending Logs from rsyslog to Vespa.ai¶

Vespa.ai is an open-source engine for real-time search, recommendation, and analytics at scale. It ingests structured documents, indexes them for low-latency search, and executes machine-learned ranking or aggregations directly where the data lives. Connecting rsyslog to Vespa.ai lets you ship data events into a streaming analytics platform that can keep indexes and features up-to-date while serving low-latency queries across massive datasets.

How do I send rsyslog messages to Vespa.ai?¶

Use the omhttp output module to POST JSON documents to Vespa’s Document v1 API. Start from the following template, which uses omhttp to POST individual rsyslog messages as Vespa documents with randomly generated IDs:

module(load="omhttp")

# Template that outputs the rsyslog message field inside a Vespa document
template(name="json-message" type="list" option.json="on") {
    constant(value="{ \"fields\": { \"message\": \"")
    property(name="msg")
    constant(value="\" }}")
}

# Template that generates a unique document path per message
template(name="vespaDocUrl" type="string"
  string="/document/v1/NAMESPACE/DOCTYPE/docid/%uuid%"
)

# Send each message to Vespa using HTTP POST
action(type="omhttp"
       server="vespa"
       serverport="8080"
       useHttps="off"
       batch="off"
       dynrestpath="on"
       restpath="vespaDocUrl"
       template="json-message")

Replace NAMESPACE and DOCTYPE with your Vespa application values. Set server and serverport to match the Vespa container endpoint (localhost and 8080 when running Vespa locally). When targeting Vespa Cloud or other TLS-protected endpoints, switch useHttps to on and provide the relevant certificates as described in the omhttp module reference.

Why is this integration useful?¶

Operational visibility: Index rsyslog events alongside other Vespa documents so operators can search by message text, host metadata, or custom JSON fields in real time.
Feature serving: Vespa can expose recent data-derived features directly to downstream applications without requiring a separate streaming or ETL layer.
Scalability: Vespa distributes both storage and query execution, allowing the data pipeline to scale with the same cluster that serves analytics queries.

Tips for production use¶

Define richer JSON documents by extending the json-message template with additional fields (for example, structured data from mmjsonparse or custom message properties).
Enable impstats to monitor delivery throughput and retry behavior when Vespa slows down or becomes temporarily unavailable. Consider imhttp if you want to publish a Prometheus metrics endpoint.
Review Vespa’s Document v1 API guide for additional operations such as updates and deletes.

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project