Pipeline Design Patterns¶

Explores practical log pipeline design patterns in rsyslog — how to branch data to multiple destinations, chain rulesets for staged processing, and isolate workloads with separate queues.

Overview¶

A log pipeline is flexible by design. You can branch it, chain multiple rulesets together, or separate workloads into isolated queues. These design patterns help adapt rsyslog to different environments — from simple local logging to complex, multi-tenant ingestion systems.

Each pattern below includes a short diagram and explanation.

Fan-out (branching)¶

The most common pattern: send each message to multiple destinations in parallel. This provides redundancy or enables different consumers (archive, search, alerting).

        flowchart LR
  R["Ruleset"]:::ruleset --> A1["omfile<br>(local archive)"]:::action
  R --> A2["omrelp<br>(reliable ship)"]:::action
  R --> A3["omelasticsearch<br>(search index)"]:::action

  classDef ruleset fill:#dae8fc,stroke:#6c8ebf;
  classDef action fill:#ffe6cc,stroke:#d79b00;

How it works: Each action in a ruleset runs independently. rsyslog queues each action’s output, so one slow destination does not delay the others.

Example:

ruleset(name="MainFlow") {
  action(type="omfile" file="/var/log/app.log")
  action(type="omrelp" target="central.example.net" port="2514")
  action(type="omelasticsearch" server="es01" index="logs-app")
}

Use cases: - Store locally and forward reliably. - Split between long-term archive and search analytics. - Maintain an audit copy on local disk.

Staged processing (chaining)¶

Break complex logic into smaller, maintainable stages. Use call to hand off between rulesets.

        flowchart LR
  I["Input(s)"]:::input --> R1["Ruleset<br>parse / normalize"]:::ruleset --> R2["Ruleset<br>enrich / filter"]:::ruleset --> R3["Ruleset<br>route / output"]:::ruleset --> A["Actions"]:::action

  classDef input fill:#d5e8d4,stroke:#82b366;
  classDef ruleset fill:#dae8fc,stroke:#6c8ebf;
  classDef action fill:#ffe6cc,stroke:#d79b00;

How it works: Each ruleset performs a specific task and then calls the next one:

ruleset(name="ParseStage") {
  action(type="mmjsonparse" container="$!data")
  call "EnrichStage"
}

ruleset(name="EnrichStage") {
  if $!data!level == "debug" then stop
  call "OutputStage"
}

ruleset(name="OutputStage") {
  action(type="omfile" file="/var/log/processed.json")
}

Benefits: - Easier to test and reason about individual stages. - Enables modular reuse (different inputs can reuse the same enrich or output stage). - Failures are isolated to one stage.

Workload isolation (multi-ruleset queues)¶

Use different queues for different inputs or workloads to prevent one source from overloading another.

        flowchart LR
  I1["Remote TCP"]:::input --> QR[("Queue<br>remote")]:::queue --> RR["Ruleset<br>remote"]:::ruleset --> AR["Remote<br>actions"]:::action
  I2["Local system"]:::input --> QL[("Queue<br>local")]:::queue --> RL["Ruleset<br>local"]:::ruleset --> AL["Local<br>actions"]:::action

  classDef input fill:#d5e8d4,stroke:#82b366;
  classDef ruleset fill:#dae8fc,stroke:#6c8ebf;
  classDef action fill:#ffe6cc,stroke:#d79b00;
  classDef queue fill:#f5f5f5,stroke:#999999,stroke-dasharray: 3 3;

How it works: Each input is assigned its own ruleset and queue. This prevents bursty remote logs from starving local system messages.

Example:

module(load="imtcp")
input(type="imtcp" port="514" ruleset="RemoteFlow")

module(load="imuxsock")   # local system logs
input(type="imuxsock" ruleset="LocalFlow")

ruleset(name="RemoteFlow" queue.type="LinkedList" queue.size="10000") {
  action(type="omrelp" target="loghost" port="2514")
}

ruleset(name="LocalFlow" queue.type="LinkedList" queue.size="2000") {
  action(type="omfile" file="/var/log/local.log")
}

Best practices: - Choose queue sizes proportional to expected input volume. - Use fewer worker threads for disk-bound outputs. - Monitor queue backlog via impstats.

Pattern selection guide¶

The table below summarizes which design pattern fits common scenarios.

Scenario	Recommended pattern
Multiple destinations, same data	Fan-out
Complex processing sequence	Staged rulesets
Independent workloads	Workload isolation

Conclusion¶

Design patterns make rsyslog configurations scalable and maintainable. Start simple with a single ruleset, then add branching or staging as your log architecture grows. Combine these patterns with tuned queues for maximum throughput and resilience.