JSON Structured Content Extraction Module (mmjsonparse)¶

Module name:: mmjsonparse
Introduced:: 6.6.0 (find-json mode added in 8.2510)
Author:: Rainer Gerhards <rgerhards@adiscon.com>

Parses JSON-structured content in log messages and exposes fields as message properties. Supports legacy cookie-prefixed parsing and a flexible find-json scan mode. Provides scan counters to assess success, failure, and truncation.

Overview¶

mmjsonparse extracts JSON content from incoming messages and exposes parsed fields beneath the structured data container (for example, $!field). It supports two operating modes:

find-json (recommended): scans the message for the first valid top-level JSON object and parses it.
cookie (legacy): parses JSON only when the message starts with the legacy @cee: prefix.

Use mmjsonparse early in the processing pipeline to make JSON fields available for filtering, normalization, or routing. The cookie mode is retained for backward compatibility and should not be used in new deployments.

Introduced and Compatibility¶

The module has been available since rsyslog 6.6.0. The enhanced find-json scanning mode was introduced in version 8.2510 and is present in all later releases.

Behavior Notes¶

This is not a generic validator. If messages contain trailing non-JSON after the parsed object and trailing content is not allowed, parsing is considered failed according to the configured policy.
Parameter names are case-insensitive. For readability, camelCase is recommended.

Parsing Modes¶

Mode	Description
`find-json`	Scans for the first top-level `{...}` anywhere in the message and parses it. Suited for modern app logs that embed JSON within other text.
`cookie` (legacy)	Parses only when the message begins with `@cee:` and is immediately followed by valid JSON. Kept for backward compatibility with historic CEE/lumberjack emitters.

Configuration Parameters¶

Action Parameters¶

Parameter	Summary
mode	Specifies the parsing mode for JSON content detection.
cookie	Defines the cookie string that must appear before the JSON content of a message.
max_scan_bytes	Maximum number of bytes to scan when searching for JSON content in find-json mode.
allow_trailing	Whether to allow non-whitespace data after the JSON object in find-json mode.
useRawMsg	Controls whether parsing operates on the raw message or only the MSG part.
container	Sets the JSON container path where parsed properties are stored.

Check Parsing Result¶

Use the $parsesuccess variable to check whether JSON parsing succeeded.

action(type="mmjsonparse" mode="find-json")

if $parsesuccess == "OK" then {
    # downstream processing for structured messages
    action(type="omfile" file="/var/log/rsyslog/structured.log")
} else if $parsesuccess == "FAIL" then {
    # handle failures (see dedicated section below)
    action(type="omfile" file="/var/log/rsyslog/nonconforming.log")
    stop
}

Handling Parsing Failures¶

If JSON extraction fails, $parsesuccess is set to "FAIL". Best practice is to keep such messages separate for inspection. Two common patterns are shown below.

A) Direct to non-conforming log with stop¶

action(type="mmjsonparse" mode="find-json")

if $parsesuccess == "FAIL" then {
    action(
      type="omfile"
      file="/var/log/rsyslog/nonconforming.log"
      template="RSYSLOG_TraditionalFileFormat"
    )
    stop    # prevent accidental downstream processing
}

# OK path continues as usual
if $parsesuccess == "OK" then {
    action(type="omfile" file="/var/log/rsyslog/structured.log")
}

B) Handoff to a dedicated inspection ruleset (with stop)¶

ruleset(name="inspectNonConforming"){
    # capture raw for triage
    template(name="rawmsg" type="string" string="%rawmsg%\\n")
    action(type="omfile" file="/var/log/rsyslog/nonconforming.raw" template="rawmsg")
    # optional: richer snapshot for short-term debugging
    # action(type="omfile" file="/var/log/rsyslog/nonconforming.meta" template="RSYSLOG_DebugFormat")
    stop
}

action(type="mmjsonparse" mode="find-json")

if $parsesuccess == "FAIL" then {
    call inspectNonConforming
} else {
    action(type="omfile" file="/var/log/rsyslog/structured.log")
}

Operational recommendations¶

Use stop on the failure path if you must ensure the message does not reach later actions unintentionally. Omit stop if you intentionally want both paths.
Apply a distinct retention policy to non-conforming logs; they can be bursty.
For transient debugging, log %rawmsg% (and, if supported, an all-JSON snapshot) to aid triage.
Run rsyslogd -N1 after edits to validate configuration syntax.
Consider preceding mmjsonparse with mmutf8fix if invalid encodings are suspected.

Statistics Counters¶

When using find-json mode, mmjsonparse maintains scan-related counters (available via rsyslog’s statistics subsystem):

Counter	Description
`scan.attempted`	Messages inspected by the find-json scanner.
`scan.found`	Messages where a JSON object was located (may still fail later on trailing rules).
`scan.failed`	Messages where a located JSON object could not be parsed (malformed).
`scan.truncated`	Scans aborted due to the `max_scan_bytes` limit.

Exposure and collection¶

impstats (recommended): Configure impstats to emit periodic statistics as messages.
Prometheus via imhttp: If imhttp is loaded with metricsPath, it exposes rsyslog statistics (including mmjsonparse counters) in Prometheus text format at the configured HTTP path. See imhttp’s Prometheus Metrics section and secure endpoints as needed (for example, metricsBasicAuthFile).

Performance guidance¶

Success rate: scan.found / scan.attempted — ability to locate JSON.
Parse quality: scan.failed / scan.found — data quality problems.
Scan efficiency: scan.truncated / scan.attempted — raise max_scan_bytes if frequent.

Processing Flow (informative)¶

Note

The diagrams below are informational and do not alter behavior.

        flowchart TD
  A["msg in"] --> B["mmjsonparse<br>mode=find-json|cookie"]
  B -->| OK | C["fields under $!"]
  C --> D["structured path"]
  B -->| FAIL | E["non-conforming"]
  E --> F{"route"}
  F -->| file+stop | G["omfile<br>nonconforming.log"]
  F -->| ruleset+stop | H["inspectNonConforming()"]

Metrics exposure (optional)¶

        flowchart LR
  M["mmjsonparse counters"] --> S["stats registry"]
  S --> I["imhttp<br>/metrics"]
  I --> P["Prometheus scrape"]

Examples¶

Find-JSON mode for embedded JSON content¶

# Basic find-json mode
action(type="mmjsonparse" mode="find-json")

# With limits and strict trailing control
action(type="mmjsonparse"
       mode="find-json"
       max_scan_bytes="32768"
       allow_trailing="off")

Mixed mode processing¶

if $msg startswith "@cee:" then {
    action(type="mmjsonparse" mode="cookie")
} else if $msg contains "{" then {
    action(type="mmjsonparse" mode="find-json")
}

JSON Structured Content Extraction Module (mmjsonparse)¶

Overview¶

Introduced and Compatibility¶

Behavior Notes¶

Notable Features¶

Parsing Modes¶

Configuration Parameters¶

Action Parameters¶

Check Parsing Result¶

Handling Parsing Failures¶

A) Direct to non-conforming log with stop¶

B) Handoff to a dedicated inspection ruleset (with stop)¶

Operational recommendations¶

Statistics Counters¶

Exposure and collection¶

Performance guidance¶

Processing Flow (informative)¶

Metrics exposure (optional)¶

Examples¶

Enable default normalization (legacy cookie mode)¶

Permit parsing messages without cookie (do not require @cee:)¶

Find-JSON mode for embedded JSON content¶

Mixed mode processing¶

See also¶