Diag: grok rule matches nothing, test with a raw line, check order and filters
Runbook for pipeline grok rules that never match: pull a raw line from Live Tail, test in the pipeline editor, check rule order and pipeline filters.
Diag: grok rule matches nothing
Symptom (exact): logs ingest, but the attributes the grok rule should extract are absent; message field holds the raw line. Pipeline shows the rule, zero matches.
Likely causes: (a) log format drifted from the sample the rule was written against, (b) processor order wrong, (c) pipeline filter excludes these logs, (d) multiline input.
Confirm:
- Live Tail: copy one raw log line, exactly as ingested.
- Open the pipeline, the grok rule, paste the line into the rule tester. No match confirms (a): diff the line against the pattern token by token.
- If it matches in the tester but not in production, check the pipeline filter (source/service/env) against the log attributes in Live Tail. Mismatch means (c).
- Check processor order: is a JSON parser or another grok running before this rule and reshaping the message? That is (b).
- If the raw line is one line of a stack trace, it is (d): the rule only ever sees fragments.
Fix: update the pattern to the real format (or better, switch the app to JSON logs and use the JSON parser), fix the filter, reorder processors, or add multiline aggregation before grok.
Verify: re-test with a fresh raw line, save, watch Live Tail for the attributes appearing on new logs. Old logs are not reprocessed: verification is always on new data.