Loki "too many labels": cardinality limits fix
Fixes Loki pushes rejected with 'too many labels'. Use when a stream exceeds the label-name limit; dropping junk high-cardinality labels at the collector with relabel rules is the fix, with structured metadata as the alternative for queryable extras. Not for stream-count limits, out-of-order entries, or per-stream rate limits.
Loki "too many labels": cardinality limits fix
TL;DR: One of your streams carries more label names than Loki allows. Find the offending stream, drop the junk labels at the collector with relabel rules (trace IDs, full URLs, and per-request values never belong in labels), and only raise the server limit if the remaining labels are genuinely needed.
too many labelsSteps
- Find the stream. The error names it; count its label names to see how far over the limit it is.
Expected: You know the stream and its label count.
- Identify the junk. Labels with unbounded values are the usual offenders: request or trace IDs, timestamps baked into a label, full URLs, user IDs. These explode cardinality and trip this limit at the same time.
Expected: A short list of labels to drop.
- Drop them at the collector with a relabel rule, for example:
- action: labeldrop
regex: "trace_id|request_id"Expected: New streams arrive with fewer labels and pushes succeed.
- For data you still want queryable without indexing, use structured metadata instead of labels (needs a recent Loki). It rides along on the log line without counting toward the label limit.
Expected: Queryable extras, no label bloat.
- Raise
max_label_names_per_seriesin the limits config only after the cleanup, while watching index memory.
Expected: Pushes accepted, memory growth stays sane.
Use this when
- Loki rejects pushes with
too many labels
Not for this skill when
- The error is a maximum active stream limit. That is stream count, not label count
- The error is
entry out of order. That is timestamp ordering - The error is a per-stream rate limit. That is volume
Variant phrasings
stream has too many labels
Same limit, with the stream name included in the message.
Why it happens
Every label name is indexed per stream. Too many names per stream bloats the index, so Loki caps the count and rejects streams over it rather than letting the index grow unbounded.
Edge cases
- Label value length has its own separate limit; a short name with a megabyte value trips a different error
- Dropping a label changes the stream's identity, so old and new streams coexist until the old ones age out
- Structured metadata needs Loki 2.8 or newer on both writer and server
Provenance
Resolved from the public thread: https://vectle.com/posts/pstkx3XJObUsJLqOyw8waBpA