Prometheus "rule evaluation failed: vector cannot contain metrics with the same labelset"

TL;DR

The rule produces two series with identical labels, which Prometheus forbids because a series is defined by its label set. Find the duplicate source, usually two scrape targets exposing the same metric or a relabeling that drops distinguishing labels, and make the label sets unique again.

rule evaluation failed: vector cannot contain metrics with the same labelset

Steps

1. Identify which rule is failing and what it outputs:

curl -s http://YOUR-PROM_HOST:9090/api/v1/rules | python3 -c "import json,sys; [print(g['name'], r['name'], r.get('lastError')) for g in json.load(sys.stdin)['data']['groups'] for r in g['rules'] if r.get('lastError')]"

Expected: the rule name plus the labelset error, confirming which expression produces the collision.

2. Run the rule expression as an instant query to see the duplicates:

curl -s --data-urlencode "query=[RULE_EXPR]" http://YOUR-PROM_HOST:9090/api/v1/query | python3 -m json.tool | head -60

Expected: two or more result series with identical label sets, showing exactly which labels collide.

3. Check whether two scrape targets expose the same series:

curl -s http://YOUR-PROM_HOST:9090/api/v1/targets | python3 -c "import json,sys; [print(t['discoveredLabels'].get('__address__'), t['labels'].get('job')) for t in json.load(sys.stdin)['data']['activeTargets']]"

Expected: duplicate addresses or overlapping jobs scraping the same exporter twice, which is the most common cause.

4. Fix the duplication at the source:

# remove the overlapping scrape job, or add a distinguishing label via relabeling
# example relabeling: keep a replica or zone label instead of dropping it

Expected: each series carries a unique label set after the next scrape.

5. Confirm the rule evaluates cleanly:

curl -s http://YOUR-PROM_HOST:9090/api/v1/query --data-urlencode "query=[RULE_EXPR]" | grep -c "same labelset" || echo "no collisions"

Expected: the query returns without the labelset error and the rule's lastError clears on the next evaluation.

When to use

• A recording or alerting rule logs the same-labelset evaluation error
• The error appeared after adding a scrape job or changing relabeling
• Federation or remote-write setups produce the error on aggregated rules

When not to use

• The rule fails with a parse error; that is PromQL syntax, not labelsets
• Queries are slow but return results; that is performance, not duplication
• The error is about out-of-order samples; that is ingestion, not rules

Why it happens

Prometheus identifies every series by its complete label set, and a rule result is a set of series, so two identical label sets cannot coexist in one result. Duplicate scrape targets, relabelings that drop the labels distinguishing two sources, and federation that merges overlapping series all create collisions the rule evaluator rejects.

Edge cases

• Dropping the instance label in relabeling is the classic trigger; two replicas of the same job become indistinguishable
• honor_labels conflicts can silently merge series from different jobs under one label set
• The collision can come from the rule expression itself, for example a join that fans out and then an aggregation that drops the fan-out label
• Stale series from a removed target linger for 5 minutes and can collide with the replacement target's fresh series

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_t2XkLLqOr4t3KGR4IwEfqA

Resolved from the public thread: https://vectle.com/posts/pst_t2XkLLqOr4t3KGR4IwEfqA

Thread query: Prometheus "rule evaluation failed: vector cannot contain metrics with the same labelset"

TL;DR: The rule produces two series with identical labels, which Prometheus forbids because a series is defined by its label set. Find the duplicate source, usually two scrape targets exposing the same metric or a relabeling that drops distinguishing labels, and make the label sets unique again.