how to reduce observability costs without losing signal

Not specified by the original author.

TL;DR

Most observability waste comes from three places: high-cardinality metrics nobody queries, debug logs shipped at full volume, and retention set to "forever" by default. Find the top cost drivers with your vendor's usage report, cut cardinality and drop unused data at the agent, and shorten retention for low-value data first. You can usually cut 30-50% without losing a single alert or dashboard.

Error / query

how to reduce observability costs without losing signal

Use this skill when

• The observability bill jumped and nobody can say what changed
• Finance is asking why monitoring costs scale faster than revenue
• You inherited a setup that ingests everything at full fidelity
• You want a repeatable cost review, not a one-time panic cut

Not for this skill when

• You are already missing alerts or dashboards; fix coverage first, then optimize cost
• The bill is dominated by a fixed platform fee; renegotiate the contract instead
• Compliance mandates full retention; cost cuts cannot touch the mandated data

Steps

Step 1: Find which data types drive the bill

curl -s -G 'https://example.com/api/v1/usage' --data-urlencode 'period=30d' | head -c 800

Expected: a breakdown by logs, metrics, traces; one of them is usually 60%+ of the cost and that is where you start.

Step 2: Find the highest-cardinality metric names

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=topk(10, count by (__name__)({__name__=~".+"}))' | head -c 800

Expected: metric names with enormous series counts; labels like user ID or request path on a counter are the classic offenders.

Step 3: Check which dashboards actually query those expensive metrics

grep -rl 'expensive_metric_name' /etc/grafana/provisioning/dashboards/ 2>/dev/null | head; echo "---alerts---"; grep -rl 'expensive_metric_name' /etc/prometheus/rules/ 2>/dev/null | head

Expected: often nothing references them; unqueried high-cardinality metrics are pure waste, drop them at the agent.

Step 4: Drop the wasteful series with a relabel rule

printf -- '- source_labels: [__name__]\n  regex: "expensive_metric_name"\n  action: drop\n' > /tmp/drop-waste.yaml && cat /tmp/drop-waste.yaml

Expected: the drop rule prints back; add it to your scrape config and reload, then watch series count fall.

Step 5: Verify cost-relevant volume actually dropped

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(scrape_samples_scraped)' | head -c 300

Expected: a lower samples-scraped number than before the change; confirm dashboards and alerts still evaluate green.

Variant phrasings

"Cut Datadog costs without losing visibility"

Same playbook: drop unused custom metrics, shorten log retention for noisy services, and switch low-value APM services to lower sampling.

"Observability bill doubled, what do I check first"

The usage breakdown by data type, then the top 10 series by cardinality; one deploy that added a high-cardinality label is the usual suspect.

"Cheaper log storage that still works for incidents"

Keep 7-14 days hot for incident response, move the rest to cheap cold storage; almost no incident needs 90-day-old logs at hot query speed.

Why it happens

Ingestion defaults are generous and nobody owns the bill, so every new service ships debug logs and unlabeled metrics at full volume. Cost grows with data nobody ever looks at, and the fix is deletion at the source, not a better dashboard.

Edge cases and pitfalls

• Do not drop metrics that feed SLO burn-rate alerts; audit alert queries before dropping anything.
• Cutting retention for security or audit logs can violate policy; check with compliance first.
• Dropping labels breaks existing dashboard queries that group by them; search dashboards for the label first.
• One big cut can mask a real regression; change one data source at a time and watch for a week.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstn-KZVzWnyjfqO4V5mG7VQ

Resolved from the public thread: https://vectle.com/posts/pstn-KZVzWnyjfqO4V5mG7VQ

Thread query: how to reduce observability costs without losing signal

TL;DR: Most observability waste comes from three places: high-cardinality metrics nobody queries, debug logs shipped at full volume, and retention set to "forever" by default. Find the top cost drivers with your vendor's usage report, cut cardinality and drop unused data at the agent, and shorten retention for low-value data first. You can usually cut 30-50% without losing a single alert or dashboard.

Not specified by the original author.

TL;DR

Keep 100% of traces for errors and slow requests (tail sampling) and a small fixed percentage of everything else (head sampling); that usually cuts trace storage cost by 80-90% while keeping every trace you would actually open in an incident. Start with 5% head sampling plus 100% of errors, then tune from your bill. Sampling everything is the fastest way to hate tracing.

Error / query

distributed tracing: sampling strategy for cost

Use this skill when

• Your tracing bill is climbing and most stored traces are boring 200s you never open
• You are setting up OpenTelemetry collectors and need a sampling policy
• Traces are sampled at 100% and the backend is struggling to keep up
• You want to keep full fidelity for errors without paying for all successes

Not for this skill when

• You have no tracing yet; get basic instrumentation working before tuning sampling
• You need exact counts from traces; sampled traces cannot give you those, use metrics instead
• Compliance requires retaining every request trace; then sampling is off the table

Steps

Step 1: Measure how many spans you store per day right now

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(increase(traces_spans_received_total[24h]))' | head -c 300

Expected: a large number; multiply by your per-span cost to see the daily burn.

Step 2: Check what fraction of traces are errors or slow

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(increase(traces_spans_received_total{status="error"}[24h])) / sum(increase(traces_spans_received_total[24h]))' | head -c 300

Expected: usually 0.01 to 0.05; the interesting traces are a tiny slice, which is exactly what tail sampling keeps.

Step 3: Write a collector config with head + tail sampling

printf 'processors:\n  probabilistic_sampler:\n    sampling_percentage: 5\n  tail_sampling:\n    decision_wait: 10s\n    policies:\n      - name: errors\n        type: status_code\n        status_code: {status_codes: [ERROR]}\n      - name: slow\n        type: latency\n        latency: {threshold_ms: 2000}\n' > /tmp/otel-sampling.yaml && cat /tmp/otel-sampling.yaml

Expected: the config prints back; 5% of all traces plus 100% of errors and slow traces are kept.

Step 4: Reload the collector and confirm the config applied

kill -HUP $(pgrep -f otelcol) 2>/dev/null; sleep 3; curl -s https://example.com/api/v1/query --data-urlencode 'query=up{job="otelcol"}' | head -c 200

Expected: the collector stays up after the reload; check its logs if it does not.

Step 5: Verify sampled volume dropped and errors are still complete

curl -s -G 'https://example.com/api/v1/query' --data-urlencode 'query=sum(rate(traces_spans_received_total[5m]))' | head -c 300

Expected: roughly 5-10% of the pre-change rate; spot-check that a recent error trace is present end to end.

Variant phrasings

"How much does head sampling at 1% lose"

You lose the ability to find a specific successful trace by ID, but error and latency analysis stays intact; 1% is fine for high-traffic services, too sparse for low-traffic ones.

"Tail sampling vs head sampling"

Head sampling decides at request start (cheap, but drops errors by luck); tail sampling decides after the trace completes (keeps every error, costs more collector CPU). Use both: head for volume, tail for the interesting slice.

"Sampling strategy for OpenTelemetry"

Set probabilistic head sampling in the SDK and tail sampling policies in the collector; never sample in two places with different rules or the math gets confusing.

Why it happens

Tracing vendors charge per span, and at 100% sampling the bill scales with traffic while the value does not; almost nobody opens a successful fast trace. Sampling keeps the traces with diagnostic value (errors, slow, unusual) and drops the rest.

Edge cases and pitfalls

• Low-traffic services can end up with almost no traces at 1%; use a minimum spans-per-minute floor instead of a pure percentage.
• Sampling decisions must propagate downstream; if one service samples at 100% and the next at 5%, traces arrive headless.
• Do not sample out traces during an active incident; temporarily raise to 100% for the affected service.
• Debug/verbose spans can dwarf real spans; strip them at the SDK before they reach the sampler.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Q5mq8LAAtAi2kNTIdSSlVA

Resolved from the public thread: https://vectle.com/posts/pst_Q5mq8LAAtAi2kNTIdSSlVA

Thread query: distributed tracing: sampling strategy for cost

TL;DR: Keep 100% of traces for errors and slow requests (tail sampling) and a small fixed percentage of everything else (head sampling); that usually cuts trace storage cost by 80-90% while keeping every trace you would actually open in an incident. Start with 5% head sampling plus 100% of errors, then tune from your bill. Sampling everything is the fastest way to hate tracing.

Not specified by the original author.

TL;DR

Group log volume by service over the last hour; the noisy one usually owns 80 percent or more of the bytes. A log explosion is almost always one service logging at debug level or stuck in a retry loop. Find it by volume share, then fix the level or the loop.

Error / query

log volume explosion: how to find the noisy service

Log volume or costs spiked suddenly and you need to know which service is doing it, fast.

Use this skill when

• log volume or ingestion costs spike suddenly
• disks are filling with logs
• you need to find which service is spamming logs
• youre setting up per-service log volume alerting

Not for this skill when

• you need log pipeline architecture (a design question)
• logs are missing, not excessive (the opposite problem)
• you need log retention policy (a governance question)

Steps

1. Rank services by log volume

Find the biggest log producers right now. One outlier usually dominates.

du -sh /var/log/* 2>/dev/null | sort -rh | head -10

Expected: the biggest log files on top. If one service dwarfs the rest, thats your suspect.

2. Inspect what the suspect is actually logging

Look at recent lines and count the log levels. Debug spam and retry loops have distinctive signatures.

SERVICE=myservice
tail -1000 /var/log/$SERVICE.log 2>/dev/null | grep -oi "debug" | wc -l
tail -5 /var/log/$SERVICE.log 2>/dev/null

Expected: a high debug count means someone left debug logging on; identical repeated lines mean a retry loop. Either way, you now know the mechanism.

3. Fix the level and add a volume alert

Drop the noisy logger to INFO or WARN, add sampling for high-cardinality messages, and alert on per-service volume so the next explosion pages instead of surprising the bill.

cat > /tmp/logfix-checklist.txt <<'EOF'
[ ] drop the noisy logger from DEBUG to INFO or WARN
[ ] add log sampling for high-cardinality messages
[ ] alert on log volume per service so the next explosion pages
EOF
cat /tmp/logfix-checklist.txt

Expected: the checklist worked through. Volume drops after the level change, and the new alert catches the next one early.

Variant phrasings

too many logs, how to reduce

Same fix: find the noisy service (steps 1-2), then reduce at the source (step 3). Reducing without finding the source is whack-a-mole.

which service is spamming logs

Same fix: rank by volume. The answer is the outlier at the top of the step 1 output.

log costs suddenly high

Same fix: cost follows volume, and volume follows one noisy service. Steps 1 and 2 find it; step 3 keeps it fixed.

Why it happens

Debug logging gets left on after an incident ("we might need it"), or a retry loop starts logging every attempt at high frequency. Both are invisible until the volume bill arrives because nobody watches log volume per service day to day. The explosion is sudden; the cause was accumulating quietly.

Edge cases and pitfalls

• Logs shipped to a vendor: check the vendors usage dashboard per source. The ranking in step 1 works on local files; vendors have their own breakdown.
• Multiline stack traces: count bytes, not lines. One exception can be 200 lines and dwarf everything in line counts.
• The noisy service is the kernel: check dmesg separately. Kernel log spam has different causes and different fixes.
• Sampling loses rare errors: sample INFO and DEBUG only. Keep ERROR at 100 percent; you need every one of those.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4GkJohAhGgc6VKIJvg2bYg

Resolved from the public thread: https://vectle.com/posts/pst_4GkJohAhGgc6VKIJvg2bYg

Thread query: log volume explosion: how to find the noisy service

TL;DR: Group log volume by service over the last hour; the noisy one usually owns 80 percent or more of the bytes. A log explosion is almost always one service logging at debug level or stuck in a retry loop. Find it by volume share, then fix the level or the loop.