## TL;DR
The agent's triage burned through the Datadog API budget mid-incident and got throttled, stalling the investigation when it mattered most. Budget the queries: fetch the key dashboards and monitors once, cache aggressively during the incident, and back off on 429. Triage on a budget beats triage that dies at the interesting moment.

```text
datadog API rate limit hit mid-pipeline-incident triage, triage failed
```

## Use this when
- Agent Datadog queries hit 429s during incident triage.
- The triage worked on small incidents but fails on big ones.
- The agent lost its observability mid-investigation.

## Not for this skill when
- Datadog returns auth errors. Fix the API key.
- The metrics are missing. That is instrumentation, not rate limiting.

## Steps
1. List the queries the triage makes and rank them by value. Note the ranked list. Verify: the query budget is explicit
2. Fetch the top queries once and cache them for the incident duration. Note the caching. Verify: repeat queries stop costing
3. Add backoff with jitter on 429. Note the pacing. Verify: throttles are absorbed
4. If throttled anyway, continue triage from the cached signals and note their age. Note the degraded mode. Verify: the investigation continues openly
5. After the incident, trim the triage's query list permanently. Note the leaner triage. Verify: the budget fits next time

## Variant phrasings
### datadog api 429 triage
The code-shaped search.
### datadog query budget incident
The technique phrasing.

Compatibility: Datadog API rate limits, per endpoint and per org. Caching strategy is client-side.

## Why it happens
Incident triage is query-hungry: the agent pulls metrics, logs, and monitor states in a burst, exactly when every other integration is also bursting. Datadog's per-org limits do not care that it is an incident. A triage that dies mid-investigation is worse than a slower triage, because the agent loses context at the moment of maximum confusion. Budgeting queries upfront and caching during the incident keeps the investigation alive on a fixed API diet.

## Edge cases / pitfalls
- Different Datadog endpoints have different limits. The triage budget must cover the hungriest endpoint, usually logs.
- Cached signals go stale during long incidents. Refresh the cache on a schedule, not just on miss.
- A triage that shares the org API budget with CI and dashboards can be throttled by someone else's burst. Account for shared usage.
- Note the cache age in every triage conclusion. Decisions on hour-old data need that caveat.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uV_s8yGjsMKdyUHKu22Mhw