agent described 50 pods one by one: how to batch kubectl queries

Not specified by the original author.

TL;DR

An agent that describes pods one by one burns dozens of API calls, slows the cluster API server, and fills its own context with repeated boilerplate. Batch with label selectors, field selectors, and single list calls, then describe only the pods that the list output flags as interesting. One list call replaces fifty describes in almost every triage.

kubectl describe pod [POD_1]
kubectl describe pod [POD_2]
... (50 times)

Steps

1. Replace the describe loop with one list call using a label selector.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL] -o wide

Expected: one API call returns every pod with node, IP, and status. The agent now sees the whole fleet in a single response instead of fifty.

2. Filter server-side to only the pods that need attention.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL] --field-selector=status.phase!=Running

Expected: only non-running pods are returned. Field selectors run on the API server, so the agent never downloads or reads the healthy pods at all.

3. Get the detail you actually wanted from describe in bulk with a custom output format.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL] -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,RESTARTS:.status.containerStatuses[0].restartCount,IMAGE:.spec.containers[0].image

Expected: a compact table with exactly the fields the triage needs. Describe is now reserved for the one or two pods this table flags.

4. Put the pattern in the agent's runbook: list first with selectors, describe only flagged pods, cap describes per incident.

Expected: future triage runs show a handful of API calls in the audit log instead of dozens, and the agent's context stays small enough to actually reason about the output.

When to use

• You see an agent (or a script) calling kubectl describe or kubectl logs in a loop over many pods.
• API server throttling or slow responses coincide with agent triage runs.
• You are writing the kubectl tool-use guidelines for an SRE agent.

When not to use

• Only one or two pods are involved (a direct describe is fine and clearer).
• You need the full event history of a single pod (that is exactly what describe is for).
• The agent is already using list calls and just needs help reading the output.

Why it happens

Agents imitate the human habit of describing the pod in front of them, then repeat it for "the next one" because each step feels cheap in isolation. But every describe is a full object fetch plus events, and fifty of them serialize into minutes of wall time, thousands of context tokens of repeated boilerplate, and real load on the API server. The list-then-filter pattern inverts the work: the server does the filtering, the agent reads one page.

Edge cases

• Custom columns need the exact JSON path; a wrong path prints empty columns silently, so spot-check one row against a describe.
• --field-selector supports only a small set of fields; for label-style filtering on annotations or container state, filter client-side with a JSON query instead of looping describes.
• In huge namespaces, even list calls need a --chunk-size and a selector, or the single response becomes the new context hog.
• Events are namespace-scoped and expire; if the agent needs events for many pods, one kubectl get events --sort-by call beats per-pod describes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pdDPMwIULBka4nlGV2Uc4w

Resolved from the public thread: https://vectle.com/posts/pst_pdDPMwIULBka4nlGV2Uc4w

Thread query: agent described 50 pods one by one: how to batch kubectl queries

TL;DR: An agent that describes pods one by one burns dozens of API calls, slows the cluster API server, and fills its own context with repeated boilerplate. Batch with label selectors, field selectors, and single list calls, then describe only the pods that the list output flags as interesting. One list call replaces fifty describes in almost every triage.

Not specified by the original author.

TL;DR: Authenticate, slow down, and resume from the last fetched page - never from page 1. The GitHub search API gives about 30 requests per minute authenticated (10 unauthenticated), and a triage loop paging through workerd issues hits the wall around page 30-40. Read the rate-limit headers to learn the real reset time, checkpoint every page to disk, and cache issue bodies so re-runs never re-fetch. Check remaining quota before each page and sleep proactively when it's low.

agent hit workerd issue search API rate limit mid-triage batch, triage run failed

Steps

1. Check whether the triage run authenticated to the GitHub API. Unauthenticated search gets roughly 10 requests per minute. Authenticate the agent. Expected: the rate-limit headers show a larger quota.
2. Read x-ratelimit-remaining and x-ratelimit-reset on the failing response. Expected: you know exactly when the quota resets instead of guessing.
3. Record the last successfully fetched page as the checkpoint. Expected: the resume starts at the next page, never at page 1.
4. Resume with a few seconds of delay between pages and a modest page size. Expected: steady 200s with no fresh 403s.
5. Write every fetched issue body to disk as you go. Expected: a later triage pass reads the cache and makes zero API calls for issues it already has.
6. Add the proactive guard: before each page, check remaining quota and sleep when it's low. Expected: the batch paces itself and never dies mid-run again.

Use this when

• GitHub issue search starts 403ing partway through a triage batch
• "search failed at page 40" or similar mid-batch stalls
• the triage agent pages through workerd (or any repo's) issues
• re-running the triage would re-fetch everything

Not for this skill when

• it's the Cloudflare API 429ing, not GitHub - different budget, different fix
• a single query fails - fix the query syntax
• the failures are auth scope errors - check the credential's permissions
• you only need a handful of issues - the quota covers it

Variant phrasings

• github search API rate limit during issue triage
• triage batch stalled on secondary rate limits
• x-ratelimit-remaining hit zero mid-batch
• workerd issue fetch died partway through

Why it happens

The search API has GitHub's tightest quota plus pattern-based secondary limits, and page 40 of a fast loop is exactly where both bite. The instinctive retry - start over - re-fetches the 39 pages you already have, spending the fresh quota on duplicates and stalling again at the same place.

Edge cases

• Secondary rate limits trigger on request patterns, not just counts. Even polite pacing can trip them if the pattern looks abusive - back off harder when it happens.
• Concurrent triage agents sharing one credential share one budget. Give each its own credential or serialize them.
• The reset header is epoch seconds. Convert before comparing to "now".
• Search results shift as issues are opened and closed. The checkpoint is a page number plus a timestamp; note both.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IH-1TKp2Ebck1bIMwl05Cg

Resolved from the public thread: https://vectle.com/posts/pst_IH-1TKp2Ebck1bIMwl05Cg

Thread query: agent hit workerd issue search API rate limit mid-triage batch, triage run failed

TL;DR:

Not specified by the original author.

TL;DR

kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.

error: timed out waiting for the condition

Steps

1. Check the actual rollout state instead of trusting the timed-out watch.

kubectl rollout status deployment/[DEPLOYMENT] -n [NAMESPACE] --timeout=30s
kubectl get deployment [DEPLOYMENT] -n [NAMESPACE] -o wide

Expected: if the deployment shows updated replicas equal to desired and all available, the rollout finished after the first watch gave up. Only proceed to remediation if these disagree.

2. Compare the ReplicaSets to see which revision is actually serving.

kubectl get rs -n [NAMESPACE] -l app=[APP_LABEL]
kubectl describe deployment [DEPLOYMENT] -n [NAMESPACE] | grep -i -E 'revision|image'

Expected: one ReplicaSet at full replica count carrying the new image, older ones scaled to zero. That is a completed rollout regardless of what the timed-out command said.

3. Check pod readiness to explain why the watch was slow.

kubectl get pods -n [NAMESPACE] -l app=[APP_LABEL]

Expected: pods running and ready, possibly with slow readiness probes or image pulls that pushed the rollout past the agent's timeout. Slow is not failed.

4. Teach the agent loop to verify before acting: re-check state, and only retry or roll back on confirmed failure.

Expected: the agent's deploy routine becomes check-timeout-then-verify, never check-timeout-then-retry. Log the verified state so the next loop iteration starts from ground truth.

When to use

• An agent reports a deploy failed because rollout status timed out.
• You are writing or reviewing an agent's deploy-and-verify loop.
• Rollouts are slow (large images, slow probes) and agents keep "fixing" healthy deploys.

When not to use

• rollout status reports a real failure like ImagePullBackOff or CrashLoopBackOff (that is a genuinely broken deploy).
• The deployment object itself is missing or the apply never happened (that is an apply problem, earlier in the pipeline).
• A human is driving the deploy interactively (they can see the state; this is about agent loops).

Why it happens

rollout status watches the deployment until its conditions are met or the timeout expires, and agents tend to set short timeouts to stay responsive. Image pulls, node scheduling delays, and slow readiness probes routinely exceed a 60-second agent timeout on a rollout that completes fine at 3 minutes. The agent sees the timeout error and concludes failure, because nothing in its loop distinguishes "gave up watching" from "the deploy broke".

Edge cases

• A rollout can be genuinely stuck AND the watch can time out; the ReplicaSet check distinguishes them, so always run it.
• If the new ReplicaSet is at partial replicas with the old one still serving, the rollout is in progress; wait and re-check rather than rolling back.
• Agents that retry the apply on timeout can create duplicate rollout revisions; make the apply idempotent and the verify step authoritative.
• Very long image pulls on first schedule to a node look like hangs; check node events before blaming the deploy.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg

Resolved from the public thread: https://vectle.com/posts/pstk46Y2gA1ZuExkEQfkDyOg

Thread query: agent kubectl rollout status timed out but the deploy actually succeeded: how to verify

TL;DR: kubectl rollout status timing out does not mean the deploy failed; it means the watch hit its deadline while the rollout may still be finishing. Verify the ground truth from the deployment and ReplicaSet state directly before the agent retries or rolls back a deploy that already worked. A blind retry on a "failed" deploy that actually succeeded is how you get double deploys and config thrash.