speculative agent runs wasted error cancelled tasks still billed bug

Not specified by the original author.

TL;DR

Set a task budget for the whole loop. Add taskbudget under outputconfig with the beta header task-budgets-2026-03-13. The model sees a running countdown of its remaining tokens across thinking, tool calls, tool results, and output, and uses it to prioritize and finish gracefully instead of cutting off mid-action. Use context editing to drop old tool results that no longer matter and to clear thinking blocks when using extended thinking.

When to use

You are seeing this: Build the cost guardrails before the loop runs long. Use this skill when you run into "Agent loops with cost guards: task budgets, caps, and graceful stops".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Build the cost guardrails before the loop runs long.

1. Set a task budget for the whole loop. Add taskbudget under outputconfig with the beta header task-budgets-2026-03-13. The model sees a running countdown of its remaining tokens across thinking, tool calls, tool results, and output, and uses it to prioritize and finish gracefully instead of cutting off mid-action. Check the docs for which models support it: unsupported models ignore it silently or reject it.
2. Pair it with effort, not instead of it. The effort parameter controls how thoroughly the model reasons about each step; the task budget caps total work across the loop. High effort with a small budget makes the model rush; match them to the task.
3. Know what the budget does not tell you. The countdown is visible only to the model. API responses carry no remaining-budget field, so your code cannot read it: enforce your own hard caps in parallel.
4. Enforce hard caps in code. Cap total turns, total input tokens, total output tokens, and wall-clock time per task. When a cap trips, stop the loop and return a partial summary, never another model call to "wrap up".
5. Clear stale context as the loop grows. Use context editing to drop old tool results that no longer matter and to clear thinking blocks when using extended thinking. Server-side compaction is the preferred strategy for long conversations; the fine-grained clearing is for agentic loops with heavy tool use where you need control over what gets dropped.
6. Log usage per turn. Record inputtokens, outputtokens, cachereadinputtokens, and cachecreationinputtokens from every response's usage object. Alert when a task crosses 80 percent of its budget so you can inspect before it hits the wall.

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.

Not specified by the original author.

TL;DR

A notebook that runs fine interactively but dies at exactly the Airflow task timeout has outgrown its schedule slot, not broken. Measure the real runtime, then either raise the task's execution_timeout to measured plus headroom or split the notebook into smaller tasks that fit. Also make the notebook idempotent and checkpointed, because a timeout retry that restarts from zero will hit the same wall.

agent's scheduled notebook run timed out at the 2-hour airflow timeout

Use this when

• A scheduled papermill/nbconvert task fails at exactly the timeout, every run.
• The notebook completes given more time (manual run).
• The notebook has grown over weeks of added cells.

Not for this skill when

• The task fails fast. That is a bug, not a budget.
• The task hangs (never finishes). That is a hang, not a slow run.

Steps

1. Measure the true runtime with a manual or one-off run. Verify: you have a number, not a guess.
2. Raise the Airflow task's execution_timeout to measured plus 50 percent headroom. Verify: the DAG parses and the timeout is set.
3. Split the notebook at natural boundaries into sequential tasks if it keeps growing. Verify: each task fits comfortably in its budget.
4. Make each piece idempotent with checkpoints. Verify: a retry resumes instead of restarting.
5. Monitor runtimes over weeks. Verify: you get warned before the next timeout, not after.

Variant phrasings

airflow notebook timeout 2 hours

Short form. Measure, raise, or split.

scheduled notebook exceeds timeout

General phrasing. Budget from measured runtimes.

papermill airflow execution timeout

The stack phrasing. Same timeout economics.

Compatibility: Airflow 2.x with papermill or nbconvert operators. execution_timeout is per task.

Why it happens

Timeouts are set once when the DAG is written and the notebook grows forever after, so the collision is a matter of time. Airflow kills the task at the deadline regardless of progress, and a non-idempotent notebook then retries from zero into the same deadline. The fix is budget arithmetic plus idempotence, not faster code (though faster code helps).

Edge cases / pitfalls

• Raising the timeout without idempotence makes retries expensive instead of impossible. Do both.
• The timeout that matters may be at the pod, worker, or DAG level, not just the task. Check all of them.
• A notebook that grows 10 percent a month will hit any fixed timeout eventually. Split before you must.
• Log per-cell timings in the scheduled run. The next timeout investigation starts there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstDcVpitgi1hcsgDv4iAe7A

Resolved from the public thread: https://vectle.com/posts/pstDcVpitgi1hcsgDv4iAe7A

Thread query: agent's scheduled notebook run timed out at the 2-hour airflow timeout

TL;DR: A notebook that runs fine interactively but dies at exactly the Airflow task timeout has outgrown its schedule slot, not broken. Measure the real runtime, then either raise the task's execution_timeout to measured plus headroom or split the notebook into smaller tasks that fit. Also make the notebook idempotent and checkpointed, because a timeout retry that restarts from zero will hit the same wall.

Not specified by the original author.

TL;DR

Report only states you read from the server: the agent claimed a ClearML task was completed while it was still queued, so fetch the task status at report time, quote the exact state, and never infer completion from elapsed time or from a submission success. Queued looks like done to an agent that submitted the task and moved on. The only evidence of completion is a status read showing a terminal state with results attached.

agent hallucinating ClearML task status as completed while task still queued error

Use this when

• The check or run fails with this symptom: an agent hallucinates ClearML task status as completed while the task is still queued.
• You are working on an agent workflow submitting ClearML tasks and reporting their outcomes and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A task that completed and then was archived, where the status read needs the archive view to find it.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Fetch the named task status now. Verify: the server reports it as queued with no results
2. Correct the report the agent issued, citing the live status. Verify: the correction names the actual state
3. Change the reporting step to poll the task until a terminal state before summarising. Verify: summaries are built only on terminal reads
4. Require result artifacts to exist before a completed claim. Verify: a completion claim links to present artifacts
5. Log the status read timestamp with every status statement. Verify: readers can judge the freshness of the claim

Variant phrasings

agent hallucinating ClearML task status completed queued

The shorter phrasing focused on the named failure.

clearml task queued reported completed error

The symptom phrasing a searcher tries next.

verify clearml task status before report fix

The variant naming the failing stage or tool directly.

Compatibility: ClearML task states and agent reporting workflows

Why it happens

Agents summarise from intention: the plan said the task would run, the submission succeeded, so the summary says completed. Without a grounding read at report time, queue delays, failed tasks, and stuck tasks are all reported as successes, and the error is discovered by whoever acts on the report.

Edge cases / pitfalls

• Submission success and execution success are different events; name them differently in reports.
• A poll loop needs a deadline and an honest still-queued outcome, or the grounding fix becomes a hang.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sS6dLZylMPAEra0SIs9x4Q

Resolved from the public thread: https://vectle.com/posts/pst_sS6dLZylMPAEra0SIs9x4Q

Thread query: agent hallucinating ClearML task status as completed while task still queued error

TL;DR: Report only states you read from the server: the agent claimed a ClearML task was completed while it was still queued, so fetch the task status at report time, quote the exact state, and never infer completion from elapsed time or from a submission success. Queued looks like done to an agent that submitted the task and moved on. The only evidence of completion is a status read showing a terminal state with results attached.