agent's braintrust eval run failed when the api key expired mid-run: how to recover

TL;DR

Rotate the expired key first, then resume from the partial run instead of starting over. Point the eval runner at the previous run id so already-scored rows are kept and only missing rows run. This saves the cost and time of the rows that already finished, and keeps scores comparable because every row is scored once.

Error

braintrust eval failed: 401 Unauthorized
run btrun_8f2k: 47 of 200 rows scored, then the API key expired mid-run

Steps

1. Confirm the failure is auth, not a bug: check the run log for 401 or Unauthorized on the first failed request after the last scored row. Expected: the error appears at a clean boundary and the scored rows before it are intact.
2. Rotate the key in the provider dashboard and store the new key in your secrets manager (never paste it into chat, logs, or the run command). Expected: the secrets manager shows the new key as active and the old one revoked.
3. Rerun with resume enabled against the original run id, e.g. braintrust eval my_eval --resume-from btrun_8f2k (check the runner docs for the exact flag). Expected: the runner reports it is skipping the 47 already-scored rows and starts at row 48.
4. After completion, verify the final counts: scored rows equals total rows, with exactly one score per row and no duplicates. Expected: the run summary shows 200 of 200 rows scored, 47 of them carrying the original timestamps.
5. If the runner has no resume option, export the partial results, remove the completed row ids from the input, run the remainder as a new run, then merge. Expected: the merged result has exactly one row per input, no row scored twice.

When to use

• The eval died with 401/Unauthorized partway through and earlier rows show scores.
• The key was rotated, revoked, or hit its expiry date in the middle of a long run.
• You want to avoid paying for, or waiting on, already-scored rows a second time.

When not to use

• The failure is in the scoring code or the input rows themselves (a resume will fail again the same way).
• The run is small and cheap; a fresh full rerun is simpler and cleaner.
• Rows are non-deterministic and you need every row scored under identical conditions (rerun everything instead).

Tool compatibility

• Braintrust evals (CLI and Python SDK), with any model provider behind them.
• The pattern applies to any eval harness that records per-row results under a run id (LangSmith, Arize, custom harnesses).

Variant phrasings

API key expired mid-run

Same recovery: rotate, resume, verify counts. The key point is never re-score finished rows.

Braintrust eval 401 unauthorized

401 means the key is dead, not that the eval is broken. Fix auth first, then resume the run.

Resume braintrust eval

Use the runner's resume option against the original run id rather than launching a fresh eval.

Why it happens

Eval runs are long-lived: hundreds of rows, each a model call, stretching past a key's expiry or a rotation someone did mid-afternoon. The first request after expiry fails, and a naive rerun starts from row zero, doubling cost and producing two runs that are painful to merge.

Edge cases

• If the old key was revoked rather than expired, rows scored just before the failure are still valid; an auth failure does not corrupt completed scores.
• Rotating the key in CI means updating the secrets-manager entry the pipeline reads, not the workflow file.
• If partial rows show as started but unscored, treat them as missing and rerun them; one duplicate check beats a gap you never notice.
• Watch for score drift: rows scored before and after a long pause can differ if the model or prompt changed in between, so note the pause in the run description.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_M7owVOiUpwZldditqs7plA