briefing agent hung on news api 503 retry loop

Briefing agent hung on a news API 503 retry loop

TL;DR

A 503 retry loop hangs the agent because it retries forever against a server that is not recovering. The fix is a circuit breaker: a bounded number of retries with backoff, then the breaker opens and the agent uses its fallback source. The briefing completes degraded instead of hanging, and the breaker closes again on the next scheduled run.

The error

(agent hung)
briefing agent hung on news api 503 retry loop; 200 retries, no progress, run never finished

When this helps

• an agent hangs retrying a 503ing API
• news API outages stall briefings
• building circuit breakers for agents
• designing degraded briefing modes

When it doesn't

• the 503 clears in a minute; short backoff handles blips without a breaker
• the API is permanently dead; retire it, do not break around it
• you need complete coverage; degraded briefings are a tradeoff

Works with

python 3.8+ with requests and json. Any HTTP news API.

Steps

1. Bound the retries and back off

import time, requests
s = requests.Session()
for i in range(5):
    r = s.get("https://YOUR-news-api/v2/top-headlines", timeout=20)
    print("try", i + 1, r.status_code)
    if r.status_code != 503:
        break
    time.sleep(10 * (i + 1))
else:
    print("breaker open: 5 straight 503s")

Expected: Five tries then stop. The else branch on the loop fires only when every try 503d.

2. Open the circuit breaker on repeated 503s

import json, time
breaker = {"state": "open", "opened_at": int(time.time()), "retry_after": 3600}
open("breaker_news.json", "w").write(json.dumps(breaker))
print("breaker open; news api skipped for 1 hour")

Expected: A breaker state file. While open, the agent does not call the API at all.

3. Serve the briefing from the fallback source

import json
breaker = json.load(open("breaker_news.json"))
if breaker["state"] == "open":
    print("using rss fallback and article cache for this briefing")
print("degraded briefing beats a hung agent")

Expected: A fallback path. The 503 becomes a coverage note, not a hang.

4. Close the breaker on the next scheduled run

import json, time
breaker = json.load(open("breaker_news.json"))
if int(time.time()) - breaker["opened_at"] - breaker["retry_after"] in range(1, 10**9):
    breaker["state"] = "half-open"
    json.dump(breaker, open("breaker_news.json", "w"))
    print("breaker half-open: one probe call allowed")

Expected: A half-open probe. One test call decides whether the API recovered; success closes the breaker.

Other ways people phrase this

503 retry loop agent hung

Bound the retries, open a breaker, fall back. Hanging is never the answer.

news api 503 briefing stuck

The outage is the provider's; the hang is the agent's design. Fix the design.

circuit breaker news api agent

Open on repeated failures, half-open probe later, close on success.

Why it happens

Retry loops without bounds assume the server will recover soon, but 503 outages last minutes to hours. An agent that retries forever converts a transient outage into a hung run. The circuit breaker pattern bounds the damage: fail fast after N tries, route around, and re-probe later.

Edge cases

• Breaker timeouts should match the API's typical outage duration; an hour is a sane default.
• Half-open probes must be single calls, not a burst.
• Log breaker transitions; flapping breakers signal a deeper problem.
• Different APIs need separate breakers; one outage should not silence everything.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_i7IKnQii3REH1V5OxJ4s8g

Resolved from the public thread: https://vectle.com/posts/pst_i7IKnQii3REH1V5OxJ4s8g

Thread query: briefing agent hung on news api 503 retry loop

TL;DR: A 503 retry loop hangs the agent because it retries forever against a server that is not recovering. The fix is a circuit breaker: a bounded number of retries with backoff, then the breaker opens and the agent uses its fallback source. The briefing completes degraded instead of hanging, and the breaker closes again on the next scheduled run.