# News agent timed out and the bulk filing pull failed mid briefing

## TL;DR
A bulk filing pull that times out mid briefing means the agent tried to download the whole batch inside the briefing's time budget. The fix is separating the pull from the briefing: the pull runs on its own schedule with its own budget, and the briefing reads from the local filing store. The briefing never waits on the network for bulk history.

## The error
```text
(briefing failed)
news agent timed out and bulk filing pull failed mid briefing; 200 filings requested, 60 fetched, timeout
```

## When this helps
- bulk filing pulls time out mid briefing
- briefings wait on large downloads
- separating pull from briefing schedules
- building local filing stores

## When it doesn't
- the pull is small; inline fetching is fine
- you need today's filings now; the pull schedule must cover it
- the store is stale; schedule pulls often enough

## Works with
python 3.8+ with json and os. EDGAR endpoints as of 2026.

## Steps
### 1. Move the bulk pull to its own scheduled job
```python
import json
job = {"name": "nightly-filing-pull", "schedule": "02:00 UTC", "budget_minutes": 120}
open("pull_job.json", "w").write(json.dumps(job, indent=2))
print("the pull runs overnight with its own budget")
```
Expected: A separate job. Bulk history never competes with the briefing for time.

### 2. Make the pull resumable with per-filing checkpoints
```python
import json
open("pull_state.json", "w").write(json.dumps({"done": 60, "total": 200}))
print("resume from 61; the timeout loses nothing")
```
Expected: A pull checkpoint. Slow nights resume instead of restarting.

### 3. Have the briefing read from the local store
```python
import json, os
store = "filings/"
print("local filings:", len(os.listdir(store)))
print("the briefing reads disk, not the network")
```
Expected: A local-first briefing. Network trouble cannot stall the briefing.

### 4. Backfill the missed filings on the next pull
```python
import json
state = json.load(open("pull_state.json"))
print("missed:", state["total"] - state["done"], "filings queued for tonight's pull")
```
Expected: A backfill queue. Gaps close on the next scheduled pull.

## Other ways people phrase this
### bulk filing pull timeout briefing
Separate the pull onto its own schedule. Brief from the local store.

### news agent filing pull failed
Resumable pulls with checkpoints. Backfill the misses.

### briefing waiting on downloads
The briefing reads disk. The network is the pull job's problem.

## Why it happens
Bulk pulls are slow and briefings are time-boxed; combining them guarantees timeouts. A separate scheduled pull with its own budget and checkpoints builds a local filing store over time, and the briefing reads from disk in seconds. The architecture separates slow collection from fast briefing.

## Edge cases
- The pull schedule must beat the briefing schedule; pull nightly, brief mornings.
- Amendments need re-pulls; the pull job checks for /A forms.
- Disk usage grows; archive old filings per a retention policy.
- A pull that never finishes needs its own watchdog and alert.
- If the pull job consistently misses its window, split it by filing type: 10-Ks one night, 8-Ks the next.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uV9k_nhr8lzxMaejrfYDLw