agent looped re-downloading the same concur export - etag ignored

TL;DR

The agent downloads the Concur export on every run without checking whether it changed, so it reprocesses identical data in a loop. Store the export's etag (or checksum) from the last run and send it as If-None-Match; when Concur returns 304 Not Modified, skip the download and the reprocessing. The etag is the change signal; ignoring it is the bug.

Error

agent looped re-downloading the same concur export  -  etag ignored

Steps

1. Confirm the loop: the agent log shows repeated full downloads with identical content. Expected: the downloads are byte-identical across runs.
2. Check whether the export endpoint returns an etag or last-modified header. Expected: the headers are available on the response.
3. Store the etag after each successful download and send it as If-None-Match on the next run. Expected: unchanged exports return 304.
4. On 304, skip the download and mark the data as current; on 200, process and store the new etag. Expected: the agent only processes changed exports.
5. Add a loop guard: alert if the agent processes the same checksum twice in a row. Expected: future ignore-the-etag regressions are caught.

When to use

• An agent re-downloads the same Concur export every run.
• Audit runs reprocess identical data.
• The export endpoint supports etags or last-modified headers.

When not to use

• The export genuinely changes between runs (the downloads are correct).
• The endpoint offers no change-detection headers (poll on a schedule instead).
• The loop is in a different stage of the pipeline.

Tool compatibility

Applies to custom audit agents fetching Concur exports (Concur APIs, Python or Node agents; current 2026 releases).

Variant phrasings

agent reprocesses the same export daily

Honor the etag and skip unchanged downloads.

concur export downloaded but never changes

The 304 path is missing. Add conditional requests.

infinite re-download loop in the audit agent

Store the checksum and break the loop on identical content.

Why it happens

Polling loops are the default agent pattern: fetch, process, repeat. Without conditional requests, every poll is a full download, and if the downstream processing has no idempotency, the same data gets re-audited forever. Etags exist to make polling cheap and correct; the agent just has to use them. The checksum guard is belt-and-suspenders for endpoints with weak header support.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_l1D4VzVF3R37wFfyLGRFcg