agent failed on truncated json patch metadata response

TL;DR

A truncated body is a transport problem with a parser symptom. The agent should detect incompleteness before parsing (content-length mismatch, a streaming JSON reader that reports unexpected end), retry with backoff, and where the API supports it, page or compress the request so responses stay small. Retrying the identical huge request forever is not a strategy; if the truncation is deterministic at a size, the fix is smaller requests or a different endpoint (bulk download of the catalog file).

agent failed on truncated json patch metadata response

Use this when

• JSON parse errors cite unexpected end of data or unterminated strings, on the largest responses only.
• The same call succeeds when filtered to a smaller result set.
• A proxy or gateway sits between the agent and the API.

Not for this skill when

• The server returns complete but invalid JSON (a schema or encoding bug). Retries and paging will not change it; report it to the API owner with a sample.
• The response is an error page in HTML with a 200 from a captive portal or SSO redirect. That is an auth interception, not truncation.

Steps

1. Capture the raw response bytes and headers for a failing call and compare the body size to the content-length header. Verify: the truncation is proven with byte counts, not inferred
2. Add completeness checks in the client: verify length where provided, parse with a streaming reader that distinguishes incomplete from invalid, and classify truncation as retryable. Verify: a deliberately truncated fixture is classified and retried, not parsed-and-crashed
3. Reduce response size: use pagination, field selection, or compression (accept-encoding) on the metadata endpoint. Verify: the largest fetch now completes intact across repeated runs
4. If truncation persists at a hard size ceiling in the path, switch that data source to its bulk form (catalog file download with resume support) and parse locally. Verify: the bulk path produces the same metadata set, validated by record counts
5. Add response-size and truncation metrics to the run log. Verify: future truncations are visible as transport events, not mysterious parse crashes

Variant phrasings

truncated json response unexpected end of data large catalog

The main phrasing.

content length mismatch proxy truncates api response

The transport phrasing.

page or bulk download patch metadata too large parse

The strategy phrasing.

Compatibility: Compatibility: Any JSON REST API; bulk catalog options exist for WSUS exports and vendor advisory feeds.

Why it happens

JSON has no partial credit: one missing closing bracket invalidates megabytes of good data, so a transport that drops the last chunk produces total failure with a confusing message. The correlation with size is the fingerprint: proxies with response limits, gateways with timeout settings, and clients reading with a fixed buffer all truncate at scale and never on small test calls, which is why the bug survives testing and blooms on the biggest, most important sync of the month.

Edge cases / pitfalls

• Do not attempt to repair truncated JSON by appending brackets; you cannot know what else was lost, and acting on a partial catalog silently under-patches.
• Retries should include a size strategy change after the second truncation, or you will loop on a deterministic ceiling.
• Checksum or record-count validation after parse is the belt-and-braces control for bulk files.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OiEwjaOKvcZ3CJcL0GHreA