agent's embedding batch too large for the api: chunk sizing with retry

TL;DR

Size embedding batches by token budget, not item count. Pack up to the model's per-request limit with headroom, and on a too-large error halve the batch and retry; the shrink converges in a few attempts.

Symptom

Embedding calls fail with request-too-large, token-limit, or 400/413 errors mentioning batch size, while smaller manual batches succeed. One long document in the batch blows the budget for everything packed with it.

Steps

1. Size batches by token budget. Tokenize (or conservatively estimate) and pack items up to the model's per-request limit minus headroom:

batch, tokens = [], 0
for text in texts:
    t = estimate_tokens(text)
    if budget_exceeded(tokens + t, TOKEN_BUDGET):
        yield batch; batch, tokens = [], 0
    batch.append(text); tokens += t

Expected output: no batch ever exceeds the token budget regardless of individual document length.

2. On a too-large error, halve the batch and retry. Binary shrink converges fast:

while batch:
    try:
        return embed(batch)
    except TooLargeError:
        batch = batch[:len(batch)//2]

Expected output: the call succeeds within a few retries instead of failing the whole run.

3. Cap items per request too. Some APIs limit array length regardless of tokens; enforce both limits when packing.

Expected output: batches pass both the token check and the item-count check.

4. Log the batch size that succeeded and reuse it as the default packing size for that model.

Expected output: later runs start near the working size instead of rediscovering it.

When to use

• Embedding errors mention batch size, token limits, or request too large
• The agent batches by fixed item count
• Document lengths vary wildly within a batch

When not to use

• Invalid API key or model-not-found errors (thats auth or naming)
• Rate-limit 429s without size complaints (thats throttling, add backoff)
• Uniform tiny inputs that never approach the limit (fixed batches are fine)

Variant phrasings

embedding call fails only on certain batches

Those batches contain the long documents. Token-budget packing fixes exactly this pattern.

413 on an API that worked last month

The provider tightened limits or your documents got longer. Shrink and re-baseline the default.

Why it happens

Agents batch by count ("100 items per call") because counts are easy. Token budgets are what the API enforces, and a single long document can exceed the whole batch's allowance, failing 99 good items along with it.

Edge cases

• Tokenizers differ per model; estimate conservatively or use the model's own tokenizer when available.
• Empty strings still count as items against array-length limits; filter or count them.
• Retries on oversized batches compound with rate limits; add backoff alongside the shrink.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_my5pjhYKjhcR9rkt911ZKA