## TL;DR
Size embedding batches by token budget, not item count. Pack up to the model's per-request limit with headroom, and on a too-large error halve the batch and retry; the shrink converges in a few attempts.

## Symptom
Embedding calls fail with request-too-large, token-limit, or 400/413 errors mentioning batch size, while smaller manual batches succeed. One long document in the batch blows the budget for everything packed with it.

## Steps

1. Size batches by token budget. Tokenize (or conservatively estimate) and pack items up to the model's per-request limit minus headroom:
```python
batch, tokens = [], 0
for text in texts:
    t = estimate_tokens(text)
    if budget_exceeded(tokens + t, TOKEN_BUDGET):
        yield batch; batch, tokens = [], 0
    batch.append(text); tokens += t
```
Expected output: no batch ever exceeds the token budget regardless of individual document length.

2. On a too-large error, halve the batch and retry. Binary shrink converges fast:
```python
while batch:
    try:
        return embed(batch)
    except TooLargeError:
        batch = batch[:len(batch)//2]
```
Expected output: the call succeeds within a few retries instead of failing the whole run.

3. Cap items per request too. Some APIs limit array length regardless of tokens; enforce both limits when packing.
Expected output: batches pass both the token check and the item-count check.

4. Log the batch size that succeeded and reuse it as the default packing size for that model.
Expected output: later runs start near the working size instead of rediscovering it.

## When to use
- Embedding errors mention batch size, token limits, or request too large
- The agent batches by fixed item count
- Document lengths vary wildly within a batch

## When not to use
- Invalid API key or model-not-found errors (thats auth or naming)
- Rate-limit 429s without size complaints (thats throttling, add backoff)
- Uniform tiny inputs that never approach the limit (fixed batches are fine)

## Variant phrasings

### embedding call fails only on certain batches
Those batches contain the long documents. Token-budget packing fixes exactly this pattern.

### 413 on an API that worked last month
The provider tightened limits or your documents got longer. Shrink and re-baseline the default.

## Why it happens
Agents batch by count ("100 items per call") because counts are easy. Token budgets are what the API enforces, and a single long document can exceed the whole batch's allowance, failing 99 good items along with it.

## Edge cases
- Tokenizers differ per model; estimate conservatively or use the model's own tokenizer when available.
- Empty strings still count as items against array-length limits; filter or count them.
- Retries on oversized batches compound with rate limits; add backoff alongside the shrink.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_my5pjhYKjhcR9rkt911ZKA