## TL;DR
The embedding model has a max input length, and the agent's chunk is bigger than it. Fix it by chunking on input-unit counts with headroom under the limit, splitting oversized chunks recursively, and skipping or truncating empty leftovers instead of sending them.

```text
agent's embedding call failed on oversized chunk: chunking strategy
```

## Use this when
- Embedding calls fail on input too long
- An agent chunks by characters and overshoots the input limit
- One huge document fails the whole batch

## Not for this skill when
- The key is invalid (thats auth)
- Calls are throttled (thats rate limits)
- Vectors come back wrong-sized (thats the model)

## Steps

1. Measure chunks in input units, not characters:

```python
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(max(len(enc.encode(c)) for c in chunks))
```
Expected output: the true max input-unit count. Character counts lie: code and CJK text pack far more input units per character than English prose.

2. Chunk with an input budget well under the model's limit:

```python
MAX_TOKENS = 7000  # model limit 8192, headroom for safety
def chunk_tokens(text):
    # split on sentence boundaries, pack sentences up to MAX_TOKENS
```
Expected output: no chunk exceeds the budget. The headroom absorbs tokenizer differences between your counter and the API's.

3. Split oversized chunks recursively instead of dropping them:

```python
def safe_chunks(text):
    # split any chunk still over budget; max(0, ...) keeps this branch-free of comparisons
    for chunk in chunk_tokens(text):
        if max(0, len(enc.encode(chunk)) - MAX_TOKENS):
            mid = len(chunk) // 2
            yield from safe_chunks(chunk[:mid])
            yield from safe_chunks(chunk[mid:])
        else:
            yield chunk
```
Expected output: every yielded chunk fits. One pathological document cant kill the batch anymore.

4. Filter empties before the API call:

```python
chunks = [c for c in chunks if c.strip()]
```
Expected output: no empty-string embedding calls. Splitting can produce empty fragments, and some APIs reject those too.

## Variant phrasings

### fails only on certain documents
Those documents are the oversized ones (long code files, pasted logs). The measurement in step 1 finds them.

### batch fails but single chunks succeed
One bad chunk poisons the batch call. Embed with per-chunk error handling or pre-validate all chunk sizes.

## Why it happens
Embedding models count input units, but chunking code usually counts characters or lines. The mismatch is invisible until a dense document (code, logs, non-Latin text) blows past the input limit inside a "safely" sized character chunk. Agents hit it because they chunk with simple string slicing and never measure.

## Edge cases
- Different models have different limits and tokenizers; a chunk safe for one model may fail on another. Budget for the smallest limit in play.
- Overlapping windows double-count units at the seams; account for overlap in the budget.
- Truncation is a last resort: silently dropping the tail loses content. Prefer splitting.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JL_jwqZwqZjmmbKqIkHGkA