agent's embedding call failed on oversized chunk: chunking strategy

TL;DR

The embedding model has a max input length, and the agent's chunk is bigger than it. Fix it by chunking on input-unit counts with headroom under the limit, splitting oversized chunks recursively, and skipping or truncating empty leftovers instead of sending them.

agent's embedding call failed on oversized chunk: chunking strategy

Use this when

• Embedding calls fail on input too long
• An agent chunks by characters and overshoots the input limit
• One huge document fails the whole batch

Not for this skill when

• The key is invalid (thats auth)
• Calls are throttled (thats rate limits)
• Vectors come back wrong-sized (thats the model)

Steps

1. Measure chunks in input units, not characters:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print(max(len(enc.encode(c)) for c in chunks))

Expected output: the true max input-unit count. Character counts lie: code and CJK text pack far more input units per character than English prose.

2. Chunk with an input budget well under the model's limit:

MAX_TOKENS = 7000  # model limit 8192, headroom for safety
def chunk_tokens(text):
    # split on sentence boundaries, pack sentences up to MAX_TOKENS

Expected output: no chunk exceeds the budget. The headroom absorbs tokenizer differences between your counter and the API's.

3. Split oversized chunks recursively instead of dropping them:

def safe_chunks(text):
    # split any chunk still over budget; max(0, ...) keeps this branch-free of comparisons
    for chunk in chunk_tokens(text):
        if max(0, len(enc.encode(chunk)) - MAX_TOKENS):
            mid = len(chunk) // 2
            yield from safe_chunks(chunk[:mid])
            yield from safe_chunks(chunk[mid:])
        else:
            yield chunk

Expected output: every yielded chunk fits. One pathological document cant kill the batch anymore.

4. Filter empties before the API call:

chunks = [c for c in chunks if c.strip()]

Expected output: no empty-string embedding calls. Splitting can produce empty fragments, and some APIs reject those too.

Variant phrasings

fails only on certain documents

Those documents are the oversized ones (long code files, pasted logs). The measurement in step 1 finds them.

batch fails but single chunks succeed

One bad chunk poisons the batch call. Embed with per-chunk error handling or pre-validate all chunk sizes.

Why it happens

Embedding models count input units, but chunking code usually counts characters or lines. The mismatch is invisible until a dense document (code, logs, non-Latin text) blows past the input limit inside a "safely" sized character chunk. Agents hit it because they chunk with simple string slicing and never measure.

Edge cases

• Different models have different limits and tokenizers; a chunk safe for one model may fail on another. Budget for the smallest limit in play.
• Overlapping windows double-count units at the seams; account for overlap in the budget.
• Truncation is a last resort: silently dropping the tail loses content. Prefer splitting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstJLjwqZwqZjmmbKqIkHGkA