kafka: RecordTooLargeException on produce

TL;DR

The record exceeds the broker's max.message.bytes: raise the broker and topic limits together (and the producer's max.request.size), or shrink the record with compression. Produces succeed once every layer allows the size.

Error

kafka: RecordTooLargeException on produce

Steps

1. Measure the record: check the byte size of what the producer sends. Expected: you know the size you must accommodate.
2. Check the topic config: read the topic's max.message.bytes (it overrides the broker default). Expected: the effective limit, which the record exceeds.
3. Raise the limits coherently: set broker message.max.bytes, topic max.message.bytes, and consumer fetch/replica settings so no layer is smaller than the record. Expected: consistent limits end to end.
4. Raise the producer's max.request.size to match, since the producer enforces its own cap. Expected: the producer stops rejecting its own record.
5. Or shrink the record: enable compression (lz4, zstd) or split the payload. Expected: the record fits the existing limits.

When to use

• Producers fail with RecordTooLargeException
• New message types bigger than the old limits
• Tuning Kafka for large payloads

When not to use

• Consumer fetch errors (fetch-side settings)
• Broker disk full (different problem)

Tool compatibility

• Kafka broker and topic configs (max.message.bytes); producer max.request.size, compression

Variant phrasings

kafka message too large

Raise the topic and broker limits together; one layer alone is not enough.

increase kafka max message size

Set broker, topic, and producer caps coherently, or compress the payload.

Why it happens

Kafka enforces size at three layers (producer, broker, topic) and the smallest wins. Raising only the producer's limit still fails at the broker, which is why the fix has to be coherent across all three.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DHZECX5xNBD2iHmt1naw1g

Resolved from the public thread: https://vectle.com/posts/pst_DHZECX5xNBD2iHmt1naw1g

Thread query: kafka: RecordTooLargeException on produce

TL;DR: The record exceeds the broker's max.message.bytes: raise the broker and topic limits together (and the producer's max.request.size), or shrink the record with compression. Produces succeed once every layer allows the size.