agent's parser failed on workerd's multi-line uncaught exception: stack split across log entries

TL;DR

Workerd tail streams exceptions as multiple log entries: the message in one, stack frames scattered across others, sometimes interleaved with unrelated logs. Line-oriented parsers capture fragments and miss the error. Reassemble by correlation (timestamp proximity and request ID), not by line, and the agent sees whole exceptions again.

The query

agent's parser failed on workerd's multi-line uncaught exception: stack split across log entries

Use this when

• The agent reports 'no error found' while tail clearly shows an exception.
• Stack traces appear as disconnected fragments in the agent's summary.
• The parser works for single-line logs but fails on exceptions.

Not for

• Tail showing no logs at all (filter or traffic problem).
• JSON vs pretty format changes (format problem, separate fix).
• Stackless errors (nothing to reassemble).

Steps

Step 1: Capture raw tail output around the failure

Save the unmodified tail stream for the failing window. Do not let the agent summarize it first.

Expected result: the raw stream shows the exception's pieces: message line, frame lines, possibly interleaved with other requests' logs.

Step 2: Find the correlation key

Workerd log entries carry timestamps and, for request-scoped logs, request identifiers. Group entries by request ID first, timestamp proximity second.

Expected result: the fragments belonging to one exception cluster together. Interleaved entries from other requests separate out.

Step 3: Rewrite the parser as a reassembler

Buffer entries for a short window, group by correlation key, and emit one exception object per group instead of one parsed line per entry.

Expected result: the agent receives whole exceptions with message plus full stack, the way a human reading the raw stream does.

Step 4: Handle the frame-line formats

Stack frames, 'caused by' chains, and async boundaries each have their own line shape. Parse them as continuations of the exception, not as independent log lines.

Expected result: no frame is mistaken for a new error, and no error loses its frames.

Step 5: Verify against a known exception

Trigger a deliberate uncaught exception in preview and check the agent's parsed output.

Expected result: the agent reports the full exception, correctly attributed, on the first try.

Why it happens

Log streams are line-oriented transports carrying multi-line structures. Human readers reassemble effortlessly; regex-per-line parsers cannot. The parser was written against tidy single-line logs and never updated for the reality of exception output.

Edge cases

• High-traffic workers interleave heavily. Shorter buffer windows reduce mixing but risk splitting slow exceptions; tune per traffic level.
• Some frames arrive out of order. Sort within the group by timestamp before presenting.
• The first line is not always the message; 'uncaught' wrappers can precede it. Treat the group, not the first line, as the unit.
• JSON log format changes the field layout but not the fragmentation problem. Reassembly is needed in both formats.

Provenance

Resolved from the public thread: https://vectle.com/posts/psta0N4rgGgNITzvBs5zecbQ