Skip to content
← Back to activity

Public problem

Eval runs much slower after upgrading Braintrust TS SDK to v1.0.0 (log flush regression)

Posted by @issueatlas · Outcome: unknown

Originally posted by issueatlas · Updated

Upgrading the Braintrust TypeScript SDK from v0.4.10 to v1.0.0 made eval runs much slower (a few hundred tasks: 1.5 min before, 5 min after). The time is spent in sequential uploads to a /logs3 API endpoint in a loop that was introduced to reduce memory usage: previously logging was not awaited per task, now the code awaits small sequential RPCs after each task. noSendLogs=true does not help.

Finding relevant solutions

This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.

History & evidence

Original discussion

Preserved replies and documents. Ordinary replies are not outcome reports.

issueatlas · · Reply permalink

This was a real regression: per-task flushing awaited sequential /logs3 uploads. The interim workaround is export BRAINTRUST_LOG_FLUSH_CHUNK_SIZE=10000000 (larger chunks, still not full speed): note it must be set so it applies to subprocesses too. The maintainers then reworked the flushing and released the proper fix in js-sdk v3.3.0 (PR #1459). The reporter confirmed v3.3.0 completely fixes the regression. Upgrade to braintrust >= 3.3.0.

Source: https://github.com/braintrustdata/braintrust-sdk-javascript/issues/1394