UI errors loading traces after Langfuse upgrade: corrupted ClickHouse parts

Problem

After upgrading the self-hosted Langfuse docker image, the UI throws internal errors loading traces and traces are unreachable: logs show ClickHouse Stream String.size for column project_id/trace_id with type String is not found and There is no file for column 'id' in data part errors. CHECK TABLE found 17 of 100 parts failing on the observations table.

Verified fix

The root cause was storage corruption in ClickHouse data parts (not a migration bug; dirty flags in schemamigrations are normal: golang-migrate always writes a dirty=1 row followed by dirty=0). The maintainer's guidance: run CHECK TABLE traces/observations in ClickHouse to find the failing parts, then `ALTER TABLE [table] DROP PART '[partname]'` for the bad ones. The reporter confirmed Langfuse worked again after cleaning the damaged parts.

Source: https://github.com/langfuse/langfuse/issues/12172

Question: After upgrading the self-hosted Langfuse docker image, the UI throws internal errors loading traces and traces are unreachable: logs show ClickHouse Stream String.size for column project_id/trace_id with type String is not found and There is no file for column 'id' in data part errors. CHECK TABLE found 17 of 100 parts failing on the observations table.

Verified resolution: The root cause was storage corruption in ClickHouse data parts (not a migration bug; dirty flags in schemamigrations are normal: golang-migrate always writes a dirty=1 row followed by dirty=0). The maintainer's guidance: run CHECK TABLE traces/observations in ClickHouse to find the failing parts, then `ALTER TABLE [table] DROP PART '[partname]'` for the bad ones. The reporter confirmed Langfuse worked again after cleaning the damaged parts.

Source: https://github.com/langfuse/langfuse/issues/12172

Thread: https://vectle.com/threads/thr_73tNyxYO0hC1k7sjKpnXWw