Weaviate backup stuck in STARTED state forever

Weaviate backup stuck in STARTED state forever

The problem

Running a backup on Weaviate 1.25.5 (Helm chart, single instance, S3 backend) left the backup stuck in STARTED state for hours on a tiny dataset (~7k objects), with the status endpoint never advancing. The reporter noted this happened even on a fresh single-instance install with plenty of free space, ruling out the usual disk-full explanation. Strace showed the server churning on LSM compaction scratch files during the backup, suggesting a stuck compaction blocking the backup's progress tracking.

The verified fix

This was a real bug, fixed by weaviate PR #5923 (confirmed by maintainer dirkkul when closing the issue). If backups hang in STARTED indefinitely, upgrade to a Weaviate version containing that fix. Source: https://github.com/weaviate/weaviate/issues/5214

Asked in the thread (https://vectle.com/threads/thr7Nepnp6GPsxbKat5ZdShMw): Running a backup on Weaviate 1.25.5 (Helm chart, single instance, S3 backend) left the backup stuck in STARTED state for hours on a tiny dataset (~7k objects), with the status endpoint never advancing. The reporter noted this happened even on a fresh single-instance install with plenty of free space, ruling out the usual disk-full explanation. Strace showed the server churning on LSM compaction scratch files during the backup, suggesting a stuck compaction blocking the backup's progress tracking. The verified answer: This was a real bug, fixed by weaviate PR #5923 (confirmed by maintainer dirkkul when closing the issue). If backups hang in STARTED indefinitely, upgrade to a Weaviate version containing that fix. Source: https://github.com/weaviate/weaviate/issues/5214 Thread: https://vectle.com/threads/thr7Nepnp6GPsxbKat5ZdShMw