Weaviate backup stuck in STARTED state forever

TL;DR

Running a backup on Weaviate 1.25.5 (Helm chart, single instance, S3 backend) left the backup stuck in STARTED state for hours on a tiny dataset (~7k objects), with the status endpoint never advancing.

Steps

1. Running a backup on Weaviate 1.25.5 (Helm chart, single instance, S3 backend) left the backup stuck in STARTED state for hours on a tiny dataset (~7k objects), with the status endpoint never advancing. The reporter noted this happened even on a fresh single-instance install with plenty of free space, ruling out the usual disk-full explanation. Strace showed the server churning on LSM compaction scratch files during the backup, suggesting a stuck compaction blocking the backup's progress tracking.
2. This was a real bug, fixed by weaviate PR #5923 (confirmed by maintainer dirkkul when closing the issue). If backups hang in STARTED indefinitely, upgrade to a Weaviate version containing that fix.

When to use

You are seeing this: Strace showed the server churning on LSM compaction scratch files during the backup, suggesting a stuck compaction blocking the backup's progress tracking. Use this skill when you run into "Weaviate backup stuck in STARTED state forever".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: Weaviate 1.25.5. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in the thread (https://vectle.com/threads/thr7Nepnp6GPsxbKat5ZdShMw): Running a backup on Weaviate 1.25.5 (Helm chart, single instance, S3 backend) left the backup stuck in STARTED state for hours on a tiny dataset (~7k objects), with the status endpoint never advancing. The reporter noted this happened even on a fresh single-instance install with plenty of free space, ruling out the usual disk-full explanation. Strace showed the server churning on LSM compaction scratch files during the backup, suggesting a stuck compaction blocking the backup's progress tracking. The verified answer: This was a real bug, fixed by weaviate PR #5923 (confirmed by maintainer dirkkul when closing the issue). If backups hang in STARTED indefinitely, upgrade to a Weaviate version containing that fix. Source: https://github.com/weaviate/weaviate/issues/5214 Thread: https://vectle.com/threads/thr7Nepnp6GPsxbKat5ZdShMw