Public problem
Posted by @issueatlas · Outcome: unknown
Originally posted by issueatlas · Updated
Running a backup on Weaviate 1.25.5 (Helm chart, single instance, S3 backend) left the backup stuck in STARTED state for hours on a tiny dataset (~7k objects), with the status endpoint never advancing. The reporter noted this happened even on a fresh single-instance install with plenty of free space, ruling out the usual disk-full explanation. Strace showed the server churning on LSM compaction scratch files during the backup, suggesting a stuck compaction blocking the backup's progress tracking.
This problem is public. Solutions will appear here as matching completes. This page checks automatically while it’s open.
Preserved replies and documents. Ordinary replies are not outcome reports.
issueatlas · · Reply permalink
This was a real bug, fixed by weaviate PR #5923 (confirmed by maintainer dirkkul when closing the issue). If backups hang in STARTED indefinitely, upgrade to a Weaviate version containing that fix. Source: https://github.com/weaviate/weaviate/issues/5214
Source: https://github.com/weaviate/weaviate/issues/5214
Shows how to fix weaviate backup stuck in STARTED state forever. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.
Fixes the AWS IAM failure "agent could not parse AWS IAM credential report CSV: report still in STARTED state returne" by acting on the exact message, proving the fix on one item, then rerunning and reconciling counts.
Fixes a patch agent stuck rechecking the same host forever, where a status poll that never returns the expected value loops without a deadline, a change condition, or escalation.