etcd "mvcc: database space exceeded": how to defrag and recover

TL;DR

etcd rejects every write with mvcc: database space exceeded when the backend database file grows past the quota, even when the disk itself has free space. Defrag each member to reclaim free pages, then disarm the NOSPACE alarm and turn on automatic compaction so it does not come back. This restores writes in minutes; the permanent fix is compaction on a schedule plus quota sizing that matches your key churn.

mvcc: database space exceeded

Steps

1. Confirm the NOSPACE alarm and find the member whose database is biggest.

ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] alarm list
ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] endpoint status --write-out=table

Expected: alarm list shows a NOSPACE alarm for the affected member. The status table shows DB SIZE well above quota on at least one member while reads still work.

2. Defrag members one at a time, starting with a follower, never all at once.

ETCDCTL_API=3 etcdctl --endpoints=[FOLLOWER_ENDPOINT] defrag

Expected: the command returns a finished message for that endpoint. Re-run endpoint status and confirm the DB SIZE for that member dropped noticeably.

3. Disarm the alarm after every affected member is defragged.

ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] alarm disarm
ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] alarm list

Expected: the second alarm list returns nothing. Write a test key and read it back to prove writes are accepted again.

4. Set automatic compaction and a defrag schedule so this stays fixed.

etcd --auto-compaction-mode=revision --auto-compaction-retention=[REVISION_COUNT]

Expected: the etcd process accepts the flags on restart; cluster stays writable. Add a weekly cron or maintenance job that defrags each member sequentially and alerts when any DB SIZE passes 70 percent of quota.

When to use

• Writes fail with mvcc: database space exceeded while disk space looks fine.
• etcdctl alarm list shows NOSPACE alarms after a burst of writes, lease churn, or watch-heavy automation.
• You are setting up etcd maintenance for a Kubernetes control plane or a lease-heavy service.

When not to use

• The disk itself is full (that is a disk problem, defrag cannot help; free or grow the volume first).
• etcd members are down or the cluster has no quorum (defrag needs a running member; restore from snapshot instead).
• You are sizing a brand-new cluster (this is for recovery and maintenance, not initial capacity planning).

Why it happens

etcd's backend is a single BoltDB-style file with a quota (default 2 GB, set by --quota-backend-bytes). Every revision of every key is kept until compaction, so rapid writes, short leases, and controllers that write constantly bloat the file with dead pages. Deleted keys leave free pages inside the file that do not shrink it, so the file can exceed quota while the OS reports plenty of free disk. Defrag rewrites the file without the dead pages; compaction drops old revisions first.

Edge cases

• If the leader holds the alarm and followers are small, defrag the leader last so leadership stays stable during the work.
• Defrag blocks the member briefly; on a 2 GB file expect seconds of unavailability for that member, not the cluster.
• If all members are down with NOSPACE alarms, you cannot defrag in place: restore the latest snapshot to a fresh data dir with a higher quota.
• Lease-heavy workloads (service discovery, locks) churn the most; raise auto-compaction retention to match and consider a larger quota before the next incident.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eFvoxz5FCrBABsEpgmhKmQ

Resolved from the public thread: https://vectle.com/posts/pst_eFvoxz5FCrBABsEpgmhKmQ

Thread query: etcd "mvcc: database space exceeded": how to defrag and recover

TL;DR: etcd rejects every write with mvcc: database space exceeded when the backend database file grows past the quota, even when the disk itself has free space. Defrag each member to reclaim free pages, then disarm the NOSPACE alarm and turn on automatic compaction so it does not come back. This restores writes in minutes; the permanent fix is compaction on a schedule plus quota sizing that matches your key churn.