## TL;DR
etcd rejects every write with `mvcc: database space exceeded` when the backend database file grows past the quota, even when the disk itself has free space. Defrag each member to reclaim free pages, then disarm the NOSPACE alarm and turn on automatic compaction so it does not come back. This restores writes in minutes; the permanent fix is compaction on a schedule plus quota sizing that matches your key churn.

```text
mvcc: database space exceeded
```

## Steps

1. Confirm the NOSPACE alarm and find the member whose database is biggest.

```bash
ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] alarm list
ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] endpoint status --write-out=table
```

Expected: `alarm list` shows a NOSPACE alarm for the affected member. The status table shows DB SIZE well above quota on at least one member while reads still work.

2. Defrag members one at a time, starting with a follower, never all at once.

```bash
ETCDCTL_API=3 etcdctl --endpoints=[FOLLOWER_ENDPOINT] defrag
```

Expected: the command returns a finished message for that endpoint. Re-run `endpoint status` and confirm the DB SIZE for that member dropped noticeably.

3. Disarm the alarm after every affected member is defragged.

```bash
ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] alarm disarm
ETCDCTL_API=3 etcdctl --endpoints=[ETCD_ENDPOINTS] alarm list
```

Expected: the second `alarm list` returns nothing. Write a test key and read it back to prove writes are accepted again.

4. Set automatic compaction and a defrag schedule so this stays fixed.

```bash
etcd --auto-compaction-mode=revision --auto-compaction-retention=[REVISION_COUNT]
```

Expected: the etcd process accepts the flags on restart; cluster stays writable. Add a weekly cron or maintenance job that defrags each member sequentially and alerts when any DB SIZE passes 70 percent of quota.

## When to use

- Writes fail with `mvcc: database space exceeded` while disk space looks fine.
- `etcdctl alarm list` shows NOSPACE alarms after a burst of writes, lease churn, or watch-heavy automation.
- You are setting up etcd maintenance for a Kubernetes control plane or a lease-heavy service.

## When not to use

- The disk itself is full (that is a disk problem, defrag cannot help; free or grow the volume first).
- etcd members are down or the cluster has no quorum (defrag needs a running member; restore from snapshot instead).
- You are sizing a brand-new cluster (this is for recovery and maintenance, not initial capacity planning).

## Why it happens

etcd's backend is a single BoltDB-style file with a quota (default 2 GB, set by `--quota-backend-bytes`). Every revision of every key is kept until compaction, so rapid writes, short leases, and controllers that write constantly bloat the file with dead pages. Deleted keys leave free pages inside the file that do not shrink it, so the file can exceed quota while the OS reports plenty of free disk. Defrag rewrites the file without the dead pages; compaction drops old revisions first.

## Edge cases

- If the leader holds the alarm and followers are small, defrag the leader last so leadership stays stable during the work.
- Defrag blocks the member briefly; on a 2 GB file expect seconds of unavailability for that member, not the cluster.
- If all members are down with NOSPACE alarms, you cannot defrag in place: restore the latest snapshot to a fresh data dir with a higher quota.
- Lease-heavy workloads (service discovery, locks) churn the most; raise auto-compaction retention to match and consider a larger quota before the next incident.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eFvoxz5FCrBABsEpgmhKmQ