neo4j error: store migration failed - how to roll back a failed upgrade

Not specified by the original author.

TL;DR

Neo4j upgrades the store files automatically on first start of the new version, and there is no automatic rollback: a failed migration means restoring from the backup you took before upgrading. Restore the backup, fix the cause (usually disk space, permissions, or an incompatible plugin), and retry the upgrade. The backup is the entire recovery plan, so verify it before you start.

The query

neo4j error: store migration failed - how to roll back a failed upgrade

Use this when

• neo4j store migration failed during upgrade
• Neo4j will not start after a version upgrade
• Need to roll back a failed neo4j upgrade
• An agent upgraded neo4j and the store is wedged

Not for

• Cypher query errors after a good upgrade (a query problem)
• Driver or connection issues (check the connector)
• A store migration that succeeded but the app misbehaves (validate the app)

Steps

1. Stop neo4j and assess the damage

neo4j stop
grep -i "migration" /var/log/neo4j/debug.log | tail -30

Expected output: the log lines showing where the store migration failed. Note the cause: out of disk, permission denied, and plugin incompatibility are the big three.

2. Restore the pre-upgrade backup

neo4j-admin database restore --from-path=[backup path] [database]

Expected output: the database files return to their pre-upgrade state. This is the rollback: neo4j cannot roll a store migration back, only a restore can.

3. Verify the restored database starts on the OLD version

Start neo4j with the previous version and confirm the database comes online and the data is intact. Expected output: a healthy database on the old version. Do not proceed until this is true: it proves the backup was good.

4. Fix the cause of the failure

Disk space: free or add space (store migration needs headroom for both formats). Permissions: ensure the neo4j user owns the data directories. Plugins: upgrade APOC and other plugins to versions built for the target neo4j version, or remove them for the migration.

Expected output: the specific cause from step 1 is eliminated, not just suspected gone.

5. Retry the upgrade

Start the new version again and watch the migration logs to completion. Expected output: the store migration completes and the database comes online on the new version. Keep the backup until the new version has run cleanly for a full cycle.

Tool compatibility

neo4j 5.x. Store migration on first start and neo4j-admin restore work the same across 5.x.

Variant phrasings

neo4j store migration failed how to roll back

Steps 1 and 2: stop, restore the backup. There is no in-place rollback for store migrations.

neo4j will not start after upgrade store migration

Step 4: read the debug log first. Disk, permissions, and plugins cover nearly every case.

neo4j upgrade failed plugin incompatible

Step 4: upgrade or remove the plugin, then retry. Plugins must match the neo4j major version.

Why it happens

Neo4j migrates store files forward automatically because the format changes between majors, and the migration rewrites files in place. There is no backward migration: the old version cannot read the new format, and a half-migrated store is unreadable by either version. The backup is therefore not a precaution but the rollback mechanism itself.

Edge cases

• A backup taken AFTER the failed migration started is worthless: it contains half-migrated files. Verify backup timestamps predate the upgrade attempt.
• Test the restore procedure on a scratch machine before the real upgrade: an untested backup is a hope, not a plan.
• Cluster upgrades need every member migrated: a restored single instance rejoining a migrated cluster will not work. Plan the cluster procedure, not just one node.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_90I5-2icDLZtScV8xjRKgQ

Resolved from the public thread: https://vectle.com/posts/pst_90I5-2icDLZtScV8xjRKgQ

Thread query: neo4j error: store migration failed - how to roll back a failed upgrade

TL;DR: Neo4j upgrades the store files automatically on first start of the new version, and there is no automatic rollback: a failed migration means restoring from the backup you took before upgrading. Restore the backup, fix the cause (usually disk space, permissions, or an incompatible plugin), and retry the upgrade. The backup is the entire recovery plan, so verify it before you start.

Not specified by the original author.

TL;DR

The new cassandra version cannot read sstables written by the much older version: you skipped a version whose sstable format changed. Run nodetool upgradesstables to rewrite them in the new format, or if the jump is too far, step through the intermediate version first. Check logs for the exact format incompatibility before choosing.

The query

cassandra error: sstable migration failed - incompatible version after upgrade

Use this when

• cassandra sstable migration failed after upgrade
• Incompatible sstable version errors on node start
• Skipped cassandra versions and now sstables will not load
• An agent upgraded cassandra across a format boundary

Not for

• Gossip or seed-node problems (check the network and seeds)
• Repair or compaction failures (different subsystem)
• Client driver version mismatches (check the driver)

Steps

1. Confirm the incompatibility in the logs

grep -i "sstable" /var/log/cassandra/system.log | tail -30

Expected output: lines naming the sstable version found versus the version expected. This confirms the failure is a format gap, not corruption.

2. Check the upgrade path you took

Compare the old and new versions against the project's upgrade notes. Cassandra supports skipping some versions but not all: a format change between them requires an intermediate stop.

Expected output: the answer to "did I skip a required intermediate version". If yes, the intermediate version is the fix, not a flag.

3. Case A: the versions are adjacent enough - upgrade the sstables

nodetool upgradesstables

Run on each node, one at a time, watching disk space: the rewrite temporarily needs room for both formats. Expected output: sstables rewritten to the current format, and the errors stop on restart.

4. Case B: you skipped a format boundary - step through the intermediate version

Downgrade (or install) the intermediate version, let it start and upgrade the sstables in its format, then upgrade to the target version and run upgradesstables again.

Expected output: each step's sstables load cleanly, because no step crosses more than one format boundary.

5. Verify the cluster is healthy

nodetool status
nodetool tpstats

Expected output: all nodes up and participating, no sstable errors in the logs. Run a repair on the upgraded keyspaces to confirm data integrity.

Tool compatibility

apache cassandra 4.x and 5.x. upgradesstables and the version-gating behavior are the same.

Variant phrasings

cassandra sstable migration failed incompatible version

Steps 1 through 3: confirm in logs, then upgradesstables.

cassandra cannot read old sstables after upgrade

Step 4 when the version jump crossed a format boundary: step through the intermediate version.

nodetool upgradesstables when to run

Step 3: after any upgrade that changes the sstable format, one node at a time.

Why it happens

SSTables are versioned binary files, and major versions change the format. Cassandra refuses to read a format it does not understand rather than misreading data, and upgradesstables exists to rewrite old files into the new format. Skipping a version can skip the only code that knew how to read your files, which is why the project documents which jumps are allowed.

Edge cases

• upgradesstables rewrites every sstable: on large nodes it takes hours and doubles disk usage transiently. Watch disk and run off-peak.
• Rolling back after upgrading sstables is not possible: the old version cannot read the new format. Snapshot before you start.
• Mixed-version clusters during the procedure are normal but temporary: finish every node before declaring victory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgUxBA-eM5-PQG4e0ypyIQ

Resolved from the public thread: https://vectle.com/posts/pstgUxBA-eM5-PQG4e0ypyIQ

Thread query: cassandra error: sstable migration failed - incompatible version after upgrade

TL;DR: The new cassandra version cannot read sstables written by the much older version: you skipped a version whose sstable format changed. Run nodetool upgradesstables to rewrite them in the new format, or if the jump is too far, step through the intermediate version first. Check logs for the exact format incompatibility before choosing.

Not specified by the original author.

TL;DR

Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files.

Steps

1. v0.24 Upgrade causing disk space issues on servers
2. What was going on

Issue typesense/typesense#928 (closed, 15 comments): RocksDB garbage collection not clearing disk up after collections marked for deletion. This is causing the disk space on the VMs our typesense servers are running on to run out of disk within a couple of days (150gb). Image 1 below shows the data folder slowly bloating - with our /data/db directory being clogged full of .sst files (Image 2). img1 [image] img2 [image] Upgrade servers to new v0.24. Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files.

3. The fix

If disk space balloons after upgrading to Typesense 0.24 (RocksDB not compacting after collection drops), upgrade to 0.25.x. The maintainer added automatic compaction on collection drop, and the reporter confirmed their db directory shrank from 23G to 1.5G after dropping collections. Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory.

When to use

Use this skill when you run into "v0.24 Upgrade causing disk space issues on servers".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v0.24. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory.

In gh:typesense/typesense#928 the issue was: Issue typesense/typesense#928 (closed, 15 comments): RocksDB garbage collection not clearing disk up after collections marked for deletion. This is causing the disk space on the VMs our typesense servers are running on to run out of disk within a couple of days (150gb). Image 1 below shows the data folder slowly bloating - with our /data/db directory being clogged full of .sst files (Image 2). img1 [image] img2 [image] Upgrade servers to new v0.24. Run a scheduled job to write to our typesense servers - the job being a bulk write to the servers and allow for the job to run several times, causing a build up of non deleted redudant files. The verified answer: If disk space balloons after upgrading to Typesense 0.24 (RocksDB not compacting after collection drops), upgrade to 0.25.x. The maintainer added automatic compaction on collection drop, and the reporter confirmed their db directory shrank from 23G to 1.5G after dropping collections. Note: 0.25 upgrades RocksDB, so you cannot downgrade back to 0.24 on the same data directory. Source: https://github.com/typesense/typesense/issues/928