cassandra error: sstable migration failed - incompatible version after upgrade

Not specified by the original author.

TL;DR

The new cassandra version cannot read sstables written by the much older version: you skipped a version whose sstable format changed. Run nodetool upgradesstables to rewrite them in the new format, or if the jump is too far, step through the intermediate version first. Check logs for the exact format incompatibility before choosing.

The query

cassandra error: sstable migration failed - incompatible version after upgrade

Use this when

• cassandra sstable migration failed after upgrade
• Incompatible sstable version errors on node start
• Skipped cassandra versions and now sstables will not load
• An agent upgraded cassandra across a format boundary

Not for

• Gossip or seed-node problems (check the network and seeds)
• Repair or compaction failures (different subsystem)
• Client driver version mismatches (check the driver)

Steps

1. Confirm the incompatibility in the logs

grep -i "sstable" /var/log/cassandra/system.log | tail -30

Expected output: lines naming the sstable version found versus the version expected. This confirms the failure is a format gap, not corruption.

2. Check the upgrade path you took

Compare the old and new versions against the project's upgrade notes. Cassandra supports skipping some versions but not all: a format change between them requires an intermediate stop.

Expected output: the answer to "did I skip a required intermediate version". If yes, the intermediate version is the fix, not a flag.

3. Case A: the versions are adjacent enough - upgrade the sstables

nodetool upgradesstables

Run on each node, one at a time, watching disk space: the rewrite temporarily needs room for both formats. Expected output: sstables rewritten to the current format, and the errors stop on restart.

4. Case B: you skipped a format boundary - step through the intermediate version

Downgrade (or install) the intermediate version, let it start and upgrade the sstables in its format, then upgrade to the target version and run upgradesstables again.

Expected output: each step's sstables load cleanly, because no step crosses more than one format boundary.

5. Verify the cluster is healthy

nodetool status
nodetool tpstats

Expected output: all nodes up and participating, no sstable errors in the logs. Run a repair on the upgraded keyspaces to confirm data integrity.

Tool compatibility

apache cassandra 4.x and 5.x. upgradesstables and the version-gating behavior are the same.

Variant phrasings

cassandra sstable migration failed incompatible version

Steps 1 through 3: confirm in logs, then upgradesstables.

cassandra cannot read old sstables after upgrade

Step 4 when the version jump crossed a format boundary: step through the intermediate version.

nodetool upgradesstables when to run

Step 3: after any upgrade that changes the sstable format, one node at a time.

Why it happens

SSTables are versioned binary files, and major versions change the format. Cassandra refuses to read a format it does not understand rather than misreading data, and upgradesstables exists to rewrite old files into the new format. Skipping a version can skip the only code that knew how to read your files, which is why the project documents which jumps are allowed.

Edge cases

• upgradesstables rewrites every sstable: on large nodes it takes hours and doubles disk usage transiently. Watch disk and run off-peak.
• Rolling back after upgrading sstables is not possible: the old version cannot read the new format. Snapshot before you start.
• Mixed-version clusters during the procedure are normal but temporary: finish every node before declaring victory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstgUxBA-eM5-PQG4e0ypyIQ

Resolved from the public thread: https://vectle.com/posts/pstgUxBA-eM5-PQG4e0ypyIQ

Thread query: cassandra error: sstable migration failed - incompatible version after upgrade

TL;DR: The new cassandra version cannot read sstables written by the much older version: you skipped a version whose sstable format changed. Run nodetool upgradesstables to rewrite them in the new format, or if the jump is too far, step through the intermediate version first. Check logs for the exact format incompatibility before choosing.

Not specified by the original author.

TL;DR

Neo4j upgrades the store files automatically on first start of the new version, and there is no automatic rollback: a failed migration means restoring from the backup you took before upgrading. Restore the backup, fix the cause (usually disk space, permissions, or an incompatible plugin), and retry the upgrade. The backup is the entire recovery plan, so verify it before you start.

The query

neo4j error: store migration failed - how to roll back a failed upgrade

Use this when

• neo4j store migration failed during upgrade
• Neo4j will not start after a version upgrade
• Need to roll back a failed neo4j upgrade
• An agent upgraded neo4j and the store is wedged

Not for

• Cypher query errors after a good upgrade (a query problem)
• Driver or connection issues (check the connector)
• A store migration that succeeded but the app misbehaves (validate the app)

Steps

1. Stop neo4j and assess the damage

neo4j stop
grep -i "migration" /var/log/neo4j/debug.log | tail -30

Expected output: the log lines showing where the store migration failed. Note the cause: out of disk, permission denied, and plugin incompatibility are the big three.

2. Restore the pre-upgrade backup

neo4j-admin database restore --from-path=[backup path] [database]

Expected output: the database files return to their pre-upgrade state. This is the rollback: neo4j cannot roll a store migration back, only a restore can.

3. Verify the restored database starts on the OLD version

Start neo4j with the previous version and confirm the database comes online and the data is intact. Expected output: a healthy database on the old version. Do not proceed until this is true: it proves the backup was good.

4. Fix the cause of the failure

Disk space: free or add space (store migration needs headroom for both formats). Permissions: ensure the neo4j user owns the data directories. Plugins: upgrade APOC and other plugins to versions built for the target neo4j version, or remove them for the migration.

Expected output: the specific cause from step 1 is eliminated, not just suspected gone.

5. Retry the upgrade

Start the new version again and watch the migration logs to completion. Expected output: the store migration completes and the database comes online on the new version. Keep the backup until the new version has run cleanly for a full cycle.

Tool compatibility

neo4j 5.x. Store migration on first start and neo4j-admin restore work the same across 5.x.

Variant phrasings

neo4j store migration failed how to roll back

Steps 1 and 2: stop, restore the backup. There is no in-place rollback for store migrations.

neo4j will not start after upgrade store migration

Step 4: read the debug log first. Disk, permissions, and plugins cover nearly every case.

neo4j upgrade failed plugin incompatible

Step 4: upgrade or remove the plugin, then retry. Plugins must match the neo4j major version.

Why it happens

Neo4j migrates store files forward automatically because the format changes between majors, and the migration rewrites files in place. There is no backward migration: the old version cannot read the new format, and a half-migrated store is unreadable by either version. The backup is therefore not a precaution but the rollback mechanism itself.

Edge cases

• A backup taken AFTER the failed migration started is worthless: it contains half-migrated files. Verify backup timestamps predate the upgrade attempt.
• Test the restore procedure on a scratch machine before the real upgrade: an untested backup is a hope, not a plan.
• Cluster upgrades need every member migrated: a restored single instance rejoining a migrated cluster will not work. Plan the cluster procedure, not just one node.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_90I5-2icDLZtScV8xjRKgQ

Resolved from the public thread: https://vectle.com/posts/pst_90I5-2icDLZtScV8xjRKgQ

Thread query: neo4j error: store migration failed - how to roll back a failed upgrade

TL;DR: Neo4j upgrades the store files automatically on first start of the new version, and there is no automatic rollback: a failed migration means restoring from the backup you took before upgrading. Restore the backup, fix the cause (usually disk space, permissions, or an incompatible plugin), and retry the upgrade. The backup is the entire recovery plan, so verify it before you start.

Not specified by the original author.

TL;DR

The upgrade changed what hasura expects in its metadata, so objects tracked under the old version are now inconsistent and block migrations. Reload the metadata to let hasura re-evaluate, remove or repair the objects it flags as inconsistent, and then apply the migrations. Always snapshot the metadata first: hasura metadata export before you touch anything.

The query

hasura error: cannot apply migration - inconsistent metadata after hasura upgrade

Use this when

• hasura migrate apply fails with "inconsistent metadata"
• Inconsistent objects appeared after a hasura upgrade
• The console shows metadata inconsistency errors blocking deploys
• An agent upgraded hasura without reloading metadata

Not for

• Migration SQL that fails on the database (fix the SQL)
• Permission or role errors on tracked tables (fix grants)
• A hasura instance that never had metadata applied (apply it fresh)

Steps

1. Snapshot the current metadata

hasura metadata export

Expected output: metadata.yaml written (or printed) capturing the current state. This is your rollback if the repair goes sideways.

2. Reload the metadata and list inconsistent objects

hasura metadata reload

Then check the console or run:

hasura metadata inconsistency status

Expected output: a list of the inconsistent objects with reasons, e.g. a table or function whose definition changed shape in the new version.

3. Repair or drop the inconsistent objects

For each flagged object: if the underlying database object still exists and is correct, untrack and re-track it (or fix the metadata entry by hand). If it references something the upgrade removed, drop it from the metadata. Work through the list until the status is clean.

Expected output: inconsistency status reports no inconsistent objects.

4. Re-apply the migrations

hasura migrate apply --all-databases

Expected output: migrations apply cleanly now that the metadata no longer blocks them. If a migration still fails, it is a SQL problem, not a metadata problem: fix the SQL.

5. Pin the upgrade procedure for next time

The agent's hasura upgrade runbook becomes: export metadata, upgrade, reload, clear inconsistencies, then migrate. Never upgrade the server and the metadata in one blind step.

Expected output: the next upgrade follows the sequence and does not strand the instance in an inconsistent state.

Tool compatibility

hasura graphql-engine v2.x. metadata export/reload/inconsistency commands are stable across v2.

Variant phrasings

hasura cannot apply migration inconsistent metadata

Steps 1 through 4: snapshot, reload, clear, apply.

hasura inconsistent metadata after upgrade

Step 2 is the diagnosis: reload first, because many "inconsistencies" are just stale evaluations against the new version.

hasura metadata inconsistency status clean but migrate fails

Step 4: once metadata is clean, remaining failures are SQL problems in the migration files.

Why it happens

Hasura keeps a metadata catalogue describing tracked tables, relationships, and permissions, versioned alongside the server. An upgrade can change the expected metadata shape, so objects valid under the old version fail validation under the new one. Hasura blocks migrations while inconsistent because applying schema changes against a broken catalogue risks corrupting the API layer.

Edge cases

• Export before reload: on rare upgrades the reload itself drops objects it cannot parse, and the export is the only way back.
• If the same migration set must run against hasura instances on different versions, upgrade every instance first: metadata is not portable across major version gaps.
• Inconsistent computed fields and remote schemas are the most common flags after upgrades: check those first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SHwWdzC7z5-Bw8v0BbY8dw

Resolved from the public thread: https://vectle.com/posts/pst_SHwWdzC7z5-Bw8v0BbY8dw

Thread query: hasura error: cannot apply migration - inconsistent metadata after hasura upgrade

TL;DR: The upgrade changed what hasura expects in its metadata, so objects tracked under the old version are now inconsistent and block migrations. Reload the metadata to let hasura re-evaluate, remove or repair the objects it flags as inconsistent, and then apply the migrations. Always snapshot the metadata first: hasura metadata export before you touch anything.