`migrate` from gcs backend is broken (context canceled / timeout)

TL;DR

Run the migration with -log-level=debug and watch the last copied key to find the culprit.

Steps

1. What was reported:

Issue hashicorp/vault#22493 (closed, 21 comments): Describe the bug

2. When trying to migrate from GCS backend to some other backend (raft initially, but same issue happens with file), the migration process fails with context canceled after some time. When using the following configuration file :

storage_source "gcs" {
 bucket = "test-vault-migation"
}

storage_destination "file" {
 path = "data"
}
cluster_addr = "YOUR_THE_HOST loopback address:8201"

3. on a VM with enough CPU/RAM/disk, located in same region than bucket, the first migration run will fail after ~37.5k items synced (this is consistent over time / retry, they all...
4. What works:

If vault operator migrate dies with 'context canceled' partway through, it may be a single oversized entry: raft's default maxentrysize is 1MB and the migration swallows the real error. The reporter fixed it by finding and deleting the offending entry (the debug logs showed which key was copied last before the failure). Alternative: raise maxentrysize in the raft storage config. Run the migration with -log-level=debug and watch the last copied key to find the culprit.

When to use

You are seeing this: What was reported: Issue hashicorp/vault#22493 (closed, 21 comments): Describe the bug When trying to migrate from GCS backend to some other backend (raft initially, but same issue happens with file), the migration process fails with context canceled after... Use this skill when you run into "migrate from gcs backend is broken (context canceled / timeout)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• YOURTHEHOST

In the GitHub issue, the reporter asked: migrate from gcs backend is broken (context canceled / timeout). The working answer that came out of the thread: If vault operator migrate dies with 'context canceled' partway through, it may be a single oversized entry: raft's default maxentrysize is 1MB and the migration swallows the real error. The reporter fixed it by finding and deleting the offending entry (the debug logs showed which key was copied last before the failure). Alternative: raise maxentrysize in the raft storage config. Run the migration with -log-level=debug and watch the last copied key to find the culprit. Source: https://github.com/hashicorp/vault/issues/22493