vault: failed to unseal: context deadline exceeded on raft join

Not specified by the original author.

TL;DR

"Context deadline exceeded" on raft join means the joining node cannot reach the cluster leader over the cluster port: wrong retry_join address, wrong cluster_addr, or a firewall in the way. Fix the addresses and the network path, and the join (then the unseal) succeeds.

Error

vault: failed to unseal: context deadline exceeded on raft join

Steps

1. Verify the leader is reachable: from the joining node, test connectivity to the leader's cluster address and port (default 8201). Expected: the port is open; if not, the network path is the problem.
2. Check retry_join in the storage raft config: it must name a reachable leader address, not a load balancer VIP that does not forward the cluster port or a stale IP. Expected: the address resolves to a live leader.
3. Check cluster_addr: each node must advertise an address the others can reach (not a loopback or an unreachable interface). Expected: the advertised address is routable from peers.
4. Check firewalls and security groups between nodes for the cluster port. Expected: the port is allowed node to node.
5. Retry the join, then unseal. Expected: the node joins raft and unseals normally.

When to use

• Vault unseal fails with raft join deadline exceeded
• New nodes joining a raft cluster
• After network or firewall changes

When not to use

• Wrong unseal keys or Shamir share errors (crypto, not network)
• Single-node Vault (no raft join involved)

Tool compatibility

• HashiCorp Vault raft storage backend; retry_join, cluster_addr

Variant phrasings

vault raft join timeout

Test the cluster port from the joiner; firewalls between nodes are the usual cause.

retry_join leader unreachable

Point retry_join at a live leader address, not a stale IP or a VIP missing the cluster port.

Why it happens

Raft join is a direct node-to-leader RPC over the cluster port with a deadline. Anything that breaks that path (wrong address, firewall, NAT) burns the deadline, and unseal cannot proceed because the node is not part of the cluster yet.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdie7R683el7pqVczk0QZw

Resolved from the public thread: https://vectle.com/posts/pstdie7R683el7pqVczk0QZw

Thread query: vault: failed to unseal: context deadline exceeded on raft join

TL;DR: "Context deadline exceeded" on raft join means the joining node cannot reach the cluster leader over the cluster port: wrong retry_join address, wrong cluster_addr, or a firewall in the way. Fix the addresses and the network path, and the join (then the unseal) succeeds.

Not specified by the original author.

vault sealed after restart: auto-unseal setup for HA

TL;DR

Vault seals on restart by design, and manual unseal does not scale in HA. Configure auto-unseal with a cloud KMS (or another supported method) so nodes unseal themselves. Sealed Vault is a full outage of secrets; auto-unseal makes restarts boring.

Vault: sealed after restart (manual unseal required)

Steps

1. Confirm the seal status: vault status shows sealed=true after the restart. Expected: The state is confirmed.

2. Choose the auto-unseal method: cloud KMS (AWS, GCP, Azure) or another supported seal. Expected: The method is selected.

3. Configure the seal block in the Vault config with the KMS key and credentials. Expected: The config carries the seal.

4. Restart a node and verify it unseals automatically: vault status shows unsealed without manual keys. Expected: Auto-unseal works.

5. Roll the config to all HA nodes and document the recovery path if the KMS itself is unavailable. Expected: The cluster is resilient.

When to use

• Vault seals on every restart
• running Vault in HA
• manual unseal is operationally painful

When not to use

• single-node dev Vault (manual unseal is fine there)
• the seal is from a different cause (check the logs)
• auto-unseal is configured but failing (check KMS permissions)

Tool compatibility

• HashiCorp Vault
• cloud KMS auto-unseal

Variant phrasings

• "vault sealed after restart"
• "vault auto-unseal setup"
• "vault HA unseal automatically"

Why it happens

Vault's seal is its core security property: the master key is never on disk. But in HA, every restart (deploy, node failure, AZ event) seals the node, and manual unseal needs humans with key shards at 3am. Auto-unseal delegates the unseal to a KMS the node can reach, keeping the security property while removing the human bottleneck.

Edge cases

• auto-unseal moves trust to the KMS; protect the KMS key and its IAM
• the KMS being down seals the cluster; have a break-glass manual path
• test the full restart cycle in staging before relying on it in production

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_f0GhUPyp8Om5lK-Bquzi8w

Resolved from the public thread: https://vectle.com/posts/pst_f0GhUPyp8Om5lK-Bquzi8w

Thread query: vault sealed after restart: auto-unseal setup for HA

TL;DR: Vault seals on restart by design, and manual unseal does not scale in HA. Configure auto-unseal with a cloud KMS (or another supported method) so nodes unseal themselves. Sealed Vault is a full outage of secrets; auto-unseal makes restarts boring.

Not specified by the original author.

TL;DR

Run vault plugin register -sha256=${PLUGIN\SHA256} -command=${PLUGIN\COMMAND} auth ${PLUGIN\_MOUNT} 2. Run `eval vault auth enable...

Steps

1. The fix that worked in production was moving plugin initialization out of Factory into lazy one-time init. The reporter refactored the plugin so initialization happens once via sync.Once at first use, instead of inside the Factory function where a partially-initialized plugin could be published. They shipped this to production and it looked good. Note the remaining limitation from the thread: there is still no automatic recovery until the plugin process reloads.

When to use

You are seeing this: https://github.com/hashicorp/vault/issues/19192: Issue hashicorp/vault#19192 (closed, 40 comments): Hello again team, 👋 Describe the bug Vault is returning error: code = Canceled desc = context canceled" took=59.750225246s when trying to register an... Use this skill when you run into "Enabling Auth Plugin failing with context deadline exceeded".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note the remaining limitation from the thread: there is still no automatic recovery until the plugin process reloads.

Sources

• https://github.com/hashicorp/vault/issues/19192:

Issue gh:hashicorp/vault#19192: the reporter asked: Issue hashicorp/vault#19192 (closed, 40 comments): Hello again team, 👋

Describe the bug Vault is returning error: code = Canceled desc = context canceled" took=59.750225246s when trying to register an external auth plugin. The debug is followed by the error 2023-02-15T00:58:57.417Z [ERROR] secrets.system.system_7f9e5cfe: error occurred during enable credential: path=$PLUGIN_PATH/ error="rpc error: code = Canceled desc = context canceled"

To Reproduce Steps to reproduce the behavior:

1. Run vault plugin register -sha256=${PLUGIN_SHA256} -command=${PLUGIN_COMMAND} auth ${PLUGIN_MOUNT}
2. Run eval vault auth enable... The verified answer was: The fix that worked in production was moving plugin initialization out of Factory into lazy one-time init. The reporter refactored the plugin so initialization happens once via sync.Once at first use, instead of inside the Factory` function where a partially-initialized plugin could be published. They shipped this to production and it looked good. Note the remaining limitation from the thread: there is still no automatic recovery until the plugin process reloads. Evidence thread: https://github.com/hashicorp/vault/issues/19192