Lambda GPU cloud: anything outside the persistent filesystem is wiped

TL;DR

Keep data transfer in the same region as the filesystem to avoid slow transfers. Billing stops only when you terminate the instance, and the instance home directory is wiped on terminate.

Fix

1. Keep data transfer in the same region as the filesystem to avoid slow transfers. Expected: You get the expected result; the problem is gone.

When to use

• You are setting up or using this Lambda GPU cloud feature.
• The symptom matches: anything outside the persistent filesystem is wiped.

When NOT to use

• Unrelated Lambda GPU cloud issues (different feature, different failure).
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported against Lambda GPU cloud. Source: https://github.com/majiayu000/hermes-agent/blob/HEAD/optional-skills/mlops/lambda-labs/SKILL.md.

Variant phrasings

anything outside the persistent filesystem is wiped

Why it happens

Terminating a Lambda instance deletes the local disk. Checkpoints must live on the filesystem.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Source

https://github.com/majiayu000/hermes-agent/blob/HEAD/optional-skills/mlops/lambda-labs/SKILL.md

Source: Terminating a Lambda instance deletes the local disk. Checkpoints must live on the filesystem.

Verified resolution: [lambda-labs skill]: billing stops only when you terminate the instance, and the instance home directory is wiped on terminate. Write checkpoints, datasets, and model outputs to the persistent filesystem (mounted under the lambda NFS path) or they are gone. There is no auto-stop: an idle instance bills until you terminate it. Keep data transfer in the same region as the filesystem to avoid slow transfers.

Source URL: https://github.com/majiayu000/hermes-agent/blob/HEAD/optional-skills/mlops/lambda-labs/SKILL.md