Attach Lambda filesystems via file_system_names at launch, or lose the data

Attach Lambda filesystems via filesystemnames at launch, or lose the data

TL;DR

When launching via the Lambda API, pass filesystemnames in the launch request to attach persistent storage, there is no way to attach it afterwards. Organize with intent: datasets, checkpoints, models, and outputs live on /lambda/nfs/[name] so they survive termination, and [HOME]/... holds only throwaway working files.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

If your training script writes checkpoints to the home directory, a terminate wipes them.

Context: Web (skills-3 lambda-labs SKILL.md): documents how filesystem attach really works. Filesystems must be attached at instance launch time: via the console by selecting the filesystem when launching, or via the API by including filesystemnames in the launch request. Best practice layout: persistent data under /lambda/nfs/storage/ (datasets, checkpoints, models, outputs), local SSD [HOME]/... only for temporary working files.

Source: Web (skills-3 lambda-labs SKILL.md): documents how filesystem attach really works. Filesystems must be attached at instance launch time: via the console by selecting the filesystem when launching, or via the API by including filesystemnames in the launch request. Best practice layout: persistent data under /lambda/nfs/storage/ (datasets, checkpoints, models, outputs), local SSD [HOME]/... only for temporary working files.

Verified resolution: When launching via the Lambda API, pass filesystemnames in the launch request to attach persistent storage, there is no way to attach it afterwards. Organize with intent: datasets, checkpoints, models, and outputs live on /lambda/nfs/[name] so they survive termination, and [HOME]/... holds only throwaway working files. If your training script writes checkpoints to the home directory, a terminate wipes them.

Source URL: https://github.com/totnormal/skills-3/blob/HEAD/AI-Research-SKILLs/09-infrastructure/lambda-labs/SKILL.md