Diagnosing intermittent eddy_cuda libcudart failures in containerized dMRI pipelines on HPC

TL;DR

On an HPC system the usual reason one job sees the library and another does not is the module system or scheduler environment: some jobs inherit CUDA library paths and some do not, or bind-mounted paths resolve differently depending on what the scheduler exposes. Log the environment inside every job before the singularity call: loaded modules, CUDA-related variables, the library search path, and whether libcudart.so.9.1 resolves. The intermittency is the diagnosis.

The intermittency is the diagnosis. Same compute node, same subject, sometimes passes and sometimes fails: that rules out the node hardware, the container image, and the data. What differs between two submissions of the same job is the job environment.

dwifslpreproc selected the CUDA eddy build (eddy_cuda9.1), which needs the CUDA 9.1 runtime library present when it starts. On an HPC system the usual reason one job sees the library and another does not is the module system or scheduler environment: some jobs inherit CUDA library paths and some do not, or bind-mounted paths resolve differently depending on what the scheduler exposes. Rerunning "fixes" it only because the resubmitted job lands in a luckier environment.

What to do:

1. Log the environment inside every job before the singularity call: loaded modules, CUDA-related variables, the library search path, and whether libcudart.so.9.1 resolves. Keep these logs.
2. Diff a failed job's environment against a successful one. The difference is the root cause; expect it to be a missing or different library path entry.
3. Make the eddy choice deterministic. Either force the CPU eddy build on CPU-only nodes so the pipeline never attempts the CUDA binary, or bind-mount a matching CUDA 9.1 library directory into the container and set the in-container library path identically for every job. A deterministic slow pipeline beats a fast flaky one.

Note the launch flags: with --containall the container sees almost nothing from the host except what is explicitly bound, so any CUDA library the binary needs must arrive through a bind mount or exist in the image, identically on every submission.

Context: Thread: Micapipe dMRI processing fails for some subjects, dwifslpreproc and libcudart.so.9.1 issue. Running micapipe 0.2.3 on Rocky Linux 8.10 via Singularity (singularity run --writable-tmpfs --containall, with many bind mounts for data, work dir, a custom dwifslpreproc, mrtrix config, ANTs, and the FreeSurfer license). Structural processing is fine; dMRI fails for some subjects with "eddy_cuda9.1: error while loading shared libraries: libcudart.so.9.1: cannot open shared object file", then dwifslpreproc errors out. There is no difference in acquisitions. Each subject runs on a different CPU node of an HPC system, but the reporter later confirmed it is not a node issue: on the same node some runs succeed and others fail, and a given subject sometimes processes fine. Rerunning structural then dMRI often makes it work. The reporter wants the root cause, not another rerun loop.

When to use

• You hit this exact issue.
• The symptom matches: Diagnosing intermittent eddy_cuda libcudart failures in containerized dMRI pipelines on HPC.

When NOT to use

• A different error or symptom; the cause here is specific to this one.

Compatibility

Per the linked source (mentions 0.2.3, 8.10, 9.1).

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Thread: Micapipe dMRI processing fails for some subjects, dwifslpreproc and libcudart.so.9.1 issue. Running micapipe 0.2.3 on Rocky Linux 8.10 via Singularity (singularity run --writable-tmpfs --containall, with many bind mounts for data, work dir, a custom dwifslpreproc, mrtrix config, ANTs, and the FreeSurfer license). Structural processing is fine; dMRI fails for some subjects with "eddy_cuda9.1: error while loading shared libraries: libcudart.so.9.1: cannot open shared object file", then dwifslpreproc errors out. There is no difference in acquisitions. Each subject runs on a different CPU node of an HPC system, but the reporter later confirmed it is not a node issue: on the same node some runs succeed and others fail, and a given subject sometimes processes fine. Rerunning structural then dMRI often makes it work. The reporter wants the root cause, not another rerun loop. Resolved: The intermittency is the diagnosis. Same compute node, same subject, sometimes passes and sometimes fails: that rules out the node hardware, the container image, and the data. What differs between two submissions of the same job is the job environment.

dwifslpreproc selected the CUDA eddy build (eddy_cuda9.1), which needs the CUDA 9.1 runtime library present when it starts. On an HPC system the usual reason one job sees the library and another does not is the module system or scheduler environment: some jobs inherit CUDA library paths and some do not, or bind-mounted paths resolve differently depending on what the scheduler exposes. Rerunning "fixes" it only because the resubmitted job lands in a luckier environment.

What to do:

1. Log the environment inside every job before the singularity call: loaded modules, CUDA-related variables, the library search path, and whether libcudart.so.9.1 resolves. Keep these logs.
2. Diff a failed job's environment against a successful one. The difference is the root cause; expect it to be a missing or different library path entry.
3. Make the eddy choice deterministic. Either force the CPU eddy build on CPU-only nodes so the pipeline never attempts the CUDA binary, or bind-mount a matching CUDA 9.1 library directory into the container and set the in-container library path identically for every job. A deterministic slow pipeline beats a fast flaky one.

Note the launch flags: with --containall the container sees almost nothing from the host except what is explicitly bound, so any CUDA library the binary needs must arrive through a bind mount or exist in the image, identically on every submission.