error while loading shared libraries: libcudart.so.13: cannot open shared object file

Not specified by the original author.

Short answer: find where libcudart.so.13 actually lives on the machine, then put that directory on the linker search path, or install the matching CUDA 13 runtime. Do not symlink a different major version in its place. `` error while loading shared libraries: libcudart.so.13: cannot open shared object file: No such file or directory 1. Locate the library: find / -name "libcudart.so*" 2>/dev/null. Two useful outcomes: a path under a CUDA install (for example the cuda lib64 directory) or a path inside your Python env (site-packages/nvidia/cu13/lib). If the search returns nothing, the runtime is genuinely absent; go to step 3. 2. If it exists, expose it to the linker for the current shell: export LD_LIBRARY_PATH with that directory prepended, then rerun the program. Expected: the program starts, or ldd YOURBINARY | grep cudart now resolves to a real path instead of not found. For system CUDA installs, adding the cuda lib64 directory to a file under /etc/ld.so.conf.d and running sudo ldconfig makes it permanent. 3. If it does not exist, install the CUDA 13 runtime: the cuda-toolkit-13 packages from NVIDIA's repo, or inside a Python-only stack python -m pip install nvidia-cuda-runtime-cu13 (then still apply step 2 to its lib directory). Expected: the find from step 1 now returns a libcudart.so.13 path. 4. Verify versions line up: nvidia-smi` shows the driver's maximum CUDA version (13.0 or higher is needed for a CUDA 13 runtime) and the failing program's expected runtime. A driver too old for CUDA 13 cannot be fixed by paths; update the driver or rebuild/reinstall the program against CUDA 12. Root cause: the linker only searches its configured paths (ld config, LDLIBRARY_PATH, standard lib dirs). CUDA runtimes installed by pip wheels live outside all of them, and a runtime of the wrong major version is invisible in a different way: present, but never a match for the .so.13 name the binary requests. When to use: Linux binaries or Python GPU packages (vLLM, PyTorch extensions, TensorRT tools) failing at startup with this exact loader error. When NOT to use: if ldd resolves cudart but the program then fails on a GLIBC version message, that is a libc mismatch, not a search-path problem. In Docker, fix the image (use a cuda runtime base image or install the runtime in the Dockerfile) instead of exporting paths at container start. Variant phrasings: "importerror libcudart.so.13 cannot open shared object file", "vllm libcudart.so.13 not found", "libcudart.so.13 missing ubuntu". Edge cases: conda environments put the runtime under the env's lib directory; the same export works with that path. Never copy or symlink a libcudart from another CUDA major version: the loader will accept the name and the program will fail later with undefined symbols.

Context: A binary linked against the CUDA 13 runtime starts, the dynamic linker looks for libcudart.so.13, and it is either not installed or not in any directory the linker searches. The .so.13 suffix is a major-version contract: a CUDA 12 runtime does not satisfy it. The current twist is that pip-installed GPU stacks (vLLM, PyTorch CUDA wheels) often ship this library inside the Python environment under site-packages/nvidia, a directory the system linker never searches by default, so the file exists on disk while the error insists it cannot be found.

Source: A binary linked against the CUDA 13 runtime starts, the dynamic linker looks for libcudart.so.13, and it is either not installed or not in any directory the linker searches. The .so.13 suffix is a major-version contract: a CUDA 12 runtime does not satisfy it. The current twist is that pip-installed GPU stacks (vLLM, PyTorch CUDA wheels) often ship this library inside the Python environment under site-packages/nvidia, a directory the system linker never searches by default, so the file exists on disk while the error insists it cannot be found.

Verified resolution: Short answer: find where libcudart.so.13 actually lives on the machine, then put that directory on the linker search path, or install the matching CUDA 13 runtime. Do not symlink a different major version in its place. `` error while loading shared libraries: libcudart.so.13: cannot open shared object file: No such file or directory 1. Locate the library: find / -name "libcudart.so*" 2>/dev/null. Two useful outcomes: a path under a CUDA install (for example the cuda lib64 directory) or a path inside your Python env (site-packages/nvidia/cu13/lib). If the search returns nothing, the runtime is genuinely absent; go to step 3. 2. If it exists, expose it to the linker for the current shell: export LD_LIBRARY_PATH with that directory prepended, then rerun the program. Expected: the program starts, or ldd YOURBINARY | grep cudart now resolves to a real path instead of not found. For system CUDA installs, adding the cuda lib64 directory to a file under /etc/ld.so.conf.d and running sudo ldconfig makes it permanent. 3. If it does not exist, install the CUDA 13 runtime: the cuda-toolkit-13 packages from NVIDIA's repo, or inside a Python-only stack python -m pip install nvidia-cuda-runtime-cu13 (then still apply step 2 to its lib directory). Expected: the find from step 1 now returns a libcudart.so.13 path. 4. Verify versions line up: nvidia-smi` shows the driver's maximum CUDA version (13.0 or higher is needed for a CUDA 13 runtime) and the failing program's expected runtime. A driver too old for CUDA 13 cannot be fixed by paths; update the driver or rebuild/reinstall the program against CUDA 12. Root cause: the linker only searches its configured paths (ld config, LDLIBRARY_PATH, standard lib dirs). CUDA runtimes installed by pip wheels live outside all of them, and a runtime of the wrong major version is invisible in a different way: present, but never a match for the .so.13 name the binary requests. When to use: Linux binaries or Python GPU packages (vLLM, PyTorch extensions, TensorRT tools) failing at startup with this exact loader error. When NOT to use: if ldd resolves cudart but the program then fails on a GLIBC version message, that is a libc mismatch, not a search-path problem. In Docker, fix the image (use a cuda runtime base image or install the runtime in the Dockerfile) instead of exporting paths at container start. Variant phrasings: "importerror libcudart.so.13 cannot open shared object file", "vllm libcudart.so.13 not found", "libcudart.so.13 missing ubuntu". Edge cases: conda environments put the runtime under the env's lib directory; the same export works with that path. Never copy or symlink a libcudart from another CUDA major version: the loader will accept the name and the program will fail later with undefined symbols.

Source URL: https://codechefvaibhavkashyap.medium.com/deploying-vllm-on-a-gcp-deep-learning-vm-my-real-world-journey-with-nvidia-l4-cuda-and-gemma-31b-464c94019f5a

Not specified by the original author.

How to fix OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory

Your torch build wants CUDA 11.0 but the loader cannot find its libraries. Either install the matching CUDA toolkit (or set LDLIBRARYPATH to its lib64 dir), or switch to the CPU-only torch wheel if you do not need the GPU. The version in the filename must match the CUDA your torch was built against.

The exact error

OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory

Fix it

1. Check what CUDA your torch expects: look at the +cuXX tag in python -m pip show torch. Expected: e.g. cu110, cu118, cu121

2. Option A (keep GPU): install that CUDA toolkit, or point the loader at it: export LD_LIBRARY_PATH=/usr/local/cuda-11.0/lib64:$LD_LIBRARY_PATH. Expected: import torch no longer raises the OSError

3. Option B (no GPU needed): pip install torch --index-url https://download.pytorch.org/whl/cpu. Expected: torch imports and runs on CPU

When this applies

import torch (or torchvision/torchgeometric) fails with libcudart.so.X / libcudnn / `libtorchcuda` not found on Linux

When this does NOT apply

Windows DLL errors or No module named 'torch'

Versions

Linux, CUDA builds of torch

Why it happens

CUDA torch wheels still resolve some CUDA libs from the system; a version mismatch or missing toolkit breaks the load.

Edge cases

nvidia-smi showing CUDA 12.x does not mean the toolkit is installed; the driver version and toolkit version are different things.

Asked about: OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory (web:iopenv torch dependency notes). The verified fix: Your torch build wants CUDA 11.0 but the loader cannot find its libraries. Either install the matching CUDA toolkit (or set LDLIBRARYPATH to its lib64 dir), or switch to the CPU-only torch wheel if you do not need the GPU. The version in the filename must match the CUDA your torch was built against. Full thread: https://www.iopenv.com/36E4IQF8I/Program-Error-Torch-Torch-Geometric-Torch-Dependency-issues-with-versions-such-as-sparse

Not specified by the original author.

TL;DR

On an HPC system the usual reason one job sees the library and another does not is the module system or scheduler environment: some jobs inherit CUDA library paths and some do not, or bind-mounted paths resolve differently depending on what the scheduler exposes. Log the environment inside every job before the singularity call: loaded modules, CUDA-related variables, the library search path, and whether libcudart.so.9.1 resolves. The intermittency is the diagnosis.

The intermittency is the diagnosis. Same compute node, same subject, sometimes passes and sometimes fails: that rules out the node hardware, the container image, and the data. What differs between two submissions of the same job is the job environment.

dwifslpreproc selected the CUDA eddy build (eddy_cuda9.1), which needs the CUDA 9.1 runtime library present when it starts. On an HPC system the usual reason one job sees the library and another does not is the module system or scheduler environment: some jobs inherit CUDA library paths and some do not, or bind-mounted paths resolve differently depending on what the scheduler exposes. Rerunning "fixes" it only because the resubmitted job lands in a luckier environment.

What to do:

1. Log the environment inside every job before the singularity call: loaded modules, CUDA-related variables, the library search path, and whether libcudart.so.9.1 resolves. Keep these logs.
2. Diff a failed job's environment against a successful one. The difference is the root cause; expect it to be a missing or different library path entry.
3. Make the eddy choice deterministic. Either force the CPU eddy build on CPU-only nodes so the pipeline never attempts the CUDA binary, or bind-mount a matching CUDA 9.1 library directory into the container and set the in-container library path identically for every job. A deterministic slow pipeline beats a fast flaky one.

Note the launch flags: with --containall the container sees almost nothing from the host except what is explicitly bound, so any CUDA library the binary needs must arrive through a bind mount or exist in the image, identically on every submission.

Context: Thread: Micapipe dMRI processing fails for some subjects, dwifslpreproc and libcudart.so.9.1 issue. Running micapipe 0.2.3 on Rocky Linux 8.10 via Singularity (singularity run --writable-tmpfs --containall, with many bind mounts for data, work dir, a custom dwifslpreproc, mrtrix config, ANTs, and the FreeSurfer license). Structural processing is fine; dMRI fails for some subjects with "eddy_cuda9.1: error while loading shared libraries: libcudart.so.9.1: cannot open shared object file", then dwifslpreproc errors out. There is no difference in acquisitions. Each subject runs on a different CPU node of an HPC system, but the reporter later confirmed it is not a node issue: on the same node some runs succeed and others fail, and a given subject sometimes processes fine. Rerunning structural then dMRI often makes it work. The reporter wants the root cause, not another rerun loop.

When to use

• You hit this exact issue.
• The symptom matches: Diagnosing intermittent eddy_cuda libcudart failures in containerized dMRI pipelines on HPC.

When NOT to use

• A different error or symptom; the cause here is specific to this one.

Compatibility

Per the linked source (mentions 0.2.3, 8.10, 9.1).

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Thread: Micapipe dMRI processing fails for some subjects, dwifslpreproc and libcudart.so.9.1 issue. Running micapipe 0.2.3 on Rocky Linux 8.10 via Singularity (singularity run --writable-tmpfs --containall, with many bind mounts for data, work dir, a custom dwifslpreproc, mrtrix config, ANTs, and the FreeSurfer license). Structural processing is fine; dMRI fails for some subjects with "eddy_cuda9.1: error while loading shared libraries: libcudart.so.9.1: cannot open shared object file", then dwifslpreproc errors out. There is no difference in acquisitions. Each subject runs on a different CPU node of an HPC system, but the reporter later confirmed it is not a node issue: on the same node some runs succeed and others fail, and a given subject sometimes processes fine. Rerunning structural then dMRI often makes it work. The reporter wants the root cause, not another rerun loop. Resolved: The intermittency is the diagnosis. Same compute node, same subject, sometimes passes and sometimes fails: that rules out the node hardware, the container image, and the data. What differs between two submissions of the same job is the job environment.

dwifslpreproc selected the CUDA eddy build (eddy_cuda9.1), which needs the CUDA 9.1 runtime library present when it starts. On an HPC system the usual reason one job sees the library and another does not is the module system or scheduler environment: some jobs inherit CUDA library paths and some do not, or bind-mounted paths resolve differently depending on what the scheduler exposes. Rerunning "fixes" it only because the resubmitted job lands in a luckier environment.

What to do:

1. Log the environment inside every job before the singularity call: loaded modules, CUDA-related variables, the library search path, and whether libcudart.so.9.1 resolves. Keep these logs.
2. Diff a failed job's environment against a successful one. The difference is the root cause; expect it to be a missing or different library path entry.
3. Make the eddy choice deterministic. Either force the CPU eddy build on CPU-only nodes so the pipeline never attempts the CUDA binary, or bind-mount a matching CUDA 9.1 library directory into the container and set the in-container library path identically for every job. A deterministic slow pipeline beats a fast flaky one.

Note the launch flags: with --containall the container sees almost nothing from the host except what is explicitly bound, so any CUDA library the binary needs must arrive through a bind mount or exist in the image, identically on every submission.