NGC Nvcr.io/nvidia Docker images that need corrections

whitesscott · August 9, 2025, 9:49pm

https://gitlab.com/nvidia/container-images/cuda

This issue has been fixed cuda/dist/12.9.* and cuda/dist/13.0.0 Dockerifles

See attachment FilesThatNeedtobeCorrected.txt (136.3 KB)

in the event you are still building docker images for any of the earlier cuda
11.4.3 11.5.2 11.6.2 11.7.1 11.8.0 12.0.1 12.1.1 12.2.2 12.3.2 12.4.1 12.5.0 12.5.1 12.6.3 12.8.0 12.8.1 .
it still needs to be fixed.

I encountered the error in the below nvcr.io/nvidia docker images. And just checked them; those are the nvcr.io/nvidia docker images I have used. The last image I checked was Il4t-cuda:12.6.11-runtime. Remembered I’d found the gitlab/cuda repo about a year ago and just git pull to update it. Hence this post.

>grep “/usr/local/nvidia” --include=*.Dockerfile -irHn

grep "/usr/local/nvidia" --include=*.j2 -irHn

templates/ubuntu/base-dockerfile.j2:87:RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
templates/ubuntu/base-dockerfile.j2:88:    && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
templates/ubuntu/base-dockerfile.j2:90:ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
templates/ubuntu/base-dockerfile.j2:91:ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64

templates/redhat/base-dockerfile.j2:70:RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
templates/redhat/base-dockerfile.j2:71:    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
templates/redhat/base-dockerfile.j2:73:ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
templates/redhat/base-dockerfile.j2:74:ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64


nvcr.io/nvidia/l4t-cuda:12.6.11-runtime 
/etc/ld.so.conf.d/nvidia-tegra.conf:/usr/lib/aarch64-linux-gnu/tegra nonexistant
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 nonexistant
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib  nonexistant
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib64  nonexistant
PATH=/usr/local/nvidia/bin:

nvcr.io/nvidia/tritonserver:25.07-trtllm-python-py3
cat /etc/ld.so.conf.d/nvidia.conf  
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib64
That should point to /usr/local/cuda/lib64 and /usr/local/cuda/targets/sbsa-linux/lib/
PATH has /usr/local/nvidia/bin which also should point to /usr/local/cuda/bin

nvcr.io/nvidia/pytorch:25.06-py3-igpu
/etc/ld.so.conf.d/nvidia.conf has /usr/local/nvidia/lib /usr/local/nvidia/lib64
/etc/ld.so.conf.d/nvidia-tegra.conf has nonexistent directory /usr/lib/aarch64-linux-gnu/tegra
PATH=/usr/local/nvidia  should be removed from path.

nvcr.io/nvidia/tritonserver:25.05-vllm-python-py3
Fix /etc/ld.so.conf.d/nvidia.conf as it has nonexistent directories.
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib64
wrong PATH=/usr/local/nvidia/bin

nvcr.io/nvidia/deepstream-l4t:7.0-triton-multiarch
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib64
PATH=/usr/local/nvidia/bin

nvcr.io/nvidia/deepstream-l4t:7.0-triton-multiarch 
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib
/etc/ld.so.conf.d/nvidia.conf:/usr/local/nvidia/lib64

AastaLLL · August 12, 2025, 5:31am

Hi,

Could you share which command you tested so we can know more about the issue?

Thanks.

whitesscott · August 12, 2025, 7:00am

Subsequent to above post I entered a gitlab issue nvidia/container-images/cuda/-/issues/231
so should I mark this as solved assuming they will fix it if it needs to be fixed?

I was creating a Dockerfile and build for Jetson AGX Orin with base image nvcr.io/nvidia/pytorch:25.06-py3-igpu. That is Ubuntu 24.04, Jetpack L4T 36.4.4, Cuda 12.9. I figured it was worth the effort of a couple of days presuming Jetpack 7 / Thor will be Ubuntu 24.04. That docker build required in docker compilation of Triton, torch2trt, visual-perception-engine and installing ROS2 Jazzy, and multiple extra python pip and apt/deb packages.

I think it was compilation of Triton that failed with compilation and linker errors. So I examined environment and /etc/ld.so.conf.d/*.conf and noticed the incorrect paths so added this to Dockerfile and also modified PATH and LD_LIBRARY_PATH.


# -- Fix base image
RUN echo "/usr/local/cuda/lib64" > /etc/ld.so.conf.d/nvidia.conf && \
   echo "/usr/lib/aarch64-linux-gnu/nvidia" >> /etc/ld.so.conf.d/nvidia.conf && \
    rm -f /etc/ld.so.conf.d/nvidia-tegra.conf && \
    ldconfig

After I’d completed a build of the docker image I thought remembered some forums.developer.nvidia posts about NGC images where the poster thought nvcc wasn’t installed; but it ultimately was a PATH problem. To be thorough I ran the catalog.ngc.nvidia docker images I had previously pulled and discovered they all had similar PATH and LD_LIBRARY_PATH issuses.

whitesscott · August 14, 2025, 1:11am

I thought the issue had been fixed for the Cuda 12.9 and 13.0 docker images and all the images that are based on them; apparently not.

ngc registry image info nvcr.io/nvidia/cuda:13.0.0-base-ubuntu24.04

 Image Information
   Name: nvidia/cuda:13.0.0-base-ubuntu24.04
   Architecture: arm64
     Image Size: 126.88 MB
     Digest: sha256:c33da1d7a948e960842549e344af32eed7d8be20d1922ff99b52c119b08c72a7
   Architecture: amd64
     Image Size: 132.89 MB
     Digest: sha256:cf5ab24c3f5040a0ea5931658874c2158e606bf740f006809aa4bb9d3334016b
   Schema Version: 1
   Signed?: True
   Last Updated: Aug 12, 2025

cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf
/usr/local/lib/x86_64-linux-gnu nonexistent directory

env Note the /nvidia/ paths don’t exist.
CUDA_VERSION=13.0.0
NVARCH=x86_64
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64
PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

whitesscott · August 14, 2025, 3:08am

I’ll close this issue as nvidia ‘6 hours ago’ added this commit to

nvidia/container-images/cuda/dist/13.0.0/*/base/Dockerfile.

-ENV PATH=/usr/local/cuda/bin:${PATH}
-ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64
+ENV PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
+ENV LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64

system · September 10, 2025, 1:30am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
My developed Docker works properly on jetpack 4.2.1 but does not work on jetpack 4.4 Jetson Nano docker	2	2731	October 18, 2021
cuda 8.0 in docker: nvcc not found CUDA Setup and Installation	1	7597	June 30, 2017
Course-Getting Started eith AI on jetson Nano-Image Classification Jetson Nano cuda	6	1046	October 15, 2021
Cicc not in nvcr.io/nvidia/l4t-base:r32.3.1 Docker and NVIDIA Docker	20	2029	June 29, 2020
Suggestion to solve Tegra Nvidia-docker issues Jetson Nano cuda , docker	20	4199	October 15, 2021
L4t-base:r35.1.0 cannot find /usr/local/cuda directory Jetson AGX Orin tensorrt , cuda , ubuntu , jetson-inference	6	1438	December 21, 2022
L4t-tensorflow:r32.4.3-tf2.2-py3 cuda linker path problem Jetson Nano tensorflow , nvbugs	4	753	August 18, 2020
Jetson docker image extension Jetson TX2 docker	9	2219	October 18, 2021
Missing library libnvbuf_utils.so.1.0.0 when running docker container Jetson AGX Xavier docker	4	2732	October 18, 2021
Accessing the GPU from docker on L4T R32.1 Jetson TX2	7	1509	October 18, 2021

NGC Nvcr.io/nvidia Docker images that need corrections

Related topics