CUDA driver version is insufficient for CUDA runtime version

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
, ,
1

I am using Jetson AGX Orin and build a container with Ubuntu22.04 and CUDA toolkit 12.5.
I choose a base image with CUDA 12.2 and update its version manually in the dockerfile.

Below is my Dockerfile.
But it keep saying “CUDA driver version is insufficient for CUDA runtime version”, the same code base can be ran in x86 with CUDA toolkit 12.5.

Any solution for this?
Thanks

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Wed_Apr_17_19:34:47_PDT_2024
Cuda compilation tools, release 12.5, V12.5.40
Build cuda_12.5.r12.5/compiler.34177558_0


FROM nvcr.io/nvidia/l4t-cuda:12.2.12-runtime

# Install nvidia-l4t-core
RUN \
    echo "deb https://repo.download.nvidia.com/jetson/common r32.4 main" >> /etc/apt/sources.list && \
    echo "deb https://repo.download.nvidia.com/jetson/t194 r32.4 main" >> /etc/apt/sources.list && \
    apt-key adv --fetch-key http://repo.download.nvidia.com/jetson/jetson-ota-public.asc && \
    mkdir -p /opt/nvidia/l4t-packages/ && \
    touch /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall

RUN apt-get update \
    && echo "Y" | apt-get install -y --no-install-recommends nvidia-l4t-core

ENV UDEV=1

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb \
  && dpkg -i cuda-keyring_1.1-1_all.deb \
  && apt-get update \
  && apt-get -y install cuda-toolkit-12-5

# Install necessary dependencies including gcc
RUN apt-get update \
    && apt-get install -y wget build-essential git cmake libzmq3-dev pkg-config curl vim python3 python3-pip docker-compose ninja-build \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /

UPDATE:

docker run --gpus all -it jetsondev bash

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as ‘csv’
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.: unknown.

3

Hi,

JetPack 6 uses r36.3 BSP and the device should be t234.
Please change it accordingly and try it again:

    echo "deb https://repo.download.nvidia.com/jetson/common r36.3 main" >> /etc/apt/sources.list && \
    echo "deb https://repo.download.nvidia.com/jetson/t234 r36.3 main" >> /etc/apt/sources.list && \

Thanks.

4

I update the Dockerfile., and run with cmd:

docker run --rm -it -v /run/jtop.sock:/run/jtop.sock jetsondev bash

Still see the error:
Cuda failed with error code CUDA driver version is insufficient for CUDA runtime version

Is there anything I can check?
I believe that if the drivers are all compatible, it should be able to start normally in the container.

Can you just re-check my dockerfile?
Thanks!

FROM nvcr.io/nvidia/l4t-cuda:12.2.12-runtime

# Install nvidia-l4t-core
RUN \
    echo "deb https://repo.download.nvidia.com/jetson/common r36.3 main" >> /etc/apt/sources.list && \
    echo "deb https://repo.download.nvidia.com/jetson/t234 r36.3 main" >> /etc/apt/sources.list && \
    apt-key adv --fetch-key http://repo.download.nvidia.com/jetson/jetson-ota-public.asc && \
    mkdir -p /opt/nvidia/l4t-packages/ && \
    touch /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall

RUN apt-get update \
    && echo "Y" | apt-get install -y --no-install-recommends nvidia-l4t-core

ENV UDEV=1

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/arm64/cuda-keyring_1.1-1_all.deb \
  && dpkg -i cuda-keyring_1.1-1_all.deb \
  && apt-get update \
  && apt-get -y install cuda-toolkit-12-5

# Install necessary dependencies including gcc
RUN apt-get update \
    && apt-get install -y wget build-essential git cmake libzmq3-dev pkg-config curl vim python3 python3-pip docker-compose ninja-build \
    && rm -rf /var/lib/apt/lists/*

# Install jetson-stats and jtop
RUN pip3 install jetson-stats
5

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Is this still an issue to support? Any result can be shared?