Trouble starting docker container with cuda after enabling MIG

I’m noticing a similar issue to steven6_wang (link here). I get “CUDA failed to initialize <…>”

Details:

I upgraded my thor from a flashed jetpack 7.0 to 7.2 using the new on board, no-flash, GUI updater. I followed the thor recommended profile 83 & 78 MIG profile setup and (see nvidia-smi dump below). (as an aside, I had to apply the fix from this other forum post -link here: tldr: delete /etc/modprobe.d/nvidia-unifiedgpudisp.conf)

Then, I run docker run --rm -it --runtime=nvidia --privileged nvcr.io/nvidia/tensorrt:25.09-py3 bash. (note - here, I used a different image than the original post. when I get a chance, I’ll run with the pytorch:26.07-py3 image too to match the earlier forum post) I ran the pytorch:26.07-py3 image, also with sudo, and I get the same error - with the only difference being that with the matching image, I get the same note about SHMEM.

The docker container starts, but I get “CUDA failed to initialize”. (not posted here - in my image using tensorrt:25.09-py3 as a base, where torch is installed, torch hits related cuda failure issues)

If I run nvidia-smi in the container, it seems to output the same info from the host (maybe because I’m using --privileged)

nvidia@nvidia:~$ docker run --rm -it --runtime=nvidia --privileged nvcr.io/nvidia/tensorrt:25.09-py3 bash

=====================
== NVIDIA TensorRT ==
=====================

NVIDIA Release 25.09 (build 207438969)
NVIDIA TensorRT Version 10.13.3
Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Container image Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

https://developer.nvidia.com/tensorrt

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/)
and the Product-Specific Terms for NVIDIA AI Products
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/).
To install the open-source samples corresponding to this TensorRT release version
run /opt/tensorrt/install_opensource.sh.  To build the open source parsers,
plugins, and samples for current top-of-tree on master or a different branch,
run /opt/tensorrt/install_opensource.sh -b <branch>
See https://github.com/NVIDIA/TensorRT for more information.

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize.  GPU functionality will not be available.
   [[ Initialization error (error 3) ]]

groups: cannot find name for group ID 993
root@cf6e0d440bff:/workspace#
root@cc8fd071fa0c:/workspace# nvidia-smi
Mon Sep 14 17:41:34 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.78                 Driver Version: 595.78         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Thor                    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   36C  N/A               5W /  N/A  | Not Supported          |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |              Shared Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                Shared BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    1   0   0  |                 N/A              | 12      0 |  1   1    1    0    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    2   0   1  |                 N/A              |  6      0 |  1   1    1    1    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

I also noticed if I add --user "$(id -u):$(id -g)" to my docker command, I also get a line saying " [[ Possible MIG misconfiguration detected ]]"

nvidia-smi from thor host (outside docker container)

nvidia@nvidia:~$ nvidia-smi
Mon Sep 14 10:34:42 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.78                 Driver Version: 595.78         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Thor                    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   36C  N/A               5W /  N/A  | Not Supported          |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                                                                                                                                                   +-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |              Shared Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                Shared BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    1   0   0  |                 N/A              | 12      0 |  1   1    1    0    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    2   0   1  |                 N/A              |  6      0 |  1   1    1    1    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
nvidia@nvidia:~$                                                                                                                                                                                                   

You could try:

nvidia-smi -L

# pick your mig instance
MIG_UUID=MIG-put-the-mig-UUID-you-want.

docker run --rm -it \
    --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES="$MIG_UUID" \
    -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
    nvcr.io/nvidia/tensorrt:26.05-py3


# If that doesn't work following will show what driver you may need in the docker container but tensorrt:26.05-py is problably equivilent to Thor.

nvidia-container-cli info
NVRM version:   595.78
CUDA version:   13.2

Device Index:   0
Device Minor:   0
Model:          NVIDIA Thor
Brand:          GeForce
GPU UUID:       GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600
Bus Location:   00000000:01:00.0
Architecture:   11.0

I haven’t tried setting NVIDIA_DRIVER_CAPABILITIES though so I’ll give that a shot. But otherwise, I was stuck after trying various combinations of -e NVIDIA_VISIBLE_DEVICES=<...> and --device=<...> that I saw in other forums. Any tips or examples to help clarify what the format/syntax of MIG_UUID should be?

Given:

nvidia@nvidia:~$ nvidia-smi  -L
GPU 0: NVIDIA Thor (UUID: GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600)
  MIG 2g.0gb      Device  0: (UUID: MIG-3a71187f-e460-5e39-8294-3de30227c005)

  MIG 1g.0gb      Device  1: (UUID: MIG-fb63de7b-c47b-5327-8d9d-6aaff7b5218e)
nvidia@nvidia:~$ 

I’ve tried:

  • -e NVIDIA_VISIBLE_DEVICES=MIG-3a71187f-e460-5e39-8294-3de30227c005 and
  • -e NVIDIA_VISIBLE_DEVICES=MIG-3a71187f-e460-5e39-8294-3de30227c005,MIG-fb63de7b-c47b-5327-8d9d-6aaff7b5218e
    
  • -e NVIDIA_VISIBLE_DEVICES="0:0"
  • -e NVIDIA_VISIBLE_DEVICES="0:0,0:1"

but I keep getting failed to construct device spec generators: unsupported device id: <...>

also, if it helps, my container cli info prints:

nvidia@nvidia:~$  nvidia-container-cli info
NVRM version:   595.78
CUDA version:   13.2

Device Index:   0
Device Minor:   0
Model:          NVIDIA Thor
Brand:          GeForce
GPU UUID:       GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600
Bus Location:   00000000:01:00.0
Architecture:   11.0

docker run --help |grep gpu
–gpus gpu-request GPU devices to add to the container

So try

--gpus 0

instead of -e NVIDIA_VISIBLE_DEVICES=“$MIG_UUID”

I just enabled mig on my Thor. Here’s what works for me now. Change the version of tensorrt:version to your version.

docker run --rm -it \
    --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=0 \
    -e CUDA_VISIBLE_DEVICES=MIG-your-mig-0  \
    --device=/dev/nvidia-caps/nvidia-cap21 \
    --device=/dev/nvidia-caps/nvidia-cap22 \
    nvcr.io/nvidia/tensorrt:26.08-py3

Hi,

Could you try a more recent container? For example, nvcr.io/nvidia/pytorch:26.07-py3.

Thanks.

hmm, still no dice, slightly different error messages. (by the way thank you guys for taking the time to help)

here’s what I’ve tried:

docker run --rm -it \                                                                                                                                                                                                 
    --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=0 \
    -e CUDA_VISIBLE_DEVICES=MIG-3a71187f-e460-5e39-8294-3de30227c005 \
    --device=/dev/nvidia-caps/nvidia-cap21 \
    --device=/dev/nvidia-caps/nvidia-cap22 \
    nvcr.io/nvidia/tensorrt:26.08-py3

<... == NVIDIA TensorRT..., GOVERNING TERMS, ...>

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize.  GPU functionality will not be available.
   [[ No CUDA-capable device is detected (error 100) ]]

# also tried substituting: nvcr.io/nvidia/pytorch:26.07-py3, and if I do that, I get the same error output plus this note:

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for PyTorch.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...
sudo docker run --rm -it --runtime=nvidia --privileged nvcr.io/nvidia/pytorch:26.07-py3
# (re-trying previous command with new image)

<... GOVERNING TERMS...>

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize.  GPU functionality will not be available.
   [[ Initialization error (error 3) ]]

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for PyTorch.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...
MIG_UUID=MIG-3a71187f-e460-5e39-8294-3de30227c005                                                                                                                                                               docker run --rm -it \
     --runtime=nvidia \
     -e NVIDIA_VISIBLE_DEVICES="$MIG_UUID" \
     -e NVIDIA_DRIVER_CAPABILITIES=compute,utility \
     nvcr.io/nvidia/pytorch:26.07-py3
# (re-trying previous command with new image)
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to generate CDI spec for mode "csv": failed to construct device spec generators: unsupported device id: MIG-3a71187f-e460-5e39-8294-3de30227c005

Run 'docker run --help' for more information

sudo apt install cuda-compat-13-4


cat mig-tensorrt.sh

#!/usr/bin/env bash

CUDA_VISIBLE_DEVICES=$(
    nvidia-smi -L |
    sed -n 's/.*Device[[:space:]]*0:.*UUID:[[:space:]]*\(MIG-[^)]*\)).*/\1/p'
)
docker run --rm -it \
    --runtime=nvidia \
    -e NVIDIA_VISIBLE_DEVICES=0 \
    -e CUDA_VISIBLE_DEVICES="$CUDA_VISIBLE_DEVICES" \
    --device=/dev/nvidia-caps/nvidia-cap12 \
    --device=/dev/nvidia-caps/nvidia-cap13 \
    nvcr.io/nvidia/tensorrt:26.08-py3

Hi,

Could you check if the Docker server works correctly in your environment first?

$ sudo systemctl status docker

Thanks

I did a reboot, reenabled mig, and checked docker status at each stage - the docker server seems to be active and healthy.

After fresh restart:

nvidia@nvidia:~$ nvidia-smi 
Thu Sep 17 13:25:11 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.78                 Driver Version: 595.78         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Thor                    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   35C  N/A               2W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3558      G   /usr/lib/xorg/Xorg                       92MiB |
|    0   N/A  N/A            3766      G   /usr/bin/gnome-shell                    110MiB |
+-----------------------------------------------------------------------------------------+

nvidia@nvidia:~$ sudo systemctl status docker
[sudo] password for nvidia: 
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-09-17 13:22:00 PDT; 2min 34s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1969 (dockerd)
      Tasks: 17
     Memory: 214.6M (peak: 215.8M)
        CPU: 1.190s
     CGroup: /system.slice/docker.service
             └─1969 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.550201526-07:00" level=info msg="Initializing buildkit"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.550885804-07:00" level=warning msg="failed check for fsverity support" error="enable fsverity failed: operation not supported" path=/var/lib/docke>
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.988371391-07:00" level=info msg="Completed buildkit initialization"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.994736558-07:00" level=info msg="Daemon has completed initialization"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.994810150-07:00" level=info msg="API listen on /run/docker.sock"
Sep 17 13:22:00 nvidia systemd[1]: Started docker.service - Docker Application Container Engine.
Sep 17 13:23:56 nvidia dockerd[1969]: time="2026-09-17T13:23:56.417839525-07:00" level=warning msg="Refreshing the CDI registry generated errors" error="failed to monitor for changes: no such file or directory" 
Sep 17 13:23:56 nvidia dockerd[1969]: time="2026-09-17T13:23:56.417892913-07:00" level=warning msg="Ignoring previous errors updating spec" error="CDI device injection failed: failed to inject devices: failed t>
Sep 17 13:23:57 nvidia dockerd[1969]: time="2026-09-17T13:23:57.272649646-07:00" level=info msg="sbJoin: gwep4 ''->'', gwep6 ''->''" eid=240837f79199 ep=a3r-nvidia-b532d936 net=host nid=b2522a1e810a
Sep 17 13:24:22 nvidia dockerd[1969]: time="2026-09-17T13:24:22.018129924-07:00" level=info msg="received task-delete event from containerd" container=da259154fc6e569acb01a0f2d0dc1129ce8c38c77afadfd59edb8764a28>

docker containers properly initialize cuda here

After isolating multiuser.target and enabling persistance mode and mig

sudo systemctl isolate multi-user.target
sudo nvidia-smi -pm 1
sudo nvidia-smi -mig 1
nvidia@nvidia:~$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-09-17 13:22:00 PDT; 6min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1969 (dockerd)
      Tasks: 17
     Memory: 214.6M (peak: 215.8M)
        CPU: 1.245s
     CGroup: /system.slice/docker.service
             └─1969 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.550201526-07:00" level=info msg="Initializing buildkit"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.550885804-07:00" level=warning msg="failed check for fsverity support" error="enable fsverity failed: operation not supported" path=/var/lib/docke>Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.988371391-07:00" level=info msg="Completed buildkit initialization"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.994736558-07:00" level=info msg="Daemon has completed initialization"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.994810150-07:00" level=info msg="API listen on /run/docker.sock"
Sep 17 13:22:00 nvidia systemd[1]: Started docker.service - Docker Application Container Engine.
Sep 17 13:23:56 nvidia dockerd[1969]: time="2026-09-17T13:23:56.417839525-07:00" level=warning msg="Refreshing the CDI registry generated errors" error="failed to monitor for changes: no such file or directory" 
Sep 17 13:23:56 nvidia dockerd[1969]: time="2026-09-17T13:23:56.417892913-07:00" level=warning msg="Ignoring previous errors updating spec" error="CDI device injection failed: failed to inject devices: failed t>Sep 17 13:23:57 nvidia dockerd[1969]: time="2026-09-17T13:23:57.272649646-07:00" level=info msg="sbJoin: gwep4 ''->'', gwep6 ''->''" eid=240837f79199 ep=a3r-nvidia-b532d936 net=host nid=b2522a1e810a
Sep 17 13:24:22 nvidia dockerd[1969]: time="2026-09-17T13:24:22.018129924-07:00" level=info msg="received task-delete event from containerd" container=da259154fc6e569acb01a0f2d0dc1129ce8c38c77afadfd59edb8764a28>lines 1-22/22 (END)                                                                                                                                                                                                nvidia 0:bash*Z                                                                                                                                                                           Thu 1:28:22 PM 2026-09-17

After enabling mig profiles

sudo nvidia-smi mig -cgi 83 -C
sudo nvidia-smi mig -cgi 78 -C
nvidia@nvidia:~$ nvidia-smi
Thu Sep 17 13:30:15 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.78                 Driver Version: 595.78         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Thor                    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   36C  N/A               5W /  N/A  | Not Supported          |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |              Shared Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                Shared BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    1   0   0  |                 N/A              | 12      0 |  1   1    1    0    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  0    2   0   1  |                 N/A              |  6      0 |  1   1    1    1    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
nvidia@nvidia:~$ nvidia-smi -L
GPU 0: NVIDIA Thor (UUID: GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600)
  MIG 2g.0gb      Device  0: (UUID: MIG-3a71187f-e460-5e39-8294-3de30227c005)
  MIG 1g.0gb      Device  1: (UUID: MIG-fb63de7b-c47b-5327-8d9d-6aaff7b5218e)
nvidia@nvidia:~$ sudo systemctl status docker
[sudo] password for nvidia: 
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-09-17 13:22:00 PDT; 8min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1969 (dockerd)
      Tasks: 17
     Memory: 214.6M (peak: 217.0M)
        CPU: 1.266s
     CGroup: /system.slice/docker.service
             └─1969 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.550201526-07:00" level=info msg="Initializing buildkit"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.550885804-07:00" level=warning msg="failed check for fsverity support" error="enable fsverity failed: operation not supported" path=/var/lib/docke>Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.988371391-07:00" level=info msg="Completed buildkit initialization"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.994736558-07:00" level=info msg="Daemon has completed initialization"
Sep 17 13:21:59 nvidia dockerd[1969]: time="2026-09-17T13:21:59.994810150-07:00" level=info msg="API listen on /run/docker.sock"
Sep 17 13:22:00 nvidia systemd[1]: Started docker.service - Docker Application Container Engine.
Sep 17 13:23:56 nvidia dockerd[1969]: time="2026-09-17T13:23:56.417839525-07:00" level=warning msg="Refreshing the CDI registry generated errors" error="failed to monitor for changes: no such file or directory"
Sep 17 13:23:56 nvidia dockerd[1969]: time="2026-09-17T13:23:56.417892913-07:00" level=warning msg="Ignoring previous errors updating spec" error="CDI device injection failed: failed to inject devices: failed t>Sep 17 13:23:57 nvidia dockerd[1969]: time="2026-09-17T13:23:57.272649646-07:00" level=info msg="sbJoin: gwep4 ''->'', gwep6 ''->''" eid=240837f79199 ep=a3r-nvidia-b532d936 net=host nid=b2522a1e810a
Sep 17 13:24:22 nvidia dockerd[1969]: time="2026-09-17T13:24:22.018129924-07:00" level=info msg="received task-delete event from containerd" container=da259154fc6e569acb01a0f2d0dc1129ce8c38c77afadfd59edb8764a28>lines 1-22/22 (END)

cuda fails to initialize now

I couldn’t get apt to install cuda-compat-13-4, but the mig-tensorrt.sh script above seems to work anyways.

I started just seeing this note, so maybe jetpack 7.2 already had some bundled compatibility

NOTE: CUDA Forward Compatibility mode ENABLED.

Playing around a bit with the previous command too, I think switching from

    --device=/dev/nvidia-caps/nvidia-cap21 \
    --device=/dev/nvidia-caps/nvidia-cap22 \

to

     --device=/dev/nvidia-caps/nvidia-cap12 \
     --device=/dev/nvidia-caps/nvidia-cap13 \

was what did it.

Thanks!

Follow up: I also played around with a smaller command and I don’t get any initialization errors if I try these:

docker run --rm -it \                                                                                                                                                                                                    
     --runtime=nvidia \
     --device=/dev/nvidia-caps/nvidia-cap12 \
     --device=/dev/nvidia-caps/nvidia-cap13 \
     nvcr.io/nvidia/pytorch:26.07-py3
# then in nvidia-smi I see only the first MIG device
+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |              Shared Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                Shared BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    1   0   0  |                 N/A              | 12      0 |  1   1    1    0    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+

docker run --rm -it \                                                                                                                                                                                                    
     --runtime=nvidia \
     --device=/dev/nvidia-caps/nvidia-cap21 \
     --device=/dev/nvidia-caps/nvidia-cap22 \
     nvcr.io/nvidia/pytorch:26.07-py3
# then in nvidia-smi I see only the second MIG device
# nvidia-smi outut <...>
+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |              Shared Memory-Usage |        Vol|        Shared         |
|      ID  ID  Dev |                Shared BAR1-Usage | SM     Unc| CE ENC  DEC  OFA  JPG |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  0    2   0   0  |                 N/A              |  6      0 |  1   1    1    1    1 |
|                  |                 N/A              |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
#

It seems that if I start to specify CUDA_VISIBLE_DEVICES with the wrong /dev/nvidia-caps/nvidia-cap* then I start to hit conflicts and get cuda initialization errors again. it seems that cap12/cap13 match with the first device MIG-317… and cap21/cap22 match with MIG-fb63… or both devices with a “;” separator and all the cap lines:

 docker run --rm -it \               
     --runtime=nvidia \              
     -e CUDA_VISIBLE_DEVICES="MIG-3a71187f-e460-5e39-8294-3de30227c005;MIG-fb63de7b-c47b-5327-8d9d-6aaff7b5218e" \
     --device=/dev/nvidia-caps/nvidia-cap12 \
     --device=/dev/nvidia-caps/nvidia-cap13 \
     --device=/dev/nvidia-caps/nvidia-cap21 \
     --device=/dev/nvidia-caps/nvidia-cap22 \
     nvcr.io/nvidia/pytorch:26.07-py3                                                                                                                                                                           

Also, so far, it seems that if I set NVIDIA_VISIBLE_DEVICES=<XX> where is anything other than 0 I get

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to generate CDI spec for mode "csv": failed to construct device spec generators: unsupported device id: <XX>

And, should I be concerned about this issue where I seem to see 0 MiB memory usage in nvidia-smi?

If I use the pytorch image in my short test and then try this short python -i test.py

import torch
test = torch.rand(10000).cuda()

then in nvidia-smi I see: 0MiB memory usage, but when I don’t turn on MIG mode, when using the same container I get non-zero output ~ 104MiB

# nvidia-smi either in host or in container outputs:
# <...> 
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0    1    0              193      C   python                                    0MiB |
+-----------------------------------------------------------------------------------------+

Excellent work on mig. Here’s script I’m using to enable it:

cat mig-setup.sh

#!/usr/bin/env bash
set -e

sudo nvidia-smi -pm 1
sudo nvidia-smi -mig 1

sudo nvidia-smi mig -cgi 83 -C
sudo nvidia-smi mig -cgi 78 -C

MIG_DEVICE_UUID=$(
    nvidia-smi -L |
    sed -n 's/.*Device[[:space:]]*0:.*UUID:[[:space:]]*\(MIG-[^)]*\)).*/\1/p'
)

export MIG_DEVICE_UUID
echo "$MIG_DEVICE_UUID"

sudo systemctl start nv-graphics.service
sudo systemctl start gdm3.service

Hi

Memory with 0MiB is expected.

You don’t need to export the compat library.
The container has its own package so there is no need to export from the native device.

After verifying the Docker service is working, it’s expected to work with the following command:

$ docker run --rm --runtime=nvidia --privileged -v /home/nvidia:/home/nvidia -it nvcr.io/nvidia/pytorch:26.07-py3

You will need to use a 2026 container to have Thor device support.
Then run with the --privileged to enable the MIG workaround.

Thanks.