I’m noticing a similar issue to steven6_wang (link here). I get “CUDA failed to initialize <…>”
Details:
I upgraded my thor from a flashed jetpack 7.0 to 7.2 using the new on board, no-flash, GUI updater. I followed the thor recommended profile 83 & 78 MIG profile setup and (see nvidia-smi dump below). (as an aside, I had to apply the fix from this other forum post -link here: tldr: delete /etc/modprobe.d/nvidia-unifiedgpudisp.conf)
Then, I run docker run --rm -it --runtime=nvidia --privileged nvcr.io/nvidia/tensorrt:25.09-py3 bash. (note - here, I used a different image than the original post. when I get a chance, I’ll run with the pytorch:26.07-py3 image too to match the earlier forum post) I ran the pytorch:26.07-py3 image, also with sudo, and I get the same error - with the only difference being that with the matching image, I get the same note about SHMEM.
The docker container starts, but I get “CUDA failed to initialize”. (not posted here - in my image using tensorrt:25.09-py3 as a base, where torch is installed, torch hits related cuda failure issues)
If I run nvidia-smi in the container, it seems to output the same info from the host (maybe because I’m using --privileged)
nvidia@nvidia:~$ docker run --rm -it --runtime=nvidia --privileged nvcr.io/nvidia/tensorrt:25.09-py3 bash
=====================
== NVIDIA TensorRT ==
=====================
NVIDIA Release 25.09 (build 207438969)
NVIDIA TensorRT Version 10.13.3
Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Container image Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
https://developer.nvidia.com/tensorrt
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/)
and the Product-Specific Terms for NVIDIA AI Products
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/).
To install the open-source samples corresponding to this TensorRT release version
run /opt/tensorrt/install_opensource.sh. To build the open source parsers,
plugins, and samples for current top-of-tree on master or a different branch,
run /opt/tensorrt/install_opensource.sh -b <branch>
See https://github.com/NVIDIA/TensorRT for more information.
ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.
[[ Initialization error (error 3) ]]
groups: cannot find name for group ID 993
root@cf6e0d440bff:/workspace#
root@cc8fd071fa0c:/workspace# nvidia-smi
Mon Sep 14 17:41:34 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.78 Driver Version: 595.78 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA Thor On | 00000000:01:00.0 Off | N/A |
| N/A 36C N/A 5W / N/A | Not Supported | N/A Default |
| | | Enabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| MIG devices: |
+------------------+----------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Shared Memory-Usage | Vol| Shared |
| ID ID Dev | Shared BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+==================================+===========+=======================|
| 0 1 0 0 | N/A | 12 0 | 1 1 1 0 1 |
| | N/A | | |
+------------------+----------------------------------+-----------+-----------------------+
| 0 2 0 1 | N/A | 6 0 | 1 1 1 1 1 |
| | N/A | | |
+------------------+----------------------------------+-----------+-----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
I also noticed if I add --user "$(id -u):$(id -g)" to my docker command, I also get a line saying " [[ Possible MIG misconfiguration detected ]]"
nvidia-smi from thor host (outside docker container)
nvidia@nvidia:~$ nvidia-smi
Mon Sep 14 10:34:42 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.78 Driver Version: 595.78 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA Thor On | 00000000:01:00.0 Off | N/A |
| N/A 36C N/A 5W / N/A | Not Supported | N/A Default |
| | | Enabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| MIG devices: |
+------------------+----------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Shared Memory-Usage | Vol| Shared |
| ID ID Dev | Shared BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+==================================+===========+=======================|
| 0 1 0 0 | N/A | 12 0 | 1 1 1 0 1 |
| | N/A | | |
+------------------+----------------------------------+-----------+-----------------------+
| 0 2 0 1 | N/A | 6 0 | 1 1 1 1 1 |
| | N/A | | |
+------------------+----------------------------------+-----------+-----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
nvidia@nvidia:~$