God bless OpenAI and ChatGPT
Resolved with a reproducible workaround
After further testing, I found that the failure is related to the PCIe link between the Thunderbolt enclosure’s downstream bridge and the RTX 5080.
The immediate parent bridge of the GPU is:
0000:03:00.0
Intel Thunderbolt 80/120G Bridge
Barlow Ridge Hub 80G 2023
PCI ID: 8086:5786
The GPU is:
0000:04:00.0
NVIDIA GeForce RTX 5080
PCI ID: 10de:2c02
By default, the bridge-to-GPU link operated at PCIe Gen4 x4:
LnkSta: Speed 16GT/s, Width x4
LnkCtl2: Target Link Speed: 16GT/s, SpeedDis-
The system became stable after I forced this link to PCIe Gen3 x4 and disabled hardware-autonomous speed changes before loading the NVIDIA kernel modules.
The working register configuration is:
BR=0000:03:00.0
sudo setpci -s "$BR" CAP_EXP+30.w=0023:002f
sudo setpci -s "$BR" CAP_EXP+10.w=0020:0020
This performs the following:
Target Link Speed: Gen3 / 8 GT/s
Hardware Autonomous Speed Disable: enabled
Link retrain: requested once
It is important to wait until link training has completely finished before loading nvidia.ko.
The stable state is:
LnkSta: Speed 8GT/s, Width x4
LnkCtl2: Target Link Speed: 8GT/s, SpeedDis+
Raw register values:
LnkCtl2: 0063
LnkSta: 7043
I initially loaded the NVIDIA driver while the raw Link Status value was 3843. Although this already reported Gen3 x4, the Link Training bit was still set, and modprobe nvidia failed with:
modprobe: ERROR: could not insert 'nvidia': No such device
The reliable procedure waits for:
Current link speed = Gen3
Current link width = x4
Link Training bit = clear
Data Link Layer Active bit = set
Several consecutive stable readings
It then waits an additional two seconds before loading the NVIDIA modules.
After the link is stable, I load:
sudo modprobe nvidia
# NVIDIA initialization sometimes changes the target for the next retrain,
# so reassert Gen3 without requesting another retrain.
sudo setpci -s "$BR" CAP_EXP+30.w=0023:002f
sudo modprobe nvidia_modeset
sudo modprobe nvidia_uvm
sudo modprobe nvidia_drm
sudo setpci -s "$BR" CAP_EXP+30.w=0023:002f
I made this persistent with a systemd oneshot service. Automatic NVIDIA loading is blocked using:
modprobe.blacklist=nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm
The service dynamically detects GPU 10de:2c02, finds its immediate parent bridge, verifies that it is Intel bridge 8086:5786, applies the Gen3 configuration, waits for link training to finish, and then explicitly loads the NVIDIA modules.
Successful service log:
egpu-gen3: GPU=0000:04:00.0, parent bridge=0000:03:00.0
egpu-gen3: before: LnkCtl2=0044 LnkSta=3044
egpu-gen3: Gen3 stable before NVIDIA: width=x4 LnkSta=7043
egpu-gen3: GPU configuration space ready: 10de:2c02
egpu-gen3: ready: LnkCtl2=0063 LnkSta=7043 width=x4
The service now completes successfully during every tested cold boot:
Active: active (exited)
Result: SUCCESS
Results after applying this workaround:
nvidia-smi: works
cuInit(): succeeds
cuDeviceGet(): succeeds
cuCtxCreate(): repeatedly succeeds
cuCtxDestroy(): repeatedly succeeds
Hashcat: works
Ollama: works
I ran multiple consecutive CUDA context creation/destruction tests and real GPU workloads without another hang.
The following changes were not required for the working setup:
No NVIDIA source patch
No HMM disable
No D3cold modification
No ACS modification
No PTM modification
No IOMMU disable
I kept:
iommu=pt
NVIDIA open driver 610.43.02
Kernel 7.0.0-28-generic
My conclusion is that the hard lock is triggered by either PCIe Gen4 operation or autonomous PCIe speed changes/retraining on the tunneled bridge-to-GPU link. The failure occurred when the first significant CUDA operation initialized the GPU, while forcing a stable Gen3 x4 link before NVIDIA driver initialization resolves it.
This is a workaround rather than an ideal permanent fix. NVIDIA may need a driver or firmware quirk for the following combination:
GPU: 10de:2c02
Parent bridge: 8086:5786
Connection: Thunderbolt 5 / tunneled PCIe
It may be useful for the driver to prevent autonomous speed changes, select a safe link speed for affected external Blackwell GPUs, or ensure that PCIe link training has completely finished before GSP initialization.