Cuda programs and nvidia-smi take many seconds to start with 460.27.04 driver

Graphics / Linux Linux
1

I’m running Fedora 33 and just upgraded to Linux 5.9.16 and NVIDIA 460.27.04 drivers. I’m seeing occasional short hangs which are 100% reproducible when running cuda programs, including nvidia-smi.

If I time a run of nvidia-smi, it takes more than 8 seconds to run. A portion of the time while it is running, my mouse stops responding, but comes back intermittently, and then behaves normally after nvidia-smi completes.

If I run journalctl, I can see that I reliably get kernel warnings each time I run that look like

Dec 29 10:21:54 kernel: NVRM: GPU 0000:09:00.0: rm_init_adapter failed, device minor number 0
Dec 29 10:21:54 kernel: NVRM: GPU 0000:09:00.0: RmInitAdapter failed! (0x26:0x65:1290)
Dec 29 10:21:50 kernel: NVRM: GPU 0000:09:00.0: rm_init_adapter failed, device minor number 0
Dec 29 10:21:50 kernel: NVRM: GPU 0000:09:00.0: RmInitAdapter failed! (0x26:0x65:1290)

I’ve attached a bug report generated by the tool nvidia-bug-report.log.gz (673.0 KB)

2

I’ll also note that I appear to have about a 15% performance regression in my CUDA/OptiX application. Unsure if that is related to whatever is going on with CUDA startup times.

3

Maybe this?
https://www.phoronix.com/scan.php?page=news_item&px=Linux-Kernel-Blocking-NV-NetGPU

4

Possible. I was coming from 455.45 drivers and 5.9.14 kernel, so I doubt that anything so fundamental has changed?

5

Update: Still seeing this issue with 460.32 driver on 5.10.7 kernel.