NVIDIA-SMI fails to connect to NVML with UNKNOWN ERROR on Quadro RTX 5000

Development Tools Other Tools System Management and Monitoring (NVML)
#1

Configuration is a Quadro RTX 5000 with monitors supported by a Quadro P400 on an I 9980 XE processor, ASUS PRIME X299 Deluxe II motherboard. I previously had a P4000 GPU in the system and have taken all measures to upgrade the software accordingly. I am NOT running an Insider build on this system but have other systems on which I can try the card. I have 4 on premises to replace the P4000 The P4000 was running with an earlier driver and corresponding version of NVIDIA-SMI.

Though the program does respond with the help request, when attempting to run the NVSMI utility for basic functions like system survey, I get this error.
C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe
Failed to initialize NVML: Unknown Error

This morning I captured a trace with Sysinternals Procmon and am going through it now. Same tool level runs on laptop with a Single GPU and no error. is there a way to run the utility in trace mode?

It occurred to me that the driver might be ordering sensitive so I reversed the order of the P400 and RTX 5000 in the PCIE slots putting the RTX in the Zeroth slot previously occupied by the P400. No change in behavior was observed.

As a FINAL verification of the Driver Level, I dropped back to the P4000 GPU Card and that got the failure also. There appears to be a FATAL FLAW in the NVIDIA System Management Interface – v442.74

#2

NVIDIA System Management Interface – v436.30 also fails but the cards work great when a monitor is hooked up.
The NVIDIA Control Panel works fine with the cards also.
As near as I can tell the issue is solely with the SMI call

#3

Am I correct in the understanding that NOBODY ELSE has seen this?

#4

I was FINALLY Able to get my RTX 5000 to install and initialize properly. It actually connected with the Application once I was able to set it up in TCC Mode. When I did the install on this particular machine I had somehow gotten an OLDER (translated obsolete) copy of NVML.DLL in the system load paths. As a result it was picking up a two year old copy of the DLL that was incompatible with the Driver. To remedy this, I had to take the machine back to the base driver set, uninstalling and deleting all of the old packages in the system (there were 4 older ones) and then manually search the system disk for obsolete versions of NVML.dll. Once I found all of them, I deleted them manually. It appears that the “clean install” does not check for obsolete versions of libraries when it cleans up. I also used the driver Uninstall from the Device manager to uninstall and delete any residual files that might be around (like old .inf files) Then I rebooted and it came up WITHOUT a Display Adapter Category in the Device Manager. so when your software did the compatibility test it did not find anything in the Device manager metadata that looked like an NVIDIA component and died on the install. I went into the Device Manager tool and did a scan for hardware changes which found the Quadro P400 with monitor connections and enumerated those, gave it the Microsoft default Driver for PNP. Then when I ran the Driver install package it worked on the P4000 card that I had in for testing. Then I powered down to install the RTX5000 card and brought it up. All seemed well except there was an extreeemely long delay during the windows initialization where the screen froze and being familiar with device drivers I let it sit for the 15 minutes it took to do the POST and whatever initialization and the screens went black for about 2 minutes and then Colour! So now I ran the NVSMI tool to see the Quadro RTX 5000 config, then did the TCC Compute Model Setup an now have a syatem to do work on. I have 3 more systems to do the insatall on. Me? Happy the rest is going to be a piece of Cake! My boss, the CEO will be happy we can meet our production goals