for a specific medical application I do need a computer that supports GPU calculation in TCC mode. For this purpose, I bought a Dell 7740 mobile workstation which is equipped with a dedicated Nvidia Quadro RTX 5000 graphics card as well as a Intel UHD Graphics 630. (Dell advertised this computer for powerful graphics applications.) The computer has 128 GB of main memory and is running with Windows 10 Pro.
In the computers BIOS on can select autotmatic switching between both graphic cards or deactivate the on board Intel UHD Graphics 630.
I could install the driver “461.09-quadro-rtx-desktop-notebook-win10-64bit-international-dch-whql.exe” as well as “cuda_11-2-0_460.89_win10.exe” without any problems.
I could start Windows PowerShell as an Admin and started nvidia-smi.exe in the c:\WINDOWS\system32 directory. The garphics card is recognized but I am unable to switch in TCC mode.
Using “nvidia-smi -g 0“ the graphics card can be addressed explicitly. But the command “nvidia-smi -g 0 -dm 1” or “nvidia-smi -g 0 -fdm 1” results with the error message “Unable to set driver model for GPU 00000000:01:00.0: Not supported. Treating as a warning and moving on. All done”.
I tried to make sure that the RTX 5000 is not in use by the Windows 10 OS. In the NVIDIA Display menu I could assign the physics calculation to CPU and the 3D mode to the on board Intel UHD graphics.
With the NVIDIA tool „Nsight Monitor“ / „Nsight Monitor options“ / „Microsoft Display Driver“ I could explicitly set „WDDM TDR Enabled“ to „false“. After rebooting the system the „Nsight Monitor“ tells me that WDDM-Modus is disabled. But also in this configuration „nvidia-smi -g 0 -dm 1“ or „nvidia-smi -g 0 -fdm 1“ failed to activate the TCC mode.
All CUDA-Sample programs for Microsoft Visual Studio 2019 are running without problems.
On every CUDA information page the RTX 5000 is announced to be compatible with alle GPU calculation modes.
But what does the error message „Unable to set driver model for GPU 00000000:01:00.0: Not supported. Treating as a warning and moving on. All done” tell me?
Is anything wrong on my computer configuration what prevents to select the TCC mode?
Is the TCC mode never compatible with the Quadro RTX 5000?
Is this a problem of my driver version? (I also tried out a large number of older driver / CUDA versions but without any success.)
Will the TCC mode be supported in future driver versions?
So far I was unable to get an answer to my questions (neither Dell nor Nvidia).
I have a Dell workstation (Precision 5820 Tower) with two Quadros, one of which is a Quadro RTX 4000 (the SKU below the Quadro RTX 5000). Running Microsoft Windows 10 Pro for Workstations with NVIDIA driver version 461.40.
Quadro P2000 00000000:17:00.0 not connected to display
Quadro RTX 4000 00000000:65:00.0 connected to display
Here is what I am seeing when I try to switch to TCC mode (from an Administrator Command Prompt):
C:\Windows\system32>nvidia-smi -i 1 -fdm 1
NOTICE: Disconnect and disable the display before next reboot. You are switching to TCC driver model with display active!
Set driver model to TCC for GPU 00000000:65:00.0.
All done.
Reboot required.
C:\Windows\system32>nvidia-smi -i 1 -fdm 0
Set driver model to WDDM for GPU 00000000:65:00.0.
All done.
Reboot required.
C:\Windows\system32>nvidia-smi -i 0 -fdm 1
Set driver model to TCC for GPU 00000000:17:00.0.
All done.
Reboot required.
C:\Windows\system32>nvidia-smi -i 0 -fdm 0
Set driver model to WDDM for GPU 00000000:17:00.0.
All done.
Reboot required.
My machine is crunching on a whole bunch of long-running compute tasks right now so I cannot reboot it at the moment, but as you can see from the above, nvidia-smi accepts my commands to switch between TCC and WDDM modes, for either GPU. One obvious difference to your system setup is that I have no built-in Intel graphics (CPU is a Xeon W 2133; Skylake-W) and I have two NVIDIA GPUs, one of which can be available to drive the display.
Why specifically are you trying to switch to TCC mode?
thanks for your answer. But I do need a definitive answer not “expect”. Who could be able to answer this question? It also remains unclear why the TCC mode should not be supportet. Are there any hardware concerns or is it just a lack of will to support this grapcis card? It is not a conventional laptop computer, it is a mobile workstation that should support all functions of a desktop PCs.
Thanks a lot for you comments! I do need the TCC mode for a specific program which calculates dose distributions in radiation oncology. Without this mode the software does not support GPU calculations. In principle everything should work on the mobile workstation, the software is running except for the GPU calculations. It might be a problem that the second graphics card is not from NVIDIA but it is build in and cannot be changed of course.
Interesting. TCC vs WDDM is pretty much orthogonal to the functionality of a CUDA program. Unless this software needs to run as a Windows service, or needs to be accessible through Windows remote desktop, there should be no functional difference between the two drivers. Or is TCC being used to improve soft-realtime behavior (lower latency, reduced jitter)?
Thanks a lot for your comment! My problem ist the following: The treatment planning system (TPS) is Varian Eclipse which was developed for a limited number of computer platforms only. As it is a medical solution there is no guarantee for hardware solutions others than specified by Varian.
As an “end user” I must accept the software “is as it is”. Specialists form Varian installed the TPS and it works except for the GPU calculations. Varian’s software engineers tried to figure out why the software is not running in GPU mode. They defined it to be a prerequisite to switch to TCC mode. As long as I cannot prove that this mode will work on my computer, they will not proceed to fix the GPU issue.
I am sorry to tell you that I do not have any further insights to the TPS. But ist is very interesting to know that WDDM does not give any functional difference.
Makes sense. I should have considered that since this is part of a medical apparatus, there are regulatory requirements which prevent changes to the apparatus including substantial software changes such as changing the driver used.
As to why TCC cannot be turned on in your machine I can only make the vague guess that this is precluded by the complicated interaction between integrated graphics and NVIDIA GPUs that one finds in mobile devices.
In practical terms, you might want to try your Quadro RTX 5000 with a different hardware platform without integrated graphics. Obviously that requires an additional NVIDIA GPU to drive the display and assumes that the Quadro RTX 5000 in question is actually a standard PCIe plug-in card, not some specialized variant for use in mobile devices.
Thanks again for you ver y quick reply!
There is no option to change the hardware configuration for me I must take the Dell 7740 “as it is”. But is absolutely impossible for me to find out differences between the Dell’s RTX 5000 and desktop versions.
Everything seems to be a matter of the driver. No chance without Nvidia’s support.
If you cannot change the system and cannot change the application, and the NVIDIA drivers don’t magically acquire a new capability apparently not in existence at this time (due to gnarly technical issues in the interaction between integrated graphics and NVIDIA GPUs I would guess), you might want to get acquainted with the idea that you will not be able to run this app with GPU acceleration on your system …
Configuration issues such as this one are clearly not at the core of “CUDA Programming and Performance”, but I have no idea where else to point you. You could file a bug report with NVIDIA, but have no idea what the benefits would be. Clearer future documentation maybe.
I do appreciate your comments! The major problem is that one cannot find out what kind of capabilities each NVIDIA GPU has.
Let’s got to the latest driver version:
The manual “NVIDIA RTX Enterprise Release 460 Drivers, Version 461.40” describes how to switch to the TCC mode.
So far, from my reading everything should be fine. Not even an idea that TCC mode should NOT work.
But why does it not work???
Even in this forum nobody seems to know what is the truth for the Nvidia Quadro RTX 5000. The German support gave up and I was asked to try it in this developer forum.
Hopefully somebody from NVIDIA will be able to answer my questions.
I think you might be overestimating how many NVIDIA employees read these forums. And you might be overestimating how much information employees of a large technology company have access to beyond that which is publicly documented. Occasionally there might be an employee present here who actually works on the aspect of the technology that someone is interested in, but I don’t recall encountering any NVIDIA employee with driver background in the CUDA forums in almost a decade.
Most of what happens here is pointing at the most suitable sub-forum, pointing at relevant official documentation, and sharing personal experience. These forums are primarily intended as a platform for user-to-user communication, not as a support channel.
My observation about TCC is that this a feature that is generally poorly documented by NVIDIA. Over the years, we have had quite a number of questions regarding TCC support with specific products, and there never seems to be good official documentation about it. Mostly it seems to be a matter of trying it. Which isn’t a very satisfying state of affairs.
Our development team confirmed that TCC mode is not supported on this mobile GPU. It is by design. I don’t have further information and will not be able to answer questions like “why?”
The development team actually confirmed this back in August of 2021. The notification escaped me. I apologize for the subsequent delay.