Can you please ask around about this, because the thread I linked specifically talks about that this worked on Linux with certain VBIOS and the 575 version driver, however does not work at all with subsequent drivers.
This is exactly the issue we have with the RTX Pro 6000 Workstation Edition on Linux. We can not pass it through to Linux VMs since the driver does not register mdev devices anymore.
In other words you spawn the MIG instances but then there is nothing registered in the Linux OS that you could pass through. Maybe the person in the thread is wrong and this was not an intentional rugpull but is a software issue but then it would be nice to have this investigated.
The fact that MIG was advertised but is not working at the moment on these Blackwell cards on Linux sucks massively and has hindered me to get more cards since I can’t use them as I intent to.
Thank you for quickly getting back to us on this though, if you could hear around about this, maybe find another employee who is familiar with the details it would help all of us here a lot.
Under Linux guests on supported hypervisors, when MIG-supported GPUs are in GPU pass-through, the same workflows workflows, tools, and Supported MIG Profiles available on bare-metal can be used.
MIG allows multiple vGPUs (and thereby VMs) to run in parallel on a single MIG-supported GPU, while preserving the isolation guarantees that vGPU provides. To configure a GPU for use with vGPU VMs, refer to the Configuring a GPU for MIG-Backed vGPUs. Refer also to the technical brief for more information on GPU partitioning with vGPU.
this would need to be read as:
you can either passthrough your full (MIG enabled) GPU to a VM, and then have multiple MIG instances in that (Linux) VM, and would schedule multiple jobs onto the different instances, or
if you needed only one MIG instance per VM, to achieve the isolation only vGPU gives, then you would have to add a hypervisor and a vGPU profile per VM onto the passed-through GPU, assign one MIG instance per VM.
This will work with any GPU that supports MIG and vGPU - as the RTX PRO 6000 server edition. But the workstation GPUs, those that do support MIG the first time (RTX PRO 5000 and 6000 workstation edition), do NOT support vGPU.
Since we are interested to learn about usecases, is this for development, or a production usecase? I see where for development/experimentation, a workstation setup is convenient, but why would a production setup favor a workstation over a server?
MIG passthrough with libvirt/KVM already worked on old kernels and older drivers. It is apparent that the Workstation Edition and MaxQ have the functioniality too. There was also no need for a vGPU driver.
You’d put the GPU in compute mode, spawn the MIG instances and the driver would register MDEV devices. You could pass these through to a virtual machine on Linux. This was confirmed working in the past on Workstation Edition cards and now it is not working anymore and behind a vGPU paywall if I am understanding this correctly. Not to mention that the vGPU paywall is only available on one version of the three.
Well, with previous 6000 class workstation GPUs, so actively cooled, there was no MIG, so the only option to share such a GPU for multiple workloads was to offer vGPU…
With introducing MIG to Blackwell 6000 class cards, MIG is the way to separate multiple workflows on such a card. If you needed tenancy separation on top, we’d assume you’d be running in a server chassis, for multiple tenants to remotely access, but not have multiple remote users access a workstation chassis (under the desk…).
ANY config we officially support needs testing and qualification, if we just enabled without any testing, it would fall apart at some time, and show like a regression, so we decided to spend the extra effort of testing, qualifying and certifying for only the server GPUs, which show vGPU adoption WAY more than workstation type GPUs…
So we do have options: MIG (on workstation edition) for separating workloads, vGPU (on server edition) for stronger tenancy separation.
If we miss a large usecase pls let us know, so we could reconsider the effort of testing and qualification over the opportunity…
this exactly the problem with NV’s approach: if it is not immediately obvious that something will bring tons of cash right away, it is blocked completely. Instead, it could be disabled by default, but possible to enable by some flag with a huge disclaimer in the description that it is not officially supported. By allowing customers to use the hardware the way they want, they would be able to figure out themselves what configuration/setup is optimal for them and NV would get a real-world feedback, instead of guessing and forcing users to navigate such authoritative choices.
@Frank_Quadro since MIG on workstation edition is the officially supported path here, wanted to flag we’re hitting a hard block on the mode switch itself, not the vBIOS gate.
Supermicro AS-4125GS-TNRT2 (NVIDIA-qualified for 8x RTX PRO 6000 Max-Q), 3x RTX PRO 6000 Blackwell Max-Q + 4x L40S. vBIOS on the Max-Q cards is 98.02.6A.00.03, above the documented minimum (98.02.6A.00.00).
Clean Debian 13 live boot, driver 610.57.04 installed and confirmed loaded (dmesg + lsmod), then rmmod nvidia and confirmed unloaded before touching the tool. Secure Boot isn’t enforcing (dmesg shows the module tainting-and-loading rather than being rejected). BIOS PCIe (Above 4G Decoding / Re-Size BAR) is already proven live on this same board - our L40S cards run 65536MiB BAR1 in production.
Display Mode Selector 1.72.0: --list enumerates all 7 adapters correctly. --version reads the L40S cards fine (95.02.66.00.15) but returns “Version N/A” on all 3 Max-Q cards - looks like even the read path is blocked, not just the write.
Every mode-switch attempt against the Max-Q cards - --gpumode compute --auto, --gpumode physical_display_disabled -i 0/1/2, and the interactive menu - fails identically:
PROGRAMMING ERROR: HW access out of range.
Given vBIOS is above minimum, driver load/unload was done exactly as documented, Secure Boot isn’t a factor, and BIOS PCIe prereqs are confirmed live elsewhere on the same board - can this be escalated with NVIDIA directly, or not?
Hi Janez, not sure what this could be really, so all I can recommend is to try with the latest version of DisplayModeSelector, which should be 1.76 IIRC, and maybe try with ONLY the MAX-Q boards in the system, so remove the L40S…
We do not test, hence not explicitly support mixing different GPUs in the same chassis, so maybe something gets confused…?
Also could try if taking a single Max-Q into a different system, like a workstation, and changing it there, individually, makes any difference…?
Agree that you should be able to change a MAX-Q to display=OFF, to then be able to enable MIG on them. (L40S comes in Display=OFF mode, and MIG can directly be enabled…).
Not a Linux expert, to come up with any ideas to further investigate under Linux…
Hi Frank,
Thank you for replying to me! I am also suspicious of mixing different gpus. We will probably do it with another workstation.
Thank you.
Will update you on my findings.
I got the “PROGRAMMING ERROR: HW access out of range” on an RTX PRO 6000 Blackwell on Ubuntu 24.04, driver 580.105, tool 1.76, Secure Boot off. I also had an RTX 6000 Ada in the same workstation.
I removed the Ada GPU and then the Blackwell mode switch worked. I believe this confirms the mixed-GPU theory from @janez.kricej and @Frank_Quadro .
Then I had trouble getting Ada 6000 GPU working. I updated my BIOS from 1203 to 1317. Then I enabled “Resize BAR Support” in BIOS. I have an ASUS Pro WS WRX90E-SAGE SE motherboard.
Both GPUs are working and MIG is enabled for RTX PRO 6000 Blackwell.
