Multi GPU support for 50xx series

frustum · August 11, 2025, 6:34am

Greetings,

We’re encountering significant issues getting multi-GPU configurations working properly on the RTX 50xx series.
The Microsoft heterogeneous multiadapter sample fails during the CreateHeap() call when using the following heap flags: D3D12_HEAP_FLAG_SHARED | D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER

github.com/microsoft/DirectX-Graphics-Samples

Samples/Desktop/D3D12HeterogeneousMultiadapter/src/D3D12HeterogeneousMultiadapter.cpp

master

//*********************************************************
//
// Copyright (c) Microsoft. All rights reserved.
// This code is licensed under the MIT License (MIT).
// THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF
// ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY
// IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR
// PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.
//
//*********************************************************

#include "stdafx.h"
#include "D3D12HeterogeneousMultiadapter.h"

const float D3D12HeterogeneousMultiadapter::TriangleHalfWidth = 0.025f;
const float D3D12HeterogeneousMultiadapter::TriangleDepth = 1.0f;
const float D3D12HeterogeneousMultiadapter::ClearColor[4] = { 0.0f, 0.2f, 0.3f, 1.0f };

D3D12HeterogeneousMultiadapter::D3D12HeterogeneousMultiadapter(int width, int height, LPCWSTR name) :
    DXSample(width, height, name),

This file has been truncated. show original

In addition, all legacy applications that previously worked flawlessly on RTX 20xx, 30xx, and 40xx series GPUs now report a driver internal error on the RTX 50xx series:
D3D12 WARNING: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DRIVER_INTERNAL_ERROR: There is strong evidence that the driver has performed an undefined operation; but it may be because the application performed an illegal or undefined operation to begin with.).

System configuration:
2x NVIDIA RTX 5070, 580.88 driver, Windows 10.

We’ve also confirmed that GravityMark fails in a similar way on a system with two RTX 5090 GPUs.

Thank you

jayventuri · September 1, 2025, 4:59pm

Hello,

this issue is reproducible on our end as well.

Apperent only on 5.xxx series GPUs from nvidia.

Issue does NOT manifest on 4.xxx, 3.xxx and 2.xxx series cards.

5..xxx does work in mGPU DX12 on VK

AMD no issues in DX12 or VK in mGPU

this not a reference to SLI or Crossfire, this is strictly DX12 mGPU

D3D12_HEAP_FLAG_SHARED | D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER

D3D12 WARNING: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DRIVER_INTERNAL_ERROR

whenever a selection such as mGPU DX12 is made from a menu item like this:

this error occurs on execution:

D3D12_HEAP_FLAG_SHARED | D3D12_HEAP_FLAG_SHARED_CROSS_ADAPTER

D3D12 WARNING: ID3D12Device::RemoveDevice: Device removal has been triggered for the following reason (DXGI_ERROR_DRIVER_INTERNAL_ERROR

Summary:

Issue is localized to 5.xxx series nVidia GPUs on DX12 mGPU (non SLI)

This is 100% reproducible on 5.xxxx GPUs and mGPU/DX12

Additionally, the issue is independent an non-relevant to any BIOS setting or motherboard

tested on Asus WS PRO, Asus Sage Gigabyte MZ72 series and MZ73 series.

OS was fully patched Windows 11, and Windows Data Center 2022 and 2025

All GPUs tested were the Founders Edition, except for the AMD GPUs of course

This applies to all drivers so far from 572.16 to 581.15

MarkusHoHo · September 2, 2025, 7:40am

Hi there @frustum and @jayventuri, nice to see you on the NVIDIA developer forums.

Usually we point users with consumer side issues to the GeForce Forums.

In this case I was able to find this issue already in our internal bug tracker.

The good news is that there is a fix to it. But it is not quite certain which release driver will contain the fix.

I am currently trying to find that out.

Thanks!

jayventuri · September 2, 2025, 2:53pm

Thank you Markus, If you need to beta test / get feedback on the fix/driver please let me know. Glad to help

Jay

frustum · September 3, 2025, 5:39am

Markus, thank you for the update. It would be great to have a driver to test.

MarkusHoHo · September 3, 2025, 10:09am

I might have been too rushed in my response, sadly. The fix had been deployed already, but it seems it addressed an Omniverse specific use-case. But it has the same signature so engineering at least has a good start what to look for.

I opened a new internal bug and will let you know if I need some more input.

One thing that might help would be a kernel (memory) dump if any of you can provide one after the issue occurs.

Thanks!

jayventuri · September 7, 2025, 3:32pm

Thank you, I’ll see what we can do for a kernel memory dump

jayventuri · September 8, 2025, 1:39pm

Sir

solved the DX12 and multiple 5.xxx series GPUs errors.

example. DX12 and dual 5090s is SOLVED

this also solved it for:

Gravitymark in DX12 mGPU AFR
Ashes of the Singularity DX12 mGPU
Strange Brigade in DX12 mGPU
And several UE5 games mGPU

UTILIZATION in GravityMark as example:
GPU 1. - 96-100%
GPU 2. - 83-94%

I still have some refinements to do, but I’m most of the way there

lastly, overall performance increased for some single GPU benchmarks/games/apps

Details to follow

J

mGPU for 5.xxx series GPUs is now working

tested on:

dual 5090 FE

dual 5080 FE

Details to follow

MarkusHoHo · September 9, 2025, 2:14pm

That is great news!

Which driver version is that now @jayventuri ?

jayventuri · September 9, 2025, 2:49pm

The fix is not specific to any driver. I guess I fixed it for all drivers from 72.x to 81x for al 5.xxx series GPUs

I tested the fix on 72.x drivers, 73.x, 76.x and 81.x

I’ll post details of the fix shortly

jayventuri · September 10, 2025, 12:38am

Since the Omniverse Kit (thank you for letting me know about the prior issue) is now working fine with the dual 5.xxx series cards in DX12 applications

So I went on a deep dive of the profile parameters in profile inspector for the Omniverse Kit.

There are several unique modifications compared to other profiles and so I tested each to eliminating the key that makes or breaks Dual 5.xxx series cards from working in mGPU mode in DX12.

The only differences, but effective are:

That was all it took.

REGARDLESS if those feature flags are used or not, they have to be disabled (both) in the profile of the app/game for DX12, or mGPU will not work

I expected something mor elaborate as I was diligently eliminating Omniverse specific keys, but alas, turns out only these two variables need to be set.

What makes NO SENSE is a side effect of some benchmarks having higher scores in Single GPU from those variables being set.

Obviously there is a significant issue in the nvidia drivers from 72.xx to 81.xx and this is just a work around, hopefully nvidia will fix it soon …

Topic		Replies	Views
Dxgi_error_device_removed dxgi_error_driver_internal_error Drivers - Linux, Windows, MacOS unreal-5	1	977	July 23, 2023
Unreal Engine: DXGI_ERROR_DEVICE_REMOVED DXGI_ERROR_DRIVER_INTERNAL_ERROR GPU - Hardware	0	1116	July 20, 2023
Standard nVidia CUDA tests fail with dual RTX 4090 Linux box Linux	54	21598	April 29, 2024
DS 6.0 Shared memory multiple GPU issue DeepStream SDK nvbugs	7	1597	June 29, 2022
[Solved] CUDA driver initialization failed - 2x RTX 5090 CUDA Setup and Installation cuda , pytorch	4	1745	May 28, 2025
RTX 5090 - ERROR: GPU:1: Error while waiting for GPU progress Drivers - Linux, Windows, MacOS	2	170	July 17, 2025
Failing to find GPU when using 2 x A5000 Kit App Template / SDK	6	48	February 5, 2025
RegisterResource sometimes fails with multiple cards General	28	3577	October 13, 2020
Cuda initialization error using two RTX 5090 GPUs CUDA Programming and Performance	0	173	April 25, 2025
Driver 560.X fails to initialize H100 GPUs, but previous versions work fine Linux kernel	4	923	October 30, 2024

Multi GPU support for 50xx series

Related topics