Driver crash with "re-shared" D3D11on12 textures since driver 580.88

Gaming and Visualization Technologies APIs DirectX, DXR, DirectCompute
,
1

This problem is related to “re-sharing” D3D11 textures originating from a D3D11-on-12 device, and seems to have started occurring in 580.88 (at least that’s the first mass-release driver we saw the problem with). This seems to need a few things for it to happen, so bear with me.

Not sure if the cross-process nature of this is relevant, but including the info just in case.
D3D11 devices in use (all connected to the same DXGI adapter):

  • D1 = D3D11-on-12 device, in process P1
  • D2 = D3D11 device, in process P2
  • D3 = D3D11 device, in process P2

The chain that leads to the crash for us was as follows:

  • D1 creates a texture T1
  • D1 gets shared handle to texture T1 (using IDXGIResource::GetSharedHandle)
  • D2 opens the texture T1 by handle as T2 (using ID3D11Device::OpenSharedResource)
  • D2 gets shared handle to T2 (using IDXGIResource::GetSharedHandle)
  • D3 opens the texture T2 by handle as T3 (using ID3D11Device::OpenSharedResource)

This last action triggers a TDR (crashes the driver); I think it was literally on open, but it could have been first attempt to use the texture for anything. This only happened in the code path where D1 is a D3D11-on-12 device, so it’s likely specific to that somehow. This same flow worked fine in NVIDIA drivers pre-580.88, and works fine on AMD and Intel.

2

Hi, we see the same problem in our Mirillis Action! screen/game recording and streaming software. In our case there is no crash but accessing any cross-process shared textures causes stalls in display (even whole desktop can freeze). This happens on all GTX and RTX GPUs with 580.xx and newer drivers.

This has been working for 10 years and now it’s broken! Rolling back to 577.00 fixes the problem immediately.

It only happens with D3D12 to D3D11 cross-process sharing and works ok for D3D11 to D3D11 cross-process sharing.

3

Thank you @OldAlot for bringing this to our attention.

I filed an internal issue to track this, and I would appreciate if you could share some detail on the platform or platforms where you have seen this happen. Specifically which OS and which GPU you are using.

Thanks!

5

And welcome @htq1 to the NVIDIA developer forums.

Feel free to also add your Setup details.

Thanks!

6

Hi Markus,

Thank you so much for your prompt reply!

I’ll do my best to help you understand what’s wrong since 580.xx update. Thousands of our user complain about this and we have tried to fix it on our side but were not able to :(

Summarizing, two big problems were found in the driver that was released after 577.00.

1. CUDA ( cuInit(0) ) no longer works.

Some part of our code are still 32bit. The GPU feature checker used CUDA to check HW version and other parameters of enumerated GPUs. Since 580.xx cuInit failed with “no device found”.

I’ve read that CUDA support for 32bit will be removed but from Hopper (Ada is supposed to be the last one with this support). But on our RTX 5080 it does not work anymore (again it works on 577.00). Since CUDA failed we had no NVENC available for our users.

We have done a workaround for this by enumerating Nvidia GPUs using DXGI and removing the CUDA part. So this is not a major issue. Just wanted to let you know that this is not working anymore.

Setup used to reproduce the problem:

RTX 5080 running on Windows 11 with 580.xx or newer drivers.

Problem does not occur on all GTX and all RTX from 2000 to 4000 used with the same PC.

2. Cross-process texture sharing.

It’s a major problem which cannot be easily fixed and no workaround exists except using GPU→CPU→GPU data transfer (which we have added today and we finally do not have these display stall/freezes).

So what exactly is going on:

  • D3D12 Game process shares a keyed mutex texture to another process
  • Another process opens the shared resource using it’s handle
  • Another process queries for 2D texture from the shared handle
  • Another process uses AquireSync to get synchronized access to this texture
  • Problem occurs, AquireSync function hags after a few calls
  • D3D12 Game process freezes
  • DirectX functions run on another process stop working and return errors
  • DXGI device used by another process is lost
  • NVENC function called in this state return errors (like mapping texture to NVENC from DirectX)
  • Game process freezes, another process freezes sometimes entire dektop freezes

Removing AquireSync fixes the problem but the texture is black (as assumed since AquireSync must be used for keyed mutex textures).

Setup used to reproduce the problem:

All GTX and RTX GPUs, even old ones.

Workaround: Dump texture data to CPU in D3D12 game process, share the CPU memory and in another process copy CPU data again back to GPU memory. We DO NOT want to use this permanently since this is not optimal especially for 4:4:4 and HDR content.

This problem does not occur for D3D11/10 game process.

Hope these details help you and please let me know if you have more questions that will help in fixing at least the cross-process texture sharing problem as this is critical for us.

Thank you!

7

We have released Action! 4.46.0 that includes a workaround for cross-process D3D12 to D3D11 texture sharing. Version 4.45.0 is no longer available on the official download page.

If you want to use the previous version please use this link to version 4.45.0 which has this problem when running with 580.xx and newer drivers:

downloads.mirillis.com/files/action_4_45_0_setup.exe (please add https:// prefix)

9

To make things clear. The test case for reproducing this problem using Action! 4.45.0:

  • Run Action! 4.45.0 on GTX or RTX with 580 or newer drivers-
  • Run any game or application rendered using D3D12
  • After HUD appears in game saying “D3D12” engine detected enter Action!’s captured video preview
  • Stalling/freezing of game or desktop will start as soon as the game image appears in preview
  • Same test but with D3D11 game/application will not cause display freezing

After rolling back to 577.00 drivers everything works OK.

[UPDATE] Still crashing/stalling/freezing with latest 591.44 drivers :(