UE5.7 DLSS crashes while pixelstreaming if H264 encoder is used

Gaming and Visualization Technologies DLSS
, ,
1

UE5.7 Pixelstreaming crashes with DXGI_ERROR_DEVICE_HUNG when using h264 encoder and DLSS is enabled. vp8, 9 and AV1 works fine. H264 worked fine in UE5.6 with dlss enabled.

Repro steps:

  1. create a new project based on the “intro to unreal” template, enable highend graphics features
  2. Download, add and enable the DLSS superres plugin, and enable the pixelstreaming 2 plugin
  3. Package
  4. run with standard pixelstreaming setup and -renderoffscreen
  5. Run standard pixelstreaming implementation with typescript frontend
  6. Connect via browser, ensure h264 is the encoder in use
  7. wait for crash - may take anything from a few seconds to minutes

crashes silently, often with no crash report(probably TDR). have been able to get crash reports only a few times, and it says Error: GPU crash detected:

  • Device 0 Removed: DXGI_ERROR_DEVICE_HUNG

Confirmed both on AWS g5 and g6 instances. with 4 different nvidia driver versions, including latest.