Unable to determine the device handle for GPU 0000:82:00.0: Unknown Error

When I use my new machine for deep learning experiments, the GPUs often get crashed.

Here is the detailed info of bug report.
nvidia-bug-report.log (8.3 MB)

I have no idea how to solve the problem,and what is problem.Can somebody help me? Thanks a lot!

Gpu: 8x4090
Nvidia Driver Version 550
**CUDNN Version 12.4
ubuntu 22.04

Hi @quatron ,
This Forum talks about issues specific to TRT.
However i will have a look and check if i can provide a possible update to help you.
Thanks