RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

Description

It used to work very well, but suddenly went wrong today.
RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable

Environment

GPU Type: 3090
Nvidia Driver Version: 460.32.03
CUDA Version: 11.2
CUDNN Version: 8.1.0
Operating System + Version:
Python Version (if applicable): 3.6.13
PyTorch Version (if applicable): 1.71

Hi @ryanyuheng,

It doesn’t seems to be TRT related issue.
Request you to please post this query in below forum:

Thanks