Tensorrt crash when using pytorch simultaneously

Description

On my windows, I can’t make trt inference when import pytorch simultaneously. I made minimal reproducible code as list below.

If i import torch before tensorrt, it will run print('done'). But if import tensorrt first, it will stuck in the for name in engine: after print the input & output name and there is no print('done').

There is a lot of import tensorrt/torch in py file of the repo, I tried to import torch before every import tensorrt but didn’t help.

Is there a best practice using pytorch and tensorrt in the same time on windows?

What I have tried:
I use psutil to check the loaded dll. If import tensorrt first, there will be on more dll called(C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvrtc64_111_0.dll). Hide it from the PATH didn’t. Maybe the issues is the order of load dll?

My $env:PATH have cudnn and cudatoolkit lib path. And pytorch will also download cudatoolkit and cudnn lib. I noticed the __init__.py of tensorrt uses ctypes.CDLL to load cudnn and some cuda libs. So I tried to remove the cudnn and cudatoolkit path from PATH and add the torch lib path(containg cudnn and cuda libs) to my $env:PATH but didn’t help.

Some relevent issues but no answer :
issue1693
issue480

script

import ctypes

import tensorrt as trt
import torch
# import psutil
# import os

# p = psutil.Process( os.getpid() )
# for dll in p.memory_maps():
#    print(dll.path)

# ctypes.CDLL("./build/bin/Release/mmdeploy_tensorrt_ops.dll")

with trt.Logger() as logger, trt.Runtime(logger) as runtime:
    with open("work_dir/trt/model2", mode='rb') as f:
        engine_bytes = f.read()
    engine = runtime.deserialize_cuda_engine(engine_bytes)



for name in engine:
    print(name)


print('done')

Environment

TensorRT Version: 8.2.3.0
GPU Type: GTX2070S
Nvidia Driver Version: 512.59
CUDA Version: 11.1
CUDNN Version: 8.2.1.0
Operating System + Version: windows 10 21H2(19044.1682)
Python Version (if applicable): 3.8.13
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.8.1+cu111
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

main.py (509 Bytes)
model2(Google Drive)

Hi,

Could you please share with us the ONNX model and trtexec command used to generate the engine to try from our end for better debugging.

Thank you.

You can refer to this script to create an onnx model.

main2.py (1.3 KB)

Hi,

We couldn’t reproduce this issue from our end. This looks like your setup-related issue.
Please make sure your env (path) is set correctly. If still face the issue please try reinstalling them(TensorRT) by following the official docs.

Thank you.