Description
Hello
I am trying to verify my TensorRT is working i get the error:
python3
>>> import tensorrt
>>> print(tensorrt.version)
>>> assert tensorrt.Builder(tensorrt.Logger())
[TensorRT] ERROR: CUDA initialization failure with error 222. Please check your CUDA installation: CUDA Installation Guide for Linux
Traceback (most recent call last):
File “”, line 1, in
TypeError: pybind11::init(): factory function returned nullptr
This error is supposed to be "cudaErrorUnsupportedPtxVersion " which im not sure how to solve.
Environment
TensorRT Version: 7.2.1.6
GPU Type:
Nvidia Driver Version: 450.80.02
CUDA Version: 11.0
CUDNN Version:
Operating System + Version: Ubuntu 18.04.5 LTS
Python Version (if applicable): 3.6
TensorFlow Version (if applicable): 1.15.3
PyTorch Version (if applicable): 1.5.0
Baremetal or Container (if container which image + tag):
Hi @daniel.bancsi,
Can you please try running your model using trtexec with verbose, and share the logs with us?
Thanks!
Hi, I met the same issue when I was using tensorrt 7.2 with docker image “nvcr.io/nvidia/tensorrt:20.12-py3”.
But This model is working well on 7.1.
I reinstalled cuda to 11.0 within a docker container (default version is 11.1) but the problem still remains.
I guess @daniel.bancsi may face the same issue.
Environment
TensorRT Version : 7.2.1.6
GPU Type :
Nvidia Driver Version : 450.66
CUDA Version : 11.0
CUDNN Version :
Operating System + Version : Ubuntu 20.04.1 LTS
[01/08/2021-16:48:31] [I] Output:
[01/08/2021-16:48:31] [I] === Build Options ===
[01/08/2021-16:48:31] [I] Max batch: explicit
[01/08/2021-16:48:31] [I] Workspace: 16 MiB
[01/08/2021-16:48:31] [I] minTiming: 1
[01/08/2021-16:48:31] [I] avgTiming: 8
[01/08/2021-16:48:31] [I] Precision: FP32
[01/08/2021-16:48:31] [I] Calibration:
[01/08/2021-16:48:31] [I] Refit: Disabled
[01/08/2021-16:48:31] [I] Safe mode: Disabled
[01/08/2021-16:48:31] [I] Save engine:
[01/08/2021-16:48:31] [I] Load engine:
[01/08/2021-16:48:31] [I] Builder Cache: Enabled
[01/08/2021-16:48:31] [I] NVTX verbosity: 0
[01/08/2021-16:48:31] [I] Tactic sources: Using default tactic sources
[01/08/2021-16:48:31] [I] Input(s)s format: fp32:CHW
[01/08/2021-16:48:31] [I] Output(s)s format: fp32:CHW
[01/08/2021-16:48:31] [I] Input build shapes: model
[01/08/2021-16:48:31] [I] Input calibration shapes: model
[01/08/2021-16:48:31] [I] === System Options ===
[01/08/2021-16:48:31] [I] Device: 0
[01/08/2021-16:48:31] [I] DLACore:
[01/08/2021-16:48:31] [I] Plugins:
[01/08/2021-16:48:31] [I] === Inference Options ===
[01/08/2021-16:48:31] [I] Batch: Explicit
[01/08/2021-16:48:31] [I] Input inference shapes: model
[01/08/2021-16:48:31] [I] Iterations: 10
[01/08/2021-16:48:31] [I] Duration: 3s (+ 200ms warm up)
[01/08/2021-16:48:31] [I] Sleep time: 0ms
[01/08/2021-16:48:31] [I] Streams: 1
[01/08/2021-16:48:31] [I] ExposeDMA: Disabled
[01/08/2021-16:48:31] [I] Data transfers: Enabled
[01/08/2021-16:48:31] [I] Spin-wait: Disabled
[01/08/2021-16:48:31] [I] Multithreading: Disabled
[01/08/2021-16:48:31] [I] CUDA Graph: Disabled
[01/08/2021-16:48:31] [I] Separate profiling: Disabled
[01/08/2021-16:48:31] [I] Skip inference: Disabled
[01/08/2021-16:48:31] [I] Inputs:
[01/08/2021-16:48:31] [I] === Reporting Options ===
[01/08/2021-16:48:31] [I] Verbose: Enabled
[01/08/2021-16:48:31] [I] Averages: 10 inferences
[01/08/2021-16:48:31] [I] Percentile: 99
[01/08/2021-16:48:31] [I] Dump refittable layers:Disabled
[01/08/2021-16:48:31] [I] Dump output: Disabled
[01/08/2021-16:48:31] [I] Profile: Disabled
[01/08/2021-16:48:31] [I] Export timing to JSON file:
[01/08/2021-16:48:31] [I] Export output to JSON file:
[01/08/2021-16:48:31] [I] Export profile to JSON file:
[01/08/2021-16:48:31] [I]
[01/08/2021-16:48:31] [I] === Device Information ===
[01/08/2021-16:48:31] [I] Selected Device: GeForce RTX 2080 Ti
[01/08/2021-16:48:31] [I] Compute Capability: 7.5
[01/08/2021-16:48:31] [I] SMs: 68
[01/08/2021-16:48:31] [I] Compute Clock Rate: 1.545 GHz
[01/08/2021-16:48:31] [I] Device Global Memory: 11019 MiB
[01/08/2021-16:48:31] [I] Shared Memory per SM: 64 KiB
[01/08/2021-16:48:31] [I] Memory Bus Width: 352 bits (ECC disabled)
[01/08/2021-16:48:31] [I] Memory Clock Rate: 7 GHz
[01/08/2021-16:48:31] [I]
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Proposal version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Split version 1
[01/08/2021-16:48:31] [E] [TRT] CUDA initialization failure with error 222. Please check your CUDA installation: CUDA Installation Guide for Linux
[01/08/2021-16:48:31] [E] Builder creation failed
[01/08/2021-16:48:31] [E] Engine creation failed
[01/08/2021-16:48:31] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # trtexec --onnx= my_model
Hi @Martin.ZJ ,
Can you please check if other CUDA application works and update us?
Thanks!
I ran that on k8s virtiual machine and I deleted that VM (ノへ￣、). Now I am using TensorRT 7.1. I will update here when I reproduce this problem.
Hi, I have the same problem with Martin.ZJ, in ubuntu 18.04 in aws machine, where there are 5 cuda+cudnn versions preinstalled by aws. both the following configuration failed:
(1) cuda 11.1+ TensorRT7.2.2
(2) cuda 10.0 + TensorRT7.0.0
Now I’am trying cuda 10.2+TensorRT7.1.3.4, but not sure it works.
Do you have any suggestion?
I solved this problem with:
driver 450.80.02 +cuda 11.1 (with cudnn 8.0 installed in cuda-11.1/lib64) + TensorRT7.2.2.3 for cuda 11.1 and cudnn 8.0 ,
and this combination is obviously not the only feasible one.
For me, there are three tips:
1.cudnn version in cuda lib64 directory must consist with cudnn version of TensorRT(you can find this in TensorRT install file name)
2.you’d better run all your commands in pure terminal not VScode terminal (enrionment variables in the two are probably different), especially you change your cuda and TensorRT configuration frequently to find out a feasible set of configuration
3. to run trtexec, you must add “sudo”, otherwise you may get an error of “cannot open engine file”