Trying to run TensorRT project on AWS, getting error

Description

Hello
I am trying to verify my TensorRT is working i get the error:

python3
>>> import tensorrt
>>> print(tensorrt.version)
>>> assert tensorrt.Builder(tensorrt.Logger())

[TensorRT] ERROR: CUDA initialization failure with error 222. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
Traceback (most recent call last):
File “”, line 1, in
TypeError: pybind11::init(): factory function returned nullptr

This error is supposed to be "cudaErrorUnsupportedPtxVersion " which im not sure how to solve.

Environment

TensorRT Version: 7.2.1.6
GPU Type:
Nvidia Driver Version: 450.80.02
CUDA Version: 11.0
CUDNN Version:
Operating System + Version: Ubuntu 18.04.5 LTS
Python Version (if applicable): 3.6
TensorFlow Version (if applicable): 1.15.3
PyTorch Version (if applicable): 1.5.0
Baremetal or Container (if container which image + tag):

Hi @daniel.bancsi,
Can you please try running your model using trtexec with verbose, and share the logs with us?

Thanks!

Hi, I met the same issue when I was using tensorrt 7.2 with docker image “nvcr.io/nvidia/tensorrt:20.12-py3”.

But This model is working well on 7.1.

I reinstalled cuda to 11.0 within a docker container (default version is 11.1) but the problem still remains.

I guess @daniel.bancsi may face the same issue.

Environment

TensorRT Version : 7.2.1.6
GPU Type :
Nvidia Driver Version : 450.66
CUDA Version : 11.0
CUDNN Version :
Operating System + Version : Ubuntu 20.04.1 LTS

[01/08/2021-16:48:31] [I] Output:
[01/08/2021-16:48:31] [I] === Build Options ===
[01/08/2021-16:48:31] [I] Max batch: explicit
[01/08/2021-16:48:31] [I] Workspace: 16 MiB
[01/08/2021-16:48:31] [I] minTiming: 1
[01/08/2021-16:48:31] [I] avgTiming: 8
[01/08/2021-16:48:31] [I] Precision: FP32
[01/08/2021-16:48:31] [I] Calibration:
[01/08/2021-16:48:31] [I] Refit: Disabled
[01/08/2021-16:48:31] [I] Safe mode: Disabled
[01/08/2021-16:48:31] [I] Save engine:
[01/08/2021-16:48:31] [I] Load engine:
[01/08/2021-16:48:31] [I] Builder Cache: Enabled
[01/08/2021-16:48:31] [I] NVTX verbosity: 0
[01/08/2021-16:48:31] [I] Tactic sources: Using default tactic sources
[01/08/2021-16:48:31] [I] Input(s)s format: fp32:CHW
[01/08/2021-16:48:31] [I] Output(s)s format: fp32:CHW
[01/08/2021-16:48:31] [I] Input build shapes: model
[01/08/2021-16:48:31] [I] Input calibration shapes: model
[01/08/2021-16:48:31] [I] === System Options ===
[01/08/2021-16:48:31] [I] Device: 0
[01/08/2021-16:48:31] [I] DLACore:
[01/08/2021-16:48:31] [I] Plugins:
[01/08/2021-16:48:31] [I] === Inference Options ===
[01/08/2021-16:48:31] [I] Batch: Explicit
[01/08/2021-16:48:31] [I] Input inference shapes: model
[01/08/2021-16:48:31] [I] Iterations: 10
[01/08/2021-16:48:31] [I] Duration: 3s (+ 200ms warm up)
[01/08/2021-16:48:31] [I] Sleep time: 0ms
[01/08/2021-16:48:31] [I] Streams: 1
[01/08/2021-16:48:31] [I] ExposeDMA: Disabled
[01/08/2021-16:48:31] [I] Data transfers: Enabled
[01/08/2021-16:48:31] [I] Spin-wait: Disabled
[01/08/2021-16:48:31] [I] Multithreading: Disabled
[01/08/2021-16:48:31] [I] CUDA Graph: Disabled
[01/08/2021-16:48:31] [I] Separate profiling: Disabled
[01/08/2021-16:48:31] [I] Skip inference: Disabled
[01/08/2021-16:48:31] [I] Inputs:
[01/08/2021-16:48:31] [I] === Reporting Options ===
[01/08/2021-16:48:31] [I] Verbose: Enabled
[01/08/2021-16:48:31] [I] Averages: 10 inferences
[01/08/2021-16:48:31] [I] Percentile: 99
[01/08/2021-16:48:31] [I] Dump refittable layers:Disabled
[01/08/2021-16:48:31] [I] Dump output: Disabled
[01/08/2021-16:48:31] [I] Profile: Disabled
[01/08/2021-16:48:31] [I] Export timing to JSON file:
[01/08/2021-16:48:31] [I] Export output to JSON file:
[01/08/2021-16:48:31] [I] Export profile to JSON file:
[01/08/2021-16:48:31] [I]
[01/08/2021-16:48:31] [I] === Device Information ===
[01/08/2021-16:48:31] [I] Selected Device: GeForce RTX 2080 Ti
[01/08/2021-16:48:31] [I] Compute Capability: 7.5
[01/08/2021-16:48:31] [I] SMs: 68
[01/08/2021-16:48:31] [I] Compute Clock Rate: 1.545 GHz
[01/08/2021-16:48:31] [I] Device Global Memory: 11019 MiB
[01/08/2021-16:48:31] [I] Shared Memory per SM: 64 KiB
[01/08/2021-16:48:31] [I] Memory Bus Width: 352 bits (ECC disabled)
[01/08/2021-16:48:31] [I] Memory Clock Rate: 7 GHz
[01/08/2021-16:48:31] [I]
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Proposal version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[01/08/2021-16:48:31] [V] [TRT] Registered plugin creator - ::Split version 1
[01/08/2021-16:48:31] [E] [TRT] CUDA initialization failure with error 222. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
[01/08/2021-16:48:31] [E] Builder creation failed
[01/08/2021-16:48:31] [E] Engine creation failed
[01/08/2021-16:48:31] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # trtexec --onnx= my_model

Hi @Martin.ZJ ,
Can you please check if other CUDA application works and update us?

Thanks!

I ran that on k8s virtiual machine and I deleted that VM (ノへ ̄、). Now I am using TensorRT 7.1. I will update here when I reproduce this problem.