Convert tensorrt engine from version 7 to 8

jothi.ramasubramanian · May 6, 2021, 4:39am

Hi,
I have tensorRT(FP32) engine model for inference which is converted using tlt-convertor in TLT version 2. Now i have a python script to inference trt engine. and i installed tensorrt in virtual environment with using this command
pip3 install nvidia-tensorrt.
It installed tensorrt version 8.0.0.3 now i trying to inference the same tensorRT engine file with tensorrt 8.0.0.3.on that time i got this error

[TensorRT] VERBOSE: Registered plugin creator - ::GridAnchor_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::GridAnchorRect_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::NMS_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Reorg_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Region_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Clip_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::LReLU_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::PriorBox_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Normalize_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ScatterND version 1
[TensorRT] VERBOSE: Registered plugin creator - ::RPROI_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::BatchedNMS_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::FlattenConcat_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::CropAndResize version 1
[TensorRT] VERBOSE: Registered plugin creator - ::DetectionLayer_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Proposal version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ProposalLayer_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::ResizeNearest_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::Split version 1
[TensorRT] VERBOSE: Registered plugin creator - ::SpecialSlice_TRT version 1
[TensorRT] VERBOSE: Registered plugin creator - ::InstanceNormalization_TRT version 1
[TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +303, GPU +0, now: CPU 322, GPU 526 (MiB)
[TensorRT] INFO: Loaded engine size: 4 MB
[TensorRT] INFO: [MemUsageSnapshot] deserializeCudaEngine begin: CPU 325 MiB, GPU 526 MiB
[TensorRT] ERROR: 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 89)
[TensorRT] ERROR: 4: [runtime.cpp::deserializeCudaEngine::74] Error Code 4: Internal Error (Engine deserialization failed.)

I used this code to load trt engine
TRT_LOGGER = trt.Logger(trt.Logger.VERBOSE)
trt.init_libnvinfer_plugins(TRT_LOGGER, ‘’)
trt_runtime = trt.Runtime(TRT_LOGGER)

with open(engine_path, “rb”) as f:
engine_data = f.read()

engine = trt_runtime.deserialize_cuda_engine(engine_data)

Is that version mismatch problem?

Morganh · May 6, 2021, 7:24am

Yes, please generate trt engine again.

jothi.ramasubramanian · May 6, 2021, 7:27am

Ok I tried with this

tlt tlt-converter -k $KEY
-d 3,320,480
-o BatchedNMS
-e /home/recode/TLT3/trt.engine
-m 16
-t fp32
-i nchw
/yolo_mobilenet_v2_epoch_010.etlt

But i got error like this
2021-05-06 12:53:56,810 [INFO] root: No mount points were found in the /home/recode/.tlt_mounts.json file.
2021-05-06 12:53:56,810 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the ~/.tlt_mounts.json file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
2021-05-06 12:53:57,609 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
(launcher) recode@RecodePC:~$ /home/.tlt_mounts.json

Morganh · May 6, 2021, 7:31am

The path is not correct.
Please read TLT Launcher — Transfer Learning Toolkit 3.0 documentation and then map your local directory to docker.

The Mounts parameter defines the paths in the local machine, that should be mapped to the docker. This is a list of json dictionaries containing the source path in the local machine and the destination path that is mapped for the TLT commands.

jothi.ramasubramanian · May 6, 2021, 7:33am

Thank you @Morganh
where should i keep this tlt_mounts.json file

Morganh · May 6, 2021, 7:36am

The launcher instance can be configured in the ~/.tlt_mounts.json file.

jothi.ramasubramanian · May 6, 2021, 7:48am

@Morganh
where could i find this json file? and how can i edit these path?

Morganh · May 6, 2021, 8:04am

Create it by yourself.

jothi.ramasubramanian · May 6, 2021, 9:10am

Yes I created the json and updated the path. then ran this
tlt tlt-converter -k $KEY
-d 3,960,1472
-o BatchedNMS
-e /home/recode/TLT3/trt.engine
-m 16
-t fp32
-i nchw
/yolo_mobilenet_v2_epoch_010.etlt

now it showing like this

2021-05-06 14:37:04,065 [INFO] root: No mount points were found in the /home/recode/.tlt_mounts.json file.
2021-05-06 14:37:04,065 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the ~/.tlt_mounts.json file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Error: no input dimensions given
2021-05-06 14:37:04,786 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Morganh · May 6, 2021, 9:13am

Can you paste the full command and full log?

jothi.ramasubramanian · May 6, 2021, 9:13am

(launcher) recode@RecodePC:~/tlt_cv_samples_v1.0.2$ tlt tlt-converter -k $KEY -d 3,960,1472 -o BatchedNMS -e /home/recode/TLT3/trt.engine -m 16 -t fp32 -i nchw /media/recode/DATA9/Jetson/tlt-experiments/install-docs-old/support_scripts/TRT_INFERENCE/fp16/yolo_mobilenet_v2_epoch_010.etlt
2021-05-06 14:37:04,065 [INFO] root: No mount points were found in the /home/recode/.tlt_mounts.json file.
2021-05-06 14:37:04,065 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the ~/.tlt_mounts.json file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Error: no input dimensions given
2021-05-06 14:37:04,786 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

jothi.ramasubramanian · May 6, 2021, 9:15am

This is my Json file

{
“Mounts”: [
{
“source”: “/home/recode/tlt_cv_samples_v1.0.2”,
“destination”: “/workspace/tlt-experiments/data”
},
{
“source”: “/home/recode/tlt_cv_samples_v1.0.2”,
“destination”: “/workspace/tlt-experiments/results”
},
{
“source”: “/home/recode/tlt_cv_samples_v1.0.2/yolo_v3/specs”,
“destination”: “/workspace/tlt-experiments/specs”
}
],
“Envs”: [
{
“variable”: “CUDA_DEVICE_ORDER”,
“value”: “PCI_BUS_ID”
}
],
“DockerOptions”: {
“shm_size”: “16G”,
“ulimits”: {
“memlock”: -1,
“stack”: 67108864
},
“user”: “1000:1000”
}
}

Morganh · May 6, 2021, 9:17am

Please note that the path after command “tlt” should be a path inside the docker. So, please modify your command line. For example, modify /home/recode/TLT3/trt.engine to something like that /workspace/tlt-experiments/… But in your json file, I find that you did not map /home/recode/TLT3. So, please modify json file too.

jothi.ramasubramanian · May 6, 2021, 9:21am

Sure Thank you

jothi.ramasubramanian · May 6, 2021, 9:46am

(launcher) recode@RecodePC:~/TLT3$ tlt tlt-converter -k $KEY -d 3,320,480 -o BatchedNMS -e /workspace/tlt-experiments/trt.engine -m 16 -t fp32 -i nchw /workspace/tlt-experiments/fp16/yolo_mobilenet_v2_epoch_010.etlt
2021-05-06 15:10:10,772 [INFO] root: No mount points were found in the /home/recode/.tlt_mounts.json file.
2021-05-06 15:10:10,772 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the ~/.tlt_mounts.json file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
[ERROR] UffParser: Unsupported number of graph 0
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[ERROR] Network must have at least one output
[ERROR] Network validation failed.
[ERROR] Unable to create engine
2021-05-06 15:10:21,485 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Now these error came.The etlt model is trained in TLT 2.0. Is that problem?

Morganh · May 6, 2021, 9:52am

Please check the key.
What is the $KEY? You can input it explicitly.

jothi.ramasubramanian · May 6, 2021, 9:53am

Yea Before i use i exported like this
export KEY=ZmpsbnVjNmJpbDdjdnAxYTHNdWViYTVsaXU6NDQwMzg3OWQtODQ2MS00YjNiLWEwNGEtZmVkZDdhZWUyY2U5

jothi.ramasubramanian · May 6, 2021, 6:18pm

@Morganh
Actually Now i can able to convert tensorrt engine with tlt-convertor. Then i did inference with python script Its again showing the same error.

[TensorRT] INFO: Loaded engine size: 76 MB
[TensorRT] INFO: [MemUsageSnapshot] deserializeCudaEngine begin: CPU 307 MiB, GPU 648 MiB
[TensorRT] ERROR: 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96)
[TensorRT] ERROR: 4: [runtime.cpp::deserializeCudaEngine::74] Error Code 4: Internal Error (Engine deserialization failed.)

Morganh · May 7, 2021, 1:20am

Where did you generate the tensorrt engine?
And where did you run inference?
If you run inference in nano, please generate trt engine directly in nano with the tlt-converter(Jetson version).

jothi.ramasubramanian · May 7, 2021, 4:09am

I am working on x86_64 machine.
I created tensorRT engine on TLT version 2.0. and I can able to run inference inside TLT docker with my python script.
Now I want to run the same inference script in conda environment.In conda environment I installed tensorRT,pycuda, pyindex everything with the command

pip3 install nvidia-pyindex
pip3 install nvidia-tensorrt
pip install pycuda

So when i run the same inference script in conda environment I got this error. How can i slove this? Please help me

[TensorRT] INFO: Loaded engine size: 76 MB
[TensorRT] INFO: [MemUsageSnapshot] deserializeCudaEngine begin: CPU 307 MiB, GPU 648 MiB
[TensorRT] ERROR: 1: [stdArchiveReader.cpp::StdArchiveReader::34] Error Code 1: Serialization (Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 96)
[TensorRT] ERROR: 4: [runtime.cpp::deserializeCudaEngine::74] Error Code 4: Internal Error (Engine deserialization failed.)

Topic		Replies	Views
Error Code 1: Serialization (Serialization assertion stdVersionRead == kSERIALIZATION_VERSION failed.Version tag does not match TAO Toolkit	20	2146	June 11, 2024
Inferring detectnet_v2 .trt model in python TAO Toolkit tensorrt	58	3572	August 17, 2021
The effect is very poor when converted to trt TAO Toolkit tensorrt , ubuntu	61	1299	September 11, 2023
Error while building TensorRT OSS 8.0.1 TensorRT	29	3315	July 16, 2021
TF-TRT issue Jetson TX2	26	3830	October 18, 2021
Cuda initialization failure when converting trt model with different GPU TensorRT tensorrt	7	6421	September 28, 2022
Tensorrt fails for custom ssd_inception Model TensorRT	18	2803	May 14, 2020
Turbocharging Meta Llama 3 Performance with NVIDIA TensorRT-LLM and NVIDIA Triton Inference Server Technical Blog	62	3559	August 28, 2024
TensorRT 3.0 RC now available with support for TensorFlow Jetson TX2	83	23207	May 21, 2018
Error deserializing trt engine when bundling with Pyinstaller Jetson AGX Xavier tensorrt	8	1432	October 18, 2021

Convert tensorrt engine from version 7 to 8

Related topics