Hello, I am having the following error while running inference on a trt engine The engine file is for object detection model ‘ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8’ from tensorflow model zoo. Error: [TensorRT] ERROR: 2: [pluginV2DynamicExtRunner.cpp::execute::115] Error Code 2: Internal Error (Assertion status == kSTATUS_SUCCESS failed.) Environment TensorRT version: 8.0.1.6 onnx version: 1.10.2 Jetpack: 4.6 Relevant Files I have uploaded my onnx and trt engine file for the model, and the script I am using for inference. And I follow the code given in the link: Speeding Up Deep Learning Inference Using TensorFlow, ONNX, and NVIDIA TensorRT | NVIDIA Technical Blog to convert onnx to trt file. inference.py (5.2 KB) model.trt (12.4 MB) model.onnx (10.4 MB) Please have a look. Thank you.

Hi, Thanks for your patience. We have a new release for the Jetson platform recently. Do you mind checking this with JetPack 4.6.1 + TensorRT 8.2 to see if it works? Thanks.

Hi, Do you convert the model into ONNX with tf2onnx? Could you share the detailed steps with us? Thanks.

Hello, Yes I used tf2onxx for conversion of saved graph to the onnx format. The following command with opset 11 was used for conversion: python -m tf2onnx.convert --saved-model tensorflow-model-path --opset 11 --output model.onnx And the following code was used to create tensorrt engine from the onnx file. This code was available on one of the nvidia jetson nano forum regarding conversion to tensorrt engine. engine.py (1.0 KB) create_engine.py (692 Bytes)

Hi, The model from TF2 Object Detection API requires some customization. Could you follow the tutorial below to convert the model into TensorRT: TensorRT/samples/python/tensorflow_object_detection_api at main ·... NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. - NVIDIA/TensorRT Thanks.

Hi, Confirmed that we can convert the ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 model into TensorRT on Jetson. Below is the detailed steps for your reference: 1. Environment $ sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-tensorflow:r32.6.1-tf2.5-py3 2. Install Prerequisites $ apt-get update $ apt-get install cmake g++ git libprotobuf-dev protobuf-compiler $ pip3 install onnx tf2onnx pillow $ git clone https://github.com/NVIDIA/TensorRT.git $ cd TensorRT/tools/onnx-graphsurgeon/ $ make install $ python3 -m pip install --force-reinstall dist/*.whl $ wget https://nvidia.box.com/shared/static/jy7nqva7l88mq9i8bw3g3sklzf4kccn2.whl -O onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl $ pip3 install onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl 3. Prepare Source $ cd ../../samples/python/tensorflow_object_detection_api/ $ git clone https://github.com/tensorflow/models.git $ cp -r models/research/object_detection . $ protoc object_detection/protos/*.proto --python_out=. Apply below change: diff --git a/samples/python/tensorflow_object_detection_api/create_onnx.py b/samples/python/tensorflow_object_detection_api/create_onnx.py index b6ac423..7292756 100644 --- a/samples/python/tensorflow_object_detection_api/create_onnx.py +++ b/samples/python/tensorflow_object_detection_api/create_onnx.py @@ -254,8 +254,8 @@ class TFODGraphSurgeon: concat_node.outputs = [] # Disconnect the last node in second preprocessing branch with parent second TensorListStack node. - tile_node = self.graph.find_node_by_op("Tile") - tile_node.outputs = [] + #tile_node = self.graph.find_node_by_op("Tile") + #tile_node.outputs = [] # Reshape nodes tend to update the batch dimension to a fixed value of 1, they should use the batch size instead. for node in [node for node in self.graph.nodes if node.op == "Reshape"]: 4. Convert Model $ wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz . $ tar -xvf ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz $ python3 create_onnx.py --pipeline_config ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/pipeline.config --saved_model ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model --onnx model.onnx --input_format NCHW $ python3 build_engine.py --onnx model.onnx --engine engine.trt --precision fp16 5. Test with TensorRT $ /usr/src/tensorrt/bin/trtexec --loadEngine=engine.trt ... [12/29/2021-08:40:07] [I] === Performance summary === [12/29/2021-08:40:07] [I] Throughput: 146.935 qps [12/29/2021-08:40:07] [I] Latency: min = 6.73804 ms, max = 8.90845 ms, mean = 6.79419 ms, median = 6.77783 ms, percentile(99%) = 7.34839 ms [12/29/2021-08:40:07] [I] End-to-End Host Latency: min = 6.7522 ms, max = 8.92114 ms, mean = 6.80563 ms, median = 6.78809 ms, percentile(99%) = 7.36255 ms [12/29/2021-08:40:07] [I] Enqueue Time: min = 4.5791 ms, max = 8.78735 ms, mean = 5.32989 ms, median = 5.07129 ms, percentile(99%) = 6.9624 ms [12/29/2021-08:40:07] [I] H2D Latency: min = 0.0742188 ms, max = 0.0761719 ms, mean = 0.0748875 ms, median = 0.0749512 ms, percentile(99%) = 0.0759277 ms [12/29/2021-08:40:07] [I] GPU Compute Time: min = 6.65723 ms, max = 8.8291 ms, mean = 6.71306 ms, median = 6.69604 ms, percentile(99%) = 7.26758 ms [12/29/2021-08:40:07] [I] D2H Latency: min = 0.00512695 ms, max = 0.00732422 ms, mean = 0.00624302 ms, median = 0.00634766 ms, percentile(99%) = 0.00708008 ms [12/29/2021-08:40:07] [I] Total Host Walltime: 1.09572 s [12/29/2021-08:40:07] [I] Total GPU Compute Time: 1.0808 s [12/29/2021-08:40:07] [I] Explanations of the performance metrics are printed in the verbose logs. [12/29/2021-08:40:07] [I] &&&& PASSED TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --loadEngine=engine.trt [12/29/2021-08:40:07] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1377, GPU 16207 (MiB) Thanks.

Hello, I tried the solution you mentioned above and tried to debug the error but its still the same. The onnx file was created successfully but during the engine generation there is an error. I have provided the log file for your reference. Please have a look. log_file.txt (2.2 MB) Thank you.

Hi, Do you use the ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 model from TensorFlow model zoo? Or a customized version? We have tested the model and it can work correctly with the above instructions. If the default version is used, would you mind trying it again with some swap memory? [image] Trt_convert converter.convert() gets killed without errors Jetson Xavier NX Hi, Would you mind to create some swap space to see if help? You will need an extra space for the swap file. Ex. sudo fallocate -l 8G [/media/mySSD/swapfile] sudo chmod 600 [/media/mySSD/swapfile] sudo mkswap [/media/mySSD/swapfile] sudo /bin/sh -c 'echo "[/media/mySSD/swapfile] \t none \t swap \t defaults \t 0 \t 0" >> /etc/fstab' sudo swapon -a Thanks. Thanks.

Hello, I am using ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 model fine-tuned for my own dataset with 3 classes. As for the swap memory I am already using 8GB of it. While debugging the error, I found some other people are having the same error: https://github.com/pskiran1/TensorRT-support-for-Tensorflow-2-Object-Detection-Models/issues/6 Can you please have a look? Thank you.

Hi, Would you mind sharing your model with us so we can give it a check? You may need some updates if a customized model is used. Thanks.

ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.zip (19.1 MB) The file contains model as well as config file. Thank you.

Hi, Thanks for sharing the model. We try to convert your model with create_onnx.py script. But meet the following error: This ORT build has ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instantiating InferenceSession. For example, onnxruntime.InferenceSession(..., providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'], ...) INFO:ModelHelper:Found Concat node 'StatefulPartitionedCall/concat_1' as the tip of WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalClassHead INFO:ModelHelper:Found Concat node 'StatefulPartitionedCall/concat' as the tip of WeightSharedConvolutionalBoxPredictor/WeightSharedConvolutionalBoxHead Traceback (most recent call last): File "create_onnx.py", line 671, in main(args) File "create_onnx.py", line 647, in main effdet_gs.process_graph(args.first_nms_threshold, args.second_nms_threshold) File "create_onnx.py", line 620, in process_graph self.graph.outputs = first_nms(-1, True, first_nms_threshold) File "create_onnx.py", line 485, in first_nms anchors_tensor = self.extract_anchors_tensor(box_net_split) File "create_onnx.py", line 311, in extract_anchors_tensor anchors_y = get_anchor(0, "Add") File "create_onnx.py", line 300, in get_anchor if (node.inputs[1].values).size == 1: AttributeError: 'Variable' object has no attribute 'values' It seems that you can convert the ONNX model successfully. Have you applied any customization based on your model? Thanks.

Inference error while using tensorrt engine on jetson nano

Robotics & Edge Computing Jetson Systems Jetson Nano

AastaLLL January 3, 2022, 4:12am 10

Hi,

Do you use the ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 model from TensorFlow model zoo?
Or a customized version?

We have tested the model and it can work correctly with the above instructions.
If the default version is used, would you mind trying it again with some swap memory?

Thanks.

Topic		Replies	Views
Inference error while using tensorrt engine on jetson nano TensorRT tensorrt	3	1931	February 1, 2022
TensorRT Inference error on Jetson nano Jetson Nano tensorrt	27	3619	January 18, 2022
Conversion from tensorrt to engine fails TensorRT tensorrt , cuda , jetson-inference	1	733	January 4, 2022
Problem converting tensorflow model to TensorRT Jetson Nano tensorrt , tensorflow	4	592	March 1, 2024
I do not get any performance improvement after using TensorRT provider for object detection model Jetson Nano tensorrt , onnx	6	1661	July 12, 2022
I am trying to convert the ONNX SSD mobilnet v2 model into TensorRT Engine. I am getting the below error Jetson AGX Xavier tensorrt , jetson	7	1123	November 15, 2021
I am trying to convert the ONNX SSD mobilnet v3 model into TensorRT Engine. I am getting the below error Jetson TX2 tensorrt , tensorflow	23	4520	February 17, 2022
How to infer using tensorRT on jetson nano? Jetson Nano tensorrt , deep-learning	3	1259	January 11, 2021
TensorRT Inference error on Jetson nano TensorRT	3	1336	December 6, 2021
Tlt-convert for custom trained YoloV4 model failed on Jetson Nano 4G TAO Toolkit	41	3041	June 28, 2021

Inference error while using tensorrt engine on jetson nano

Related topics