Using YOLOV8 with Deepstream

[user93959]

I trained a YOLOv8 model on a single class and exported to engine format, then I modified the model-engine-file parameter in the source1_usb_dec_infer_resnet_int8.txt file, the remaining parameters remained untouched, I use the following command to run:

./deepstream-app -c /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/source1_usb_dec_infer_resnet_int8.txt 

I get the following errors at the top:

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
ERROR: [TRT]: 1: [runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
ERROR: Deserialize engine failed from file: /home/souf/Desktop/best.engine
0:00:06.587444562 12176 0xaaaabdda22d0 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2083> [UID = 1]: deserialize engine from file :/home/souf/Desktop/best.engine failed
0:00:07.020572498 12176 0xaaaabdda22d0 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2188> [UID = 1]: deserialize backend context from engine from file :/home/souf/Desktop/best.engine failed, try rebuild
0:00:07.020646165 12176 0xaaaabdda22d0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
WARNING: [TRT]: Missing scale and zero-point for tensor output_bbox/bias, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor conv1/kernel, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor conv1/bias, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor bn_conv1/moving_variance, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor bn_conv1/Reshape_1/shape, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor bn_conv1/batchnorm/add/y, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor bn_conv1/gamma, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
// the warnings are duplicated many times before app is launched

The app still launched, but with a different model because it was detecting trees and people etc…, my model only trained on cars.

I read that the error could be because of writing permissions, so I ran a second time with sudo, and I got the exact same behavior as above, but different errors:

(gst-plugin-scanner:12317): GStreamer-WARNING **: 20:36:31.579: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory

(gst-plugin-scanner:12317): GStreamer-WARNING **: 20:36:31.594: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:12317): GStreamer-WARNING **: 20:36:32.976: adding type GstEvent multiple times
Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
ERROR: [TRT]: 1: [runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
ERROR: Deserialize engine failed from file: /home/souf/Desktop/best.engine
0:00:09.398626323 12315 0xaaaadd3e04d0 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2083> [UID = 1]: deserialize engine from file :/home/souf/Desktop/best.engine failed
0:00:09.826965524 12315 0xaaaadd3e04d0 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2188> [UID = 1]: deserialize backend context from engine from file :/home/souf/Desktop/best.engine failed, try rebuild
0:00:09.827029207 12315 0xaaaadd3e04d0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
WARNING: [TRT]: Missing scale and zero-point for tensor output_bbox/bias, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor conv1/kernel, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor conv1/bias, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor bn_conv1/moving_variance, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
WARNING: [TRT]: Missing scale and zero-point for tensor bn_conv1/Reshape_1/shape, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
... WARNINGS duplicated many times

I flashed with Jetpack 6.0, DeepStream 7

Please let me know if I need to provide any other details, any help is appreciated.

[fanzh]

The TensorRT engine is bound to the GPU. the engine created on other GPU is invalid on a new GPU. please refer to this code for how to create a new engine. after engine is generated, you can set model-engine-file to the engine path.

[user93959]

Thank you for your reply @fanzh

The model was trained in Google Colab, but exported to engine format within the same Jetson device in question, does that count as different GPUs?

Another question, does training then exporting a YOLO model in a Jetson device have the same effect as training using TAO toolkit commands? Same effects meaning same performance, accuracy etc…

[fanzh]

1. about “Google Colab”, does it use a different TRT version? from the error “Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)”, deepstream failed to load the engine.
2. the second question would be outside of DeepStream. it generating model by two framework, maybe the accuracy and performance are different.

[user93959]

I trained the model using ultralytics in Google colab, at that point there is no TRT, just a regular YOLO .pt file, when I copied the model to the Jetson device I used ultralytics command line to convert it to engine, the last command requires TensorRT installed.

[fanzh]

could you share how did you convert it to engine by TensorRT? please refer to topic1 and topic2, please try this solution.

[fanzh]

please refer to the solution in my last comment. we suggest using DeepStream to generate engine instead of using ultralytics. you can set onnx model in nvinfer’s configuration file.

[user93959]

This did it! Thanks a lot!