Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc): GPU
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc): Yolo_v4
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here): NA
• Training spec file(If have, please share here): NA
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.): NA
I need to deploy a Yolo_v4 model trained using TAO 3.0 in Deepstream 8.0, since tao-converter is deprecated and the exported file is .etlt, I used nvinfer to build the engine but I got this error:
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1261 Deserialize engine failed because file path: /home/me/detector/detector.engine open error
0:00:00.321296740 3265927 0x5b20590ec6e0 WARN nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<detector> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2097> [UID = 1]: deserialize engine from file :/home/me/detector/detector.engine failed
0:00:00.321311770 3265927 0x5b20590ec6e0 WARN nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<detector> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2202> [UID = 1]: deserialize backend context from engine from file :/home/me/detector/detector.engine failed, try rebuild
0:00:00.321319070 3265927 0x5b20590ec6e0 INFO nvinfer gstnvinfer.cpp:685:gst_nvinfer_logger:<detector> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2123> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnxOpImporters.cpp:6521: Attribute caffeSemantics not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
WARNING: [TRT]: BatchedNMSPlugin is deprecated since TensorRT 9.0. Use INetworkDefinition::addNMS() to add an INMSLayer OR use EfficientNMS plugin.
WARNING: [TRT]: Calibration Profile is not defined. Calibrating with Profile 0
ERROR: [TRT]: Unexpected exception _Map_base::at
Segmentation fault (core dumped)
It seems that the model was exported with BatchedNMSPlugin which is deprecated since TensorRT 9.0 and Deepstream 8.0 uses TensorRT 10.9 so nvinfer cannot build an engine for this .etltfile.
Can I re-export the original .tlt file (trained using TAO 3.0) to .onnx using TAO 5.0? If not, how do I deploy a model trained using TAO 3.0 to Deepstream 8.0?
Please try either of below options.
- Please try to change the .etlt file to .onnx file. Reference: tao_toolkit_recipes/tao_forum_faq/FAQ.md at main · NVIDIA-AI-IOT/tao_toolkit_recipes · GitHub.
- Please try to change the .tlt file to .hdf5 file. Reference: tao_toolkit_recipes/tao_forum_faq/FAQ.md at main · NVIDIA-AI-IOT/tao_toolkit_recipes · GitHub. And then export to .onnx file using TAO5.0.
@Morganh
After I convert the .etlt file to .onnx file and use trtexec to build an int8 engine from the .onnx file.
The .onnx file was converted from a .etlt file exported using TAO 3.0 following the link you provided, the calibration file is for the .etlt file, I reuse the same the calibration file for the .onnx file to build engine file.
I got this error:
[11/07/2025-13:22:05] [I] [TRT] No checker registered for op: BatchedNMSDynamic_TRT. Attempting to check as plugin.
[11/07/2025-13:22:05] [I] [TRT] No importer registered for op: BatchedNMSDynamic_TRT. Attempting to import as plugin.
[11/07/2025-13:22:05] [I] [TRT] Searching for plugin: BatchedNMSDynamic_TRT, plugin_version: 1, plugin_namespace:
[11/07/2025-13:22:05] [W] [TRT] onnxOpImporters.cpp:6521: Attribute caffeSemantics not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[11/07/2025-13:22:05] [W] [TRT] BatchedNMSPlugin is deprecated since TensorRT 9.0. Use INetworkDefinition::addNMS() to add an INMSLayer OR use EfficientNMS plugin.
[11/07/2025-13:22:05] [I] [TRT] Successfully created plugin: BatchedNMSDynamic_TRT
[11/07/2025-13:22:05] [I] Finished parsing network model. Parse time: 0.015883
[11/07/2025-13:22:05] [I] Set shape of input tensor Input for optimization profile 0 to: MIN=1x3x832x4096 OPT=3x3x832x4096 MAX=3x3x832x4096
[11/07/2025-13:22:05] [I] FP32 and INT8 precisions have been specified - more performance might be enabled by additionally specifying --fp16 or --best
[11/07/2025-13:22:05] [I] Set calibration profile for input tensor Input to 3x3x832x4096
[11/07/2025-13:22:05] [I] [TRT] Calibration table does not match calibrator algorithm type.
[11/07/2025-13:22:05] [I] [TRT] Perform graph optimization on calibration graph.
[11/07/2025-13:22:05] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[11/07/2025-13:22:06] [I] [TRT] Compiler backend is used during engine build.
[11/07/2025-13:22:07] [I] [TRT] Detected 1 inputs and 4 output network tensors.
[11/07/2025-13:22:07] [I] [TRT] Total Host Persistent Memory: 430976 bytes
[11/07/2025-13:22:07] [I] [TRT] Total Device Persistent Memory: 0 bytes
[11/07/2025-13:22:07] [I] [TRT] Max Scratch Memory: 55355904 bytes
[11/07/2025-13:22:07] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 284 steps to complete.
[11/07/2025-13:22:07] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 29.9706ms to assign 19 blocks to 284 nodes requiring 1458625536 bytes.
[11/07/2025-13:22:07] [I] [TRT] Total Activation Memory: 1458625536 bytes
[11/07/2025-13:22:07] [I] [TRT] Total Weights Memory: 6216192 bytes
[11/07/2025-13:22:07] [I] [TRT] Compiler backend is used during engine execution.
[11/07/2025-13:22:07] [I] [TRT] Engine generation completed in 1.58926 seconds.
[11/07/2025-13:22:07] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +1391, now: CPU 0, GPU 1397 (MiB)
[11/07/2025-13:22:07] [I] [TRT] Starting Calibration.
[11/07/2025-13:22:07] [E] Error[3]: IExecutionContext::executeV2: Error Code 3: API Usage Error (Parameter check failed, condition: nullPtrAllowed. Tensor "Input" is bound to nullptr, which is allowed only for an empty input tensor, shape tensor, or an output tensor associated with an IOuputAllocator.)
[11/07/2025-13:22:07] [E] Error[2]: [calibrator.cpp::calibrateEngine::1236] Error Code 2: Internal Error (Assertion context->executeV2(bindings.data()) failed. )
[11/07/2025-13:22:07] [E] Engine could not be created from network
[11/07/2025-13:22:07] [E] Building engine failed
[11/07/2025-13:22:07] [E] Failed to create engine from model or file.
[11/07/2025-13:22:07] [E] Engine set up failed
trtexec command:
/usr/src/tensorrt/bin/trtexec \
--onnx=/home/me/Downloads/8.0-engine-build-test/models/detector/yolov4.onnx \
--saveEngine=/home/me/Downloads/8.0-engine-build-test/models/detector/yolov4.engine \
--minShapes=Input:1x3x832x4096 \
--optShapes=Input:3x3x832x4096 \
--maxShapes=Input:3x3x832x4096 \
--int8 \
--calib=/home/me/Downloads/8.0-engine-build-test/models/detector/yolov4.bin \
--memPoolSize=workspace:1G
For float 16 precision, I can build the engine from the .onnx file, but with these warnings:
[11/07/2025-13:44:43] [I] [TRT] No checker registered for op: BatchedNMSDynamic_TRT. Attempting to check as plugin.
[11/07/2025-13:44:43] [I] [TRT] No importer registered for op: BatchedNMSDynamic_TRT. Attempting to import as plugin.
[11/07/2025-13:44:43] [I] [TRT] Searching for plugin: BatchedNMSDynamic_TRT, plugin_version: 1, plugin_namespace:
[11/07/2025-13:44:43] [W] [TRT] onnxOpImporters.cpp:6521: Attribute caffeSemantics not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
[11/07/2025-13:44:43] [W] [TRT] BatchedNMSPlugin is deprecated since TensorRT 9.0. Use INetworkDefinition::addNMS() to add an INMSLayer OR use EfficientNMS plugin.
[11/07/2025-13:44:43] [I] [TRT] Successfully created plugin: BatchedNMSDynamic_TRT