Hi @Morganh,
Here is a summary of what we have tried so far:
- Build int8 engine using TAO 3.0 exported files (yolo4.etlt and yolo4.bin) using DeepStream 8.0
nvinfer
Outcome:
Failed to build int8 engine, TAO 3.0 exported files are incompatible with DeepStream 8.0
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1261 Deserialize engine failed because file path: /home/me/detector/detector.engine open error
0:00:00.321296740 3265927 0x5b20590ec6e0 WARN nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<detector> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2097> [UID = 1]: deserialize engine from file :/home/me/detector/detector.engine failed
0:00:00.321311770 3265927 0x5b20590ec6e0 WARN nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<detector> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2202> [UID = 1]: deserialize backend context from engine from file :/home/me/detector/detector.engine failed, try rebuild
0:00:00.321319070 3265927 0x5b20590ec6e0 INFO nvinfer gstnvinfer.cpp:685:gst_nvinfer_logger:<detector> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2123> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnxOpImporters.cpp:6521: Attribute caffeSemantics not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.
WARNING: [TRT]: BatchedNMSPlugin is deprecated since TensorRT 9.0. Use INetworkDefinition::addNMS() to add an INMSLayer OR use EfficientNMS plugin.
WARNING: [TRT]: Calibration Profile is not defined. Calibrating with Profile 0
ERROR: [TRT]: Unexpected exception _Map_base::at
Segmentation fault (core dumped)
- We followed your suggestions in Does TAO 5.0 support exporting a model trained by TAO 3.0? - #3 by Morganh
suggestion 1: change the .etlt file to .onnx file
suggestion 2: change the .tlt file to .hdf5 file, and then export to .onnx file using TAO5
Outcome:
In both suggestions, what we ended up with is an .onnx file. I can build float 16 engine from the .onnx file for DeepStream 8.0, but I cannot build int 8 engine because both suggestion only provide an .onnx file, there is no new calibration cache (cal.bin) file. The calibration cache file exported by TAO 3.0 cannot be used with the .onnx file.
- We followed your suggestions in Does TAO 5.0 support exporting a model trained by TAO 3.0? - #7 by Morganh
Please use 5.5 tao-deploy docker to generate cal.bin again.
Outcome:
Failed to build int8 engine with the following errors:
[I] [TRT] Calibration table does not match calibrator algorithm type.
[E] Error[3]: IExecutionContext::executeV2: Error Code 3: API Usage Error (Parameter check failed, condition: nullPtrAllowed. Tensor "Input" is bound to nullptr, which is allowed only for an empty input tensor, shape tensor, or an output tensor associated with an IOuputAllocator.)
[E] Error[2]: [calibrator.cpp::calibrateEngine::1236] Error Code 2: Internal Error (Assertion context->executeV2(bindings.data()) failed. )
- We followed your suggestions in Does TAO 5.0 support exporting a model trained by TAO 3.0? - #9 by Morganh
For QAT workflow […]. You need not use --cal_cache_file . Please refer to below and retry. Thanks.
We shared the outcomes for two cases in Does TAO 5.0 support exporting a model trained by TAO 3.0? - #10 by hyperlight
- Trained using TAO 3.0 + Exported using TAO 5.5
- Trained using TAO 5.5 + Exported using TAO 5.5
Again, in both cases, we ended up with is an .onnx file:
- float 16 engine: build works, detection works
- int 8 engine: build works, detection failed due to missing calibration cache file.
- We followed your suggestions in Does TAO 5.0 support exporting a model trained by TAO 3.0? - #11 by Morganh and Does TAO 5.0 support exporting a model trained by TAO 3.0? - #13 by Morganh
All of above items follow tao’s pipeline. Temporally not use trtexec and deepstream. Just to narrow down to check if tao’s pipeline works.
Could you do one more experiment? Please use the trtexec which is inside the deploy docker to generate int8 engine and run inference again.
Outcome:
We can create an int8 engine and the int8 engine also successfully detected the objects and drew bounding boxes. The deploy docker gives the same outcomes as running yolo_v4 gen_trt_engine and yolo_v4 inference to generate int8 engine and run inference using TAO launcher.
- We followed your suggestions in Does TAO 5.0 support exporting a model trained by TAO 3.0? - #15 by Morganh
You can add --cal_cache_file /path/to/export_qat/cal.bin inyolo_v4 gen_trt_engine command line. Then, cal.bin will be generated.
Outcome:
Failed to build int8 engine with the following errors:
[I] [TRT] Calibration table does not match calibrator algorithm type.
[E] Error[3]: IExecutionContext::executeV2: Error Code 3: API Usage Error (Parameter check failed, condition: nullPtrAllowed. Tensor "Input" is bound to nullptr, which is allowed only for an empty input tensor, shape tensor, or an output tensor associated with an IOuputAllocator.)
[E] Error[2]: [calibrator.cpp::calibrateEngine::1236] Error Code 2: Internal Error (Assertion context->executeV2(bindings.data()) failed. )
Please let me state our goal again:
Our goal is to deploy yolov4 model trained by TAO 3.0 as int8 precision egnine in DeepStream 8.0, how do we do that?
