Custom action recognition model not working with action recognition app

• Hardware Platform (Jetson / GPU): Ubuntu 22.02 X86 workstation w/ Ada Lovelace RTX2000
• DeepStream Version: 7.1
• TensorRT Version: 10.3.0.26-1+cuda12.5
• NVIDIA GPU Driver Version (valid for GPU only): Driver Version: 580.65.06, cuda version 12.6
• Issue Type: bug

I am having an issue swapping out the model in deepstream-3d-action-recognition with a custom one we have trained (not using the TAO/resnet). It was built with pytorch and it has the exact same dimensions (except more action classes on output) as the previous model. Here is the code we used to export the model below:

input_tensor = torch.randn(1,3,32,256,256).to(device)

torch.onnx.export(
model,
input_tensor,
‘recognition_net.onnx’,
input_names=[‘input_tensor’],
output_names=[‘cls_scores’],
export_params=True,
do_constant_folding=True,
verbose=False,
opset_version=11,
dynamic_axes={
‘input_tensor’: {
0: ‘batch_size’,
3: ‘height’,
4: ‘width’
}
}
)

And this code snippet successfully runs inference outside of deepstream for us with the model:

session = onnxruntime.InferenceSession(‘recognition_net.onnx’)
input_feed = {‘input_tensor’: input_tensor.cpu().data.numpy()}
outputs = session.run([‘cls_scores’], input_feed=input_feed)

But when we plug it into the action recognition app, with most of the settings the same, it fails with a TensorRT error.

I have added the preprocessing and the inference configuration files too.

config_preprocess_3d_custom.txt (2.6 KB)

config_infer_primary_recognition_net.txt (2.5 KB)

Would anyone know how to help me with this? The only difference I can see between the resnet onnx and our is that it has a width and height of 224 and ours is 255, which I have fixed in the config, but nevertheless the dimensional is off and it fails to build an engine. I assume that this is an issue with our onnx model, or is there something else I am missing?

Thanks and Best Regards,

Portia

The components version need to meet the requirements. Please refer to the compatibility table. DeepStream docker is recommended.

Thank you for your suggestions. We tried with 7.1 triarch docker image, but it still failed with the error below, (same error):

ERROR: [TRT]: IOptimizationProfile::setDimensions: Error Code 3: API Usage Error (Parameter check failed, condition: std::all_of(dims.d, dims.d + dims.nbDims, (int32_t x) noexcept { return x >= 0; }). )
ERROR: [TRT]: IOptimizationProfile::setDimensions: Error Code 3: API Usage Error (Parameter check failed, condition: std::all_of(dims.d, dims.d + dims.nbDims, (int32_t x) noexcept { return x >= 0; }). )
ERROR: [TRT]: IOptimizationProfile::setDimensions: Error Code 3: API Usage Error (Parameter check failed, condition: std::all_of(dims.d, dims.d + dims.nbDims, (int32_t x) noexcept { return x >= 0; }). )
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1097 Explicit config dims is invalid
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:817 Failed to configure builder options
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:530 failed to build trt engine.
0:00:01.739689312 214 0x59d1079c19b0 ERROR nvinfer gstnvinfer.cpp:678:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2126> [UID = 1]: build engine file failed
0:00:01.866769426 214 0x59d1079c19b0 ERROR nvinfer gstnvinfer.cpp:678:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2213> [UID = 1]: build backend context failed
0:00:01.867202660 214 0x59d1079c19b0 ERROR nvinfer gstnvinfer.cpp:678:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1351> [UID = 1]: generate backend failed, check config file settings
0:00:01.868596135 214 0x59d1079c19b0 WARN nvinfer gstnvinfer.cpp:914:gst_nvinfer_start: error: Failed to create NvDsInferContext instance
0:00:01.868609281 214 0x59d1079c19b0 WARN nvinfer gstnvinfer.cpp:914:gst_nvinfer_start: error: Config file path: config_infer_primary_recognition_net.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED

Our versions are as listed below, everything should be fine but the issue persists:

root@6ca6b03940f4:/fsi_service# deepstream-app --version-all
deepstream-app version 7.1.0
DeepStreamSDK 7.1.0
CUDA Driver Version: 12.9
CUDA Runtime Version: 12.6
TensorRT Version: 10.3
cuDNN Version: 9.0
libNVWarp360 Version: 2.0.1d3

Any other ideas?

We tried to convert the onnx model to engine using TensorRT and received the following errors:

r_context_impl.cpp:2080> [UID = 1]: deserialize engine from file :/opt/nvidia/FSI_Engine/recognition_net_mom.onnx failed
0:00:00.318587757 301820 0x5a09b56bb070 WARN nvinfer gstnvinfer.cpp:681:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2185> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/FSI_Engine/recognition_net_mom.onnx failed, try rebuild
0:00:00.318601935 301820 0x5a09b56bb070 INFO nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2106> [UID = 1]: Trying to create engine from model files
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:568 failed to build network since there is no model file matched.
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:518 failed to build network.
0:00:01.076815574 301820 0x5a09b56bb070 ERROR nvinfer gstnvinfer.cpp:678:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2126> [UID = 1]: build engine file failed
0:00:01.089133269 301820 0x5a09b56bb070 ERROR nvinfer gstnvinfer.cpp:678:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2213> [UID = 1]: build backend context failed
0:00:01.089377361 301820 0x5a09b56bb070 ERROR nvinfer gstnvinfer.cpp:678:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1351> [UID = 1]: generate backend failed, check config file settings
0:00:01.089424001 301820 0x5a09b56bb070 WARN nvinfer gstnvinfer.cpp:914:gst_nvinfer_start: error: Failed to create NvDsInferContext instance
0:00:01.089432858 301820 0x5a09b56bb070 WARN nvinfer gstnvinfer.cpp:914:gst_nvinfer_start: error: Config file path: config_infer_primary_recognition_net.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED

Please refer to this topic. you may set force-implicit-batch-dim=1 in nvinfer configuration file.