Why are my output engine sizes all zero

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hardware Platform: x86 with L4
DeepStream Version : 7.1
TensorRT Version : 10.8
How to reproduce the issue ? : build a dynamic engine from an ONNX model with some input profile - e.g. min 1x3x640x640 opt 5x3x640x640 max 8x3x640x640 , then run the model in a deepstream pipeline

I have a dynamic batch size engine, which seems to work fine with trtexec, but in the deepstream pipeline it produces no outputs.

it reports the input/output sizes as follows:

image
image1689×167 20.5 KB

my expectation would be that the output shapes do follow the input shapes?

3

The number of dimensions of your model seems to be 0. You can check the output layer shape of your onnx through https://netron.app/

//  in the /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer/nvdsinfer_func_utils.cpp
SplitFullDims(const nvinfer1::Dims& fullDims, NvDsInferDims& dims, int& batch)
{
    if (!fullDims.nbDims)
    {
        dims.numDims = 0;
        dims.numElements = 0;
        batch = 0;
    }
    else
    {
4

image
image964×1354 52.7 KB

Netron says the output dimensions are not zero.
Here is the trtexec output:

image
image1835×422 64.7 KB

So the input shape is set to dynamic as intended? I did specify min,opt,max as you can see in the previous post and there is no option to force the outputs to be dynamic?

I was assuming they are set to the input dimensions at runtime?

5

anything?
I am happy to share the model (onnx and the converted engine) - this is a fairly standard export from pth to ONNX with dynamic axes, so I am lost at why this doesnt translate to deepstream?

6

Refer to the TrtEngine::getFullDimsLayersInfo function in /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer/nvdsinfer_model_builder.cpp.

For the engine file, only the input layer supports min/opt/max. For the output layer, it is normal for min/opt/max to be 0.

I think there is something wrong with the onnx model when it was exported，has the network structure changed after export?

If DeepStream is to work properly, you must first ensure that the onnx is correct.

There may be many reasons why pt does not work after exporting onnx, such as dynamic shape or lack of related operators.

7

I think the outputs did not get generated for other reasons - the onnx model is fine , I checked with Netron and trtexec - it is just confusing that the reporting for dynamic models in deepstream comes back with this conflicting and non-sensical information.

This masked the other issue, so would be great if the output dimensions could be reported properly at runtime.

8

unfortunately, the issue still remains.

I am using this code (taken from here and modified slightly) to convert the dynamic ONNX model:

The original script does not work with dynamic models, so I added :

 profile = self.builder.create_optimization_profile()
 for i in range(self.network.num_inputs):
             input = self.network.get_input(i)
             assert input.shape[0] == -1
             min_shape = [min_batch_size] + list(input.shape[1:])
             opt_shape = [opt_batch_size] + list(input.shape[1:])
             max_shape = [max_batch_size] + list(input.shape[1:])
             profile.set_shape(input.name, min_shape, opt_shape, max_shape)
             
 self.config.add_optimization_profile(profile)

inserted at line 256 (TensorRT/samples/python/tensorflow_object_detection_api/build_engine.py at 64e56ab6cbe95c8220d80bc7ff10ccf61935759a · NVIDIA/TensorRT · GitHub)

This will build a , presumably, dymamic engine.

When loading this engine into deepstream, it displays the dimensions correcly (as shown earlier), but the custom box conversion function does seem to get called on the same frame now all the time.

Just converting the ONNX model with trtexec, without specifying a profile, will yield an engine that seems to work - but the batched input is gone.

Any ideas how I can use the python engine building script - which I also want to use for calibration and quantization, to produce a working engine?

9

1.DS-7.1 only tested on TensorRT-10.3, so please test it on corresponding branch

2.The documentation contains outdated guidelines, TensorRT-10.3 corresponding docker image is nvcr.io/nvidia/tensorflow:24.08-tf2-py3

3.I used the TRT-10.3 docker image to export onnx, but it didn’t work with DeepStream. I don’t know why. It may be a issue in the documentation. I don’t know much about TRT. You can try to get more help at