Custom ONNX model, TensorRT Engine with PyCUDA in Deepstream

• Hardware Platform (Jetson / GPU) dGPU (Tesla T4)
• DeepStream Version 6.1.1
• TensorRT Version 8.4.1.5
• NVIDIA GPU Driver Version (valid for GPU only) 515.65.01
• Issue Type( questions, new requirements, bugs) Question

Using the ONNX model from WoodScape/omnidet at master · valeoai/WoodScape · GitHub

Note: I have TensorRT engine of the above ONNX model which works in TensorRT framework with PyCUDA. The same engine file (with NHWC layout) can’t run on Deepstream pipeline due to NCHW memory layout requirement of Deepstream. I tried to convert the engine file to NCHW with no luck.

It would be easier if I can reuse my inference code using PyCUDA inside Deepstream Pipeline.
Do you have any reference document for using PYCUDA with Deepstream?
I saw this but it is not of much help

There is “network-input-order” parameter with nvinfer configuration. network-input-order, you don’t need to convert your model from NCHW to NHWC. You also need to input “infer-dims” parameter to give the correct input dimensions of your model with gst-nvinfer. Please read the document carefully.

gst-nvinfer is TensorRT based which is CUDA accelerated already.

Hi Fiona,
Thanks for your response.
I’ve seen the plugin “nvinfer” but using that plugin for our custom model will require implementing custom inference method. Since we have PyCUDA based inference script for our custom model, I was wondering if I could reuse that script for a quick check in Deepstream.

Can you tell us what kind of custom inference method do you need?

Below is the method I’m using for inference and I’m looking for equivalent functionality in Deepstream

When I had used NVInfer with tensorRT engine file, I had seen the issue below

Similar to the issue seen here “RGB/BGR input format specified but network input channels is not 3”

Seems you need consecutive several frames as the model input, right?

Please refer to deepstream-3d-action-recognition sample in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-action-recognition, this is also a model with consecutive frames inputs.

Ok, I will check it out.
But my initial question was that, is replacing nvinfer with PyCuda inference possible in Deepstream? Let me know.

DeepStream is NvBufSurface batch based. If the PyCuda inference plugin can handle NvMetaBatch and NvBufSurface correctly, it can work with other DeepStream plugins. NVIDIA DeepStream SDK API Reference: NvBufSurface Types and Functions

ok, I’ll read the docs and try to understand the details you mentioned.

Also, second question is,
Even after setting the “network-input-order” and “infer-dims”, I’m seeing the error “RGB/BGR input format specified but network input channels is not 3”.
This means, Deepstream Doesn’t automatically support NHWC layout. I need to to add custom function to convert input frame from NCHW to NHWC right?
And if you could provide an example to do so, it would be great.

Your model is multiple input layers model. The default gst-nvinfer just accept single input layer model. So please change your model to single input layer model and refer to the sample of deepstream-3d-action-recognition sample in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-action-recognition, this is also a model with consecutive frames inputs.

1 Like

Thanks Fiona, I’ll check it out and try the way you have mentioned.