# ONNX model integration with deepstream parallel inference app

**URL:** <https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895>\
**Category:** DeepStream SDK\
**Tags:** onnx, python, deepstream\
**Created:** [June 16, 2023, 9:51am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895 "2023-06-16T09:51:15Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![pradyyadav](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradyyadav/32/259133_2.png) [@pradyyadav](https://forums.developer.nvidia.com/u/pradyyadav)\
**Post date:** [June 16, 2023, 9:51am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/1 "2023-06-16T09:51:15Z")

</div>

Please provide complete information as applicable to your setup.

**• Hardware Platform: GPU**  
**• DeepStream Version: 6.2**  
**• TensorRT Version: 8.5.2.2**  
**• NVIDIA GPU Driver Version: 525.85.12**  
**• Issue Type: Question**

Hi,  
I want to develop a video analytics solution using deepstream parallel inference app as shown in the [deepstream parallel inference app repo](https://github.com/NVIDIA-AI-IOT/deepstream_parallel_inference_app).

I want to develop a single branch for which I have a trained ONNX model file, but I am unsure how to integrate it with the pipeline.

I have done it already for TAO trained models, where I was easily able to find config files for etlt files, which I just needed to modify according to my desire, but I cannot do the same for ONNX models. Hence I would like to know how one can use his custom ONNX model. I tried using BYOM converter, but some of the objects in my model are not supported by BYOM.

Regards,  
Pradyumna Yadav

---

<div class="post-metadata">

**Author:** ![yingliu](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yingliu/32/136703_2.png) [@yingliu](https://forums.developer.nvidia.com/u/yingliu)\
**Post date:** [June 16, 2023, 2:48pm UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/2 "2023-06-16T14:48:07Z")

</div>

> [@pradyyadav](#):
>
> but I cannot do the same for ONNX models.

Can you elaborate what config do you want to apply?  
Deepstream supports onnx model (by option onnx-file in the config file), can it work for your case?

---

<div class="post-metadata">

**Author:** ![pradyyadav](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradyyadav/32/259133_2.png) [@pradyyadav](https://forums.developer.nvidia.com/u/pradyyadav)\
**Post date:** [June 17, 2023, 12:37pm UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/4 "2023-06-17T12:37:49Z")

</div>

Can you share me an example of such configuration file where ONNX model is integrated.

---

<div class="post-metadata">

**Author:** ![yingliu](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yingliu/32/136703_2.png) [@yingliu](https://forums.developer.nvidia.com/u/yingliu)\
**Post date:** [June 19, 2023, 2:43am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/5 "2023-06-19T02:43:12Z")

</div>

Here is one example: deepstream-6.2/sources/apps/sample\_apps/deepstream-audio/configs

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [June 19, 2023, 2:47am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/6 "2023-06-19T02:47:41Z")

</div>

DeepStream already support ONNX model. [Gst-nvinfer — DeepStream 6.2 Release documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html)

What are the input layers and output layers of your ONNX model?

---

<div class="post-metadata">

**Author:** ![pradyyadav](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradyyadav/32/259133_2.png) [@pradyyadav](https://forums.developer.nvidia.com/u/pradyyadav)\
**Post date:** [June 19, 2023, 10:10am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/7 "2023-06-19T10:10:47Z")

</div>

These are the input and output of the ONNX model.

Input

```auto
name: input_1:0
type: float32[num,256,256,3]

```

Output:

```auto
name: predictions/Softmax:0
type: float32[num,2]

```

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [June 19, 2023, 10:20am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/8 "2023-06-19T10:20:37Z")

</div>

The model looks like a classifier.

Please refer to the classifier sample here: [NVIDIA-AI-IOT/deepstream\_tao\_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps)

---

<div class="post-metadata">

**Author:** ![pradyyadav](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradyyadav/32/259133_2.png) [@pradyyadav](https://forums.developer.nvidia.com/u/pradyyadav)\
**Post date:** [June 19, 2023, 10:51am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/9 "2023-06-19T10:51:32Z")

</div>

Thank you for sharing the resource, but I mentioned it already, this model isn’t TAO trained. Thus I don’t have the model configs neither the model engine that are usually generated from TAO.

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [June 19, 2023, 10:56am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/10 "2023-06-19T10:56:40Z")

</div>

Since it is the same question as [Deepstream branch giving same output from a TAO trained model - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/deepstream-branch-giving-same-output-from-a-tao-trained-model/256872/12)

Let’s discuss in [Deepstream branch giving same output from a TAO trained model - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/deepstream-branch-giving-same-output-from-a-tao-trained-model/256872/12). Close this topic.

---

<div class="post-metadata">

**Author:** ![pradyyadav](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradyyadav/32/259133_2.png) [@pradyyadav](https://forums.developer.nvidia.com/u/pradyyadav)\
**Post date:** [June 19, 2023, 11:05am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/11 "2023-06-19T11:05:12Z")

</div>

> [@Fiona.Chen](#):
>
> Since it is the same question as [Deepstream branch giving same output from a TAO trained model - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/deepstream-branch-giving-same-output-from-a-tao-trained-model/256872/12)

I would request you to understand the difference in the two forums, this forum is for ONNX model integration (not TAO trained) and [Deepstream branch giving same output from a TAO trained model - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/deepstream-branch-giving-same-output-from-a-tao-trained-model/256872/12) is a forum for a TAO trained model where I am unable to get desired outputs on real data inference.

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [June 19, 2023, 12:23pm UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/12 "2023-06-19T12:23:36Z")

</div>

you need to write the class parsing function by yourself since your output layer is not CHW dimension.

And the label file should be in such format:  
`ink_thrown;no__ink`

---

<div class="post-metadata">

**Author:** ![pradyyadav](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/pradyyadav/32/259133_2.png) [@pradyyadav](https://forums.developer.nvidia.com/u/pradyyadav)\
**Post date:** [June 23, 2023, 6:19am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/13 "2023-06-23T06:19:04Z")

</div>

> [@Fiona.Chen](#):
>
> you need to write the class parsing function by yourself since your output layer is not CHW dimension.

I think I can get some help using this: [Using a Custom Model with DeepStream — DeepStream 6.3 Release documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_using_custom_model.html#custom-output-parsing). So figuring out what needs to be changed here. You can suggest some for me to speed up on this.

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [June 23, 2023, 9:44am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/14 "2023-06-23T09:44:19Z")

</div>

**There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks**

There is a sample of customizing classifier output parsing with gst-nvinfer. Please refer to [NVIDIA-AI-IOT/deepstream\_lpr\_app: Sample app code for LPR deployment on DeepStream (github.com)](https://github.com/NVIDIA-AI-IOT/deepstream_lpr_app)

The LPR model is a classifier which does not output CHW dimension.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [July 24, 2023, 6:15am UTC](https://forums.developer.nvidia.com/t/onnx-model-integration-with-deepstream-parallel-inference-app/256895/16 "2023-07-24T06:15:23Z")

</div>

Closed on July 24, 2023, 6:15am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
