Inquiry on how to create a custom DeepStream Plugin

[kbs23]

• Hardware Platform (Jetson / GPU)

GPU - NVIDIA RTX 4090 (x2)

---

• DeepStream Version

DeepStream 7.1
Base Docker: nvcr.io/nvidia/deepstream:7.1-gc-triton-devel

---

• JetPack Version (valid for Jetson only)

N/A

---

• TensorRT Version

TensorRT 10.13.0

---

• NVIDIA GPU Driver Version (valid for GPU only)

Driver Version: 550.163.01
CUDA Version: 12.6

---

• Issue Type (questions, new requirements, bugs)

Question / New Requirement
→ I want to create a custom DeepStream plugin to support a multi-input model (Grounding DINO).

---

• How to reproduce the issue?

N/A (This is a new requirement, not a bug)

---

• Requirement details

I am trying to integrate a Grounding DINO model (open-vocabulary object detector) into DeepStream 7.1.

This model requires multi-inputs:

• An image tensor
• A text prompt tensor (e.g., token IDs or embeddings)

I have already successfully:

• Converted the model into a .engine file using TensorRT

Now, I would like to run this model inside DeepStream using nvinfer.

My specific request:

1. How can I create a custom DeepStream plugin to handle such multi-input models?
2. Is there any official guide, documentation, or example plugin code to:

• Write a custom preprocessing plugin (libnvdsinfer_custompreproc_*.so)
• Write a custom parser plugin (nvdsinfer_customparser_*.so)

3. What interfaces or structures must I implement to make these plugins compatible with nvinfer?
4. Are there any sample plugins or models (e.g., DETR, DINO) that support multi-input or transformer-based inference that I can refer to?

Any documentation, source code examples, or starting templates would be greatly appreciated.

Thank you!

[yuweiw]

Please refer to this topic 338052. DeepStream does not support GenAI models.

[kbs23]

"Thank you for the reference to topic 338052. I’d like to respectfully clarify whether Grounding DINO should be categorized as a GenAI model in this context.

Technical clarification needed:

Grounding DINO, while incorporating transformer architecture and text processing, functions primarily as an object detection model - similar to traditional YOLO or RCNN models but with text-guided detection capabilities.

Key points for consideration:

• Primary output: Bounding boxes and detection scores (same as traditional detectors)
• Core use case: Object detection in video streams (typical DeepStream application)
• Model behavior: Deterministic detection, not generative content creation

My specific questions:

1. Does the “GenAI” limitation apply to any model using transformers/attention mechanisms?
2. Or does it specifically target generative models (text/image generation, chatbots, etc.)?
3. Would other vision-transformer models (like DETR, Swin Transformer) also be considered “GenAI”?

Context: I’m trying to use this for standard video analytics (accident detection, fire detection) - which seems aligned with DeepStream’s core purpose, just with a more advanced detection model.

Would appreciate clarification on the technical boundaries of this limitation."

[yuweiw]

Could you attach the link of your Grounding DINO model?
For the multi-inputs, this might not be an insurmountable problem. You can use our nvdspreprocess plugin to customize that or modify our nvinfer source code directly.

But for the text prompt tensor, does this input require operations such as tokenization and embedding? Currently, there is no ready-made module in Gstreamer for performing these operations. This might require a significant amount of customized code. This might be the technical boundaries of this limitation.

[kbs23]

Hello,

I am working on integrating a fine-tuned Grounding DINO model with DeepStream. The fine-tuning was done using this notebook:

notebooks/tao_launcher_starter_kit/grounding_dino/finetune_grounding_dino.ipynb

In the tao_tutorials, the following NGC model is used:

Grounding DINO | NVIDIA NGC

I came across this forum reply regarding DeepStream integration:

Grounding DINO Deployment with DeepStream Missing from TAO Integration Documentation

You may also refer to DeepStream 7.1 + Grounding DINO Integration Issue: Multi-Input Model Support - #4 by Fiona.Chen. You may consider standalone Inference if you run Grounding DINO models directly (outside DeepStream) . You can refer to the TAO source code from tao-pytorch branch tao_pytorch_backend/nvidia_tao_pytorch/cv/grounding_dino/scripts/inference.py at main · NVIDIA/tao_pytorch_backend · GitHub or tao-deploy branch tao_deploy/nvidia_tao_deploy/cv/grounding_dino/scripts/inference.py at …

From what I have gathered, there are no available reference materials on using the nvdspreprocess plugin for Grounding DINO integration. I also understand that since DeepStream does not natively provide modules for tokenization and embedding required by the text prompt input, a significant amount of customized development is necessary.

Any further guidance or resources regarding this integration would be greatly appreciated.

Thank you.

[yuweiw]

You can refer to our https://docs.nvidia.com/jetson/jps/inference-services/gdino.html. This might require further improvement, but the source code is all within the docker container. You can refer to it and do the necessary improvements.