Dynamic Management of Video Sources and nvinfer Plugins for Multi-Model Inference

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson )
• DeepStream Version 6.0.1
• JetPack Version (valid for Jetson only) 4.6
• TensorRT Version 8.0.1.6
• NVIDIA GPU Driver Version (valid for GPU only) 10.2.300
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Due to business requirements, it is necessary to dynamically select models based on different sources and perform inference with multiple models on the same source. Therefore, dynamic management of input video streams and the ability to dynamically add or remove nvinfer plugins is required. Is this supported, and are there any related Python examples available for reference?

“dynamically select models based on different sources” can be implement by dynamically construct multiple DeepStream pipelines in one app.
“perform inference with multiple models on the same source” can be implement by multiple inferencing models in one DeepStream pipeline.

DeepStream SDK has provided APIs to implement the function you mentioned. But the application should be implemented by yourself.

The ability to dynamically add or remove a video source in a running pipeline, as well as launching a new instance of the nvinfer model inference plugin for each source, while maintaining parallel processing of models, and closing the inference plugin when not needed, is supported in the provided code.

The image illustrates my pipeline. However, how can I dynamically load new models into the pipeline as described above? Meanwhile, how should the relevant plugins handle this to ensure stable operation? Is there any documentation available for features involving components such as streammux, nvinfer, tee, etc., which need to provide dynamism? Are these features supported in DS 6.4?

I am currently considering creating a new pipeline each time a new model is loaded. For example, if I load four models, namely face model, person model, safehat model, and fire model, I will create four pipelines accordingly. When a video stream needs to be added, I can refer to the implementation in deepstream_python_apps/apps/runtime_source_add_delete/ . I have tested this approach myself, and it works correctly. Do you have any suggestions?

It is not necessary, if the sources and models are all changed, it is really a new pipeline. You need to destroy the original pipeline and restart a new one. It is no meaning to keep the old pipeline running.

What do you mean by load new models? Will the original two models be changed both?

It depends. Can you tell us in which situation you need to change the model? Why did you want the pipeline continue to run while the inferencing is not needed.

Do you need the four models to inference on the same sources? If so, one pipeline is enough.

sources → nvstreammux → nvinfer1(face) → nvinfer2(person) → nvinfer3(safehat) → nvinfer4(fire) → …

If the four models should infer on different models, you need to construct pipeline according to the sources.
E.G.
If you want model1 and model2 inference on source1, source2, source 3 while model3 and model4 inference on source 4 and source5, two pipeline is enough.

pipeline1: source1 + source2 + source3 → nvstreammux → nvinfer1(model1)->nvinfer2(model2)->…
pipeline2: source4 + source5 → nvstreammux → nvinfer3(model3) → nvinfer4(model4)->…

If the four models infers on different sources but there are crossing sources, E.G. model1 and model2 inference on source1, source2 and source3; model3 and model4 inference on source2, source 3 and source 4. Then you need the solution in NVIDIA-AI-IOT/deepstream_parallel_inference_app: A project demonstrating how to use nvmetamux to run multiple models in parallel. (github.com)

The dynamic source add/remove needs you to guarantee there is always at least one source in use in the pipeline branches(if you use tee(s) in your pipeline) when you add/remove sources, or else it will fail.

We have a video management platform that can integrate multiple camera devices on one hand and manage multiple Jetson devices on the other hand. Clients can configure multiple cameras to run specific algorithms on designated Jetson devices through the platform’s web interface. The relationship between cameras and models is many-to-many. Initially, the devices do not run any models, and models are only loaded for detection after user configuration.

  1. When a user removes all cameras associated with a particular algorithm, it is necessary to unload the corresponding model and free up the related device resources, including GPU resources.
  2. Users may have numerous cameras, and configurations are assigned to different models in batches. In this case, the corresponding models need to be loaded into the pipeline as needed, rather than loading all models at the beginning.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

This is what the pipeline stop will do.

You can add all models from the beginning, and choose to display some of the output according to the user’s configurations.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.