Multiple model pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

For customized yolo models,

  1. Is it possible to run these separate models simultaneously on one stream
  2. How to build multiple model pipeline

Is there a reference or example, for the 2 scenarios.
The custom deepstream yolo models were created according to https://docs.nvidia.com/metropolis/deepstream/4.0/Custom_YOLO_Model_in_the_DeepStream_YOLO_App.pdf

Please help

Thank you

3

for q#1, yes, the models in different nvinfer/nvinferserver plugins can run simultaneously.
for q#2, you can refer to deepstream-test2 sample.

BTW, ds doc - NVIDIA DeepStream SDK Developer Guide — DeepStream 6.1.1 Release documentation

4

Thank you for the quick response.

  1. To implement a multiple yolo model pipeline on same stream, would it require separate deepstream_app_config and config_infer_primary_yolo files for each model?

  2. Would the nvdsinfer_custom_impl_yolo folder need to be edited for all the custom yolo deepstream models its present in? Could you please tell which other files need to be modified to implement: multiple yolo models into one deepstream app that can produce separate output files for each model on same input stream.

  3. The deepstream test2 app only supports primary detector and secondary classifier on detected objects. How can multiple yolo detectors run simultaneously on same stream and how to customize test2 app for it.

Please clarify my questions.

5

  1. To implement a multiple yolo model pipeline on same stream, would it require separate deepstream_app_config and config_infer_primary_yolo files for each model?
    yes, refer to deepstream-test2

  2. Would the nvdsinfer_custom_impl_yolo folder need to be edited for all the custom yolo deepstream models its present in?
    basically, yes.
    Could you please tell which other files need to be modified to implement: multiple yolo models into one deepstream app that can produce separate output files for each model on same input stream.
    will you run it will deepstream-app? if you will, based on the informantion you shared, I don’t see others needed to modify.

  3. The deepstream test2 app only supports primary detector and secondary classifier on detected objects. How can multiple yolo detectors run simultaneously on same stream and how to customize test2 app for it.
    simialr as deeepstream-test2, insert another infer after the 2nd nvinfer.

6

thank you for clarifying.
could you please clarify what files or parameters needed to be modified in nvdsinfer_custom_impl_yolo to run with deepstream-test2. Sorry I just started with deepstream.

I would run deepstream-test2 with independent yolo models but on same stream without cascading them. In that case it would be one pgie config file and one sgie config file, but the secondary detector would also work simultaneously on full frame with the primary detector and both detect different objects. Would that be possible to implement with deepstream-test2.

7

Please try to read and understand objectDetector_Yolo sample, deepstream-test2 basd on DS doc - NVIDIA DeepStream SDK Developer Guide — DeepStream 6.1.1 Release documentation .
They are easy enough to read and understand as start for development, that’s why we provided the sample and doc.

8

sure, thank you for the help.

9

could I please know if saving output as file in deepstream-app2 is possible? thank you

10

what output? please clarify.

11

currently, I have read that deepstream-test2 app displays output in a window, is it possible to save it in an .mp4 file? It is possible to save the output as file in objectDetector_Yolo, by changing sink type, so could I please know if it is possible to save in deepstream test2 app as well?

12

yes, you can .

This is the reference command, that is, you need to change the current sink “nvegltransform ! nveglglessink” to “nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=script.mp4”

gst-launch-1.0 -e rtspsrc location=rtsp://10.19.225.102/media/video1 latency=100 ! queue ! rtph264depay ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=config_infer_primary_facedetectir.txt ! nvmultistreamtiler rows=1 columns=1 width=1280 height=720 ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=script.mp4

13

Sure, Thank you very much. This helps a lot.

14

Could you please clarify

  1. Is it possible to save the output to .mp4 file (as above for deepstream test2 app) in back to back detectors?
  2. In deepstream test2 app, how to customize and use two yolo detectors both on full frame (both as primary detectors) without any classifiers. and produce two output video files simultaneously.
    Would deepstream-test2 be a better solution or back to back detectors?
    Please do clarify. Thank you for your time.
15

yes, the smilar change as above to the sink part.

if you want both two gies to do detection on full frames, you can refer to back to back sample.

16

thank you. That would generate 2 output files for 2 detectors simultaneously on full frame?

Thank you very much