I need to run inference using a video that has a very large frame size, approximately 1900X480. I would like to crop it into three separate frames of approximately 640X480 before running inference on this, since the model I am using will perform better on smaller frame sizes. Will I need to write a plugin to do this? Any guidance is much appreciated.
I am using Deepstream 5.0 on a Xavier AGX with JetPack 4.4.1 on it.
Thank you,
Ramya.
DS supports to crop or scale the frame, i.e. 1900X480 here, to smaller frame resolution required by the model for inference with the full frame.
For exmaple, if the input resolution is 640x320, DS can scale 1900x480 to 640x320 and feed this scaled frame into the model for inference.
With this support, do you still need to break/crop the frame into multiple frames?
Thank you for your reply mchi, I really appreciate it.
How do I crop the images? I have not been able to do it yet. Do I need to change the code? Or is it a config change?
Ideally, I would like to crop a frame into three smaller sub-frames and use them separately for inference. But, if I can just crop, I can use the same input thrice and for each one I will crop differently. I do not want it to scale a large frame to a smaller frame.
you can create the pipeline like below
Thank you for getting back to me quickly. Can I use one of the sample apps to create this pipeline? Can I specify the crop parameters in the config file or will I need to change the code to do this?
I am sorry if my questions are silly, I am new to Deepstream.
Can I use one of the sample apps to create this pipeline?
Depends on your other parts of your pipeline, you can find the introduction of samples in C/C++ Sample Apps Source Details — DeepStream DeepStream Version: 5.0 documentation .
For this part, you need to add the code by yourself, no DS sample has similar code as this.
For the crop configure, can’t add the config parameters in config file, that is, you need to add the config in code.
about crop of nvvideoconvert, you could refer to Nvvideoconvert automatically overrides src-crop property
Thank you, I will take a look at it.
Hi mchi,
I am working on developing the pipeline with a tee and three nvvideoconvert instances. Will this ensure that the three crops will be generated in parallel? Also, will all the three crops contain the same source_id when they merge at the streammux?
Thank you so much!
Will this ensure that the three crops will be generated in parallel?
On Jetson platform, the crop is done by hardware accelerator - VIC in soc, so essentially, they are precessed in sequence, but hardware accelerator - VIC can process it very very quickly ( <1ms), you can alsmot think they are processed in parallel.
will all the three crops contain the same source_id when they merge at the streammux?
I don’t think so, they should have different source_id because the source_id is assigned in nvstreammux
Thank you so much.
I was able to crop a frame into three subframes and use the inference engine to perform inference. However, the throughput has come down considerably. Is it because there are more number of frames on which the inference needs to be run? Is there any way I can speed this up?
Thank you,
Ramya.
I think so, because the inference frames increase 3x.
can you increase the ‘interval’ of dsnvinfer?
Thank you for the suggestion, I will try it.
I need to stream metadata using Kafka. Will I be able to connect other Gstreamer elements (gst-nvmsgbroker and gst-msgconv) to do this?
yes, you can refer to DS sample - sample_apps/deepstream-test5
I tried increasing the interval and did not see any significant improvement. I am trying to see if I can introduce any threading to improve performance. Is there a way I can introduce the tee and the nvvideoconverts in deepstream-test5? My hope is that I will not need to recreate the multithreading that may already exist there.
Thank you,
Ramya.