DeepStream SDK and decoding RTSP on GPU

Please find the below complete information as applicable to our setup.

  • Hardware Platform -GPU (RTX series NVIDIA GPU, 16GB RAM, i5-7500 CPU)

  • DeepStream Version - 5.1.0

  • TensorRT Version - 7.2.3.4

  • NVIDIA GPU Driver Version - driver version - 450.51 with cuda - 11.0, cudnn - 8.1.0

  • Issue Type - questions

  • Requirement details -

  1. Currently in our inference pipeline, we are restarting the camera whenever we get any frame none. Is this feature available in deepstream to restart the camera if we are not able to read any frame (if camera goes down).
  2. We have pre-processing and post processing part for our detection pipeline. Can we customize this part in Deepstream.
  3. How can we pass the output from nvinfer( frames and predictions) to python script to further process the results (Is there will be any latency issues in later stages).
  4. How can we convert the Deepstream supported format(RGBA/BGRx) to normal RGB/BGR without any change in image quality
  1. What kind of camera? CSI, USB or ip camera? What kind of restart is needed? What do you mean by ‘if camera goes down’? Will the camera driver report error to upper layer software? There is already reconnection sample in deepstream-app for rtsp sources. DeepStream Reference Application - deepstream-app — DeepStream 5.1 Release documentation
  2. What kind of pre-processing and post-processing? What kind of model are you using?Detector, classifier, segmentation or anything else? What is the input and output of the model?
  3. You can write Deepstream python app. Python Sample Apps Source Details — DeepStream 5.1 Release documentation. There is latency in every stage. The latency may be small or large due to what is done in the stage.
  4. If there is only format change from RGBA/BGRx to RGB/BGR, there is no image quality change with or without deepstream.
1 Like
  1. What kind of camera? CSI, USB or ip camera? - IP camera

What kind of restart is needed? - camera pipeline restart, not the hardrware restart
Will the camera driver report error to upper layer software? - No

There is already reconnection sample in deepstream-app for rtsp sources. DeepStream Reference Application - deepstream-app — DeepStream 5.1 Release documentation

  1. What kind of pre-processing and post-processing? --We are doing Image preprocessing, normalization and transformation in the pre-processing part. Post processing includes converting bounding box coordinates from YOLO format to original, NMS and filtering top predictions per class(Class ID, bounding box, confidence).
    What kind of model are you using? Detector, classifier, segmentation or anything else? - Detection, YOLOv5
    What is the input and output of the model? - Input : Tensor image, Output : Tensor output of size (1, 16125, 106) → 106 (4 bbox, confidence and 101 confidence per class).

  2. You can write Deepstream python app. Python Sample Apps Source Details — DeepStream 5.1 Release documentation. There is latency in every stage. The latency may be small or large due to what is done in the stage. - Will be working on it and let you know.

  3. If there is only format change from RGBA/BGRx to RGB/BGR, there is no image quality change with or without deepstream. - there is change in colour for original image when converted from RGBA to RGB( we are using cv2.COLOR_RGBA2RGB)

  1. Currently, gst-nvinfer has already supported all pre-processing and post-processing you need. Please refer to Gst-nvinfer — DeepStream 5.1 Release documentation. /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer.
    There is also sample of yolov3 and yolov4, [Object Detection — Transfer Learning Toolkit 3.0 documentation] , sample code: [GitHub - NVIDIA-AI-IOT/deepstream_tlt_apps: Sample apps to demonstrate how to deploy models trained with TLT on DeepStream]

4.It is just format change, the color will not be changed. With deepstream, all video and images buffers are HW buffers, and the color transformation is done by HW. There is no NV deepstream component can output RGB/BGR data.

1 Like

deepstream_python.py (14.0 KB)
I am able to run pipeline for multiple RTSP sources. But when I tried to convert the frames from Gst Buffer to numpy array using the following lines:
n_frame=pyds.get_nvds_buf_surface(hash(gst_buffer),frame_meta.batch_id)
frame_image=np.array(n_frame,copy=True,order=‘C’)
frame=cv2.cvtColor(frame_image,cv2.COLOR_RGBA2BGRA)
I got the following error:
Traceback (most recent call last):
File “deepstream_python.py”, line 81, in tiler_src_pad_buffer_probe
n_frame = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
RuntimeError: get_nvds_buf_Surface: Currently we only support RGBA color Format
I tried to put the nvvideoconvert and capsfilter before tiler to convert the frames to RGBA.
nvvidconv1 = Gst.ElementFactory.make(“nvvideoconvert”, “convertor1”)
if not nvvidconv1:
sys.stderr.write(" Unable to create nvvidconv1 \n")
print(“Creating filter1 \n “)
caps1 = Gst.Caps.from_string(“video/x-raw(memory:NVMM), format=RGBA”)
filter1 = Gst.ElementFactory.make(“capsfilter”, “filter1”)
if not filter1:
sys.stderr.write(” Unable to get the caps filter1 \n”)
filter1.set_property(“caps”, caps1)
But still got the same error.

You are using cv2.cvtColor(frame_image,cv2.COLOR_RGBA2BGRA) in tiler_src_pad_buffer_probe, and tiler_src_pad_buffer_probe is probed in nvstreammux src pad. Why do you think “to put the nvvideoconvert and capsfilter before tiler to convert the frames to RGBA.” can help to convert the data in nvstreammux src pad to RGBA?

1 Like

Thanks for your help…
I corrected the tiler_src_pad and memory type for nvstreammux, nvvideoconvert and tiler to get rid of the error.

deepstream_test.py (16.4 KB)
@Fiona.Chen We are trying to setup up a pipeline where we want numpy frames from Deepstream after Decoding and aggregation.

  1. We tried the following script for that( We are able to convert Gst-Buffers to numpy ) at later stage we want to pass these frames to our detection pipeline. So we just want a simple function which will take the gst-Buffers as input and return the numpy frames for detection pipeline. (If you have any better solution for this please suggest).
  2. Currently I want to disable the output stream in the script. How to do that?

What is your detection pipeline? A separated process different to deepstream app?
There is already a detector in deepstream_test.py, do you mean you will need a second detector?
2. Do you mean you don’t want to display the output?

@Fiona.Chen
What is your detection pipeline? A separated process different to deepstream app?- Yes we are not using the nvinfer from deepstream. We are passing the frames(numpy) to our custom detection pipeline.
2. Do you mean you don’t want to display the output? - Yes we don’t want to display the output we just want the frames.

  1. So you just use deepstream to generate tiling frames(no inference is needed in deepstream) and want to send the frames to another process? If so, you don’t need to use deepstream.
  2. Just remove nvegltransform and nveglglessink, replace with fakesink will work.
1 Like

@Fiona.Chen

  1. So you just use deepstream to generate tiling frames(no inference is needed in deepstream) and want to send the frames to another process? If so, you don’t need to use deepstream. - We want the GPU accelerated decoding for RTSP source that’s why we thought of using deepstream for decoding.
  2. Just remove nvegltransform and nveglglessink, replace with fakesink will work. – Our platform is not Jetson so we are not using nvegltransform and we already replaced the nveglglessink to fakesink in the script(output display is still there).
  1. Back to the original question. We have no special suggestion for this because deepstream has no restriction for any way to transferring your data. The numpy data is just a normal python data, has nothing to do with deepstream, you can use them as you like.
  2. If there is only fakesink in the pipeline, there will be no display.

@Fiona.Chen
Please let me know if there any way to run the Deepstream for a specific number of frames.

Can you elaborate this with details? What is the scenario? Deepstream is a SDK, it is not an application. What do you mean to “run the Deepstream”? For inference several frames in a video or something else?

@Fiona.Chen

  1. Suppose I want to do the inference in deepstream-python. So it will do the inference until we terminate manually or there is any error in pipeline.

If we want to run the inference for let’s say 10000 frames. Is there any way to terminate the process after certain number of frames ?
2. If we are using deepstream-python how to restart the camera in between process( do we need to initialize the all the source elements again if we encounter any of the RTSP source is not working) ?

  1. Sure. You can stop the pipeline by change state or send “EOS” event to the pipeline once you want to stop, it will exit.
    Events: Seeking, Navigation and More,
    What are states?
  2. You can stop the rtspsrc rtspsrc by change the element state to NULL. And when you want the stream to start again, you can change the state to PLAYING again. No need to initialize if the ip camera is always working. BTW. Since there are very complicated threads management in gstreamer, you need to maintain the pipeline and element states very carefully in your application. And gstreamer python is not easy for debugging since all working parts are implemented in c/c++, it is better to use c/c++ instead to use python.

These are very basic gstreamer development skills, you can refer to Gstreamer resources for more information and samples.

Does the process mean inference?

1 Like

Thanks, @Fiona.Chen.

Yes.

There are some skills to skip some frames with deepstream nvinfer Gst-nvinfer — DeepStream 5.1 Release documentation.

1 Like

@Fiona.Chen

When we changed the source state from PLAYING → NULL the pipeline stopped for source but when we changed it back to PLAYING the pipeline is unable to initialize the source again.
When we changed the source state from PLAYING → PAUSED the pipeline didn’t stop for source(we are still getting the frames)