How to use the cv2.imshow() function with DeepStream SDK 5.0

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

How to use the cv2.imshow() function with DeepStream SDK 5.0

I’ve heard that you can now access image data directly from DeepStream SDK 5.0.
Via pyds.get_nvds_buf_surface() we have confirmed that the image can be handled by OpenCV.

                n_frame=pyds.get_nvds_buf_surface(hash(gst_buffer),frame_meta.batch_id)
                frame_image=np.array(n_frame,copy=True,order='C')
                #covert the array into cv2 default color format
                print("ffs")
                frame_image=cv2.cvtColor(frame_image,cv2.COLOR_RGBA2BGRA)
                frame_image=draw_bounding_boxes(frame_image,obj_meta,obj_meta.confidence)

But I want the result to appear on the screen after processing in OpenCV.
So I want to use cv2.imshow(), but it stops at the cv2.waitKey(1) part.
Is there a way to display the results of processing in OpenCV?

Translated with DeepL Translate: The world's most accurate translator (free version)

3

Hi @natukage405,
please refer to deepstream_python_apps/deepstream_imagedata-multistream.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

5

I’m making it with that as a reference.
I’m having trouble because it doesn’t show up when I run cv2.imshow(“Test”,frame_image) after get_nvds_buf_surfac()

6

Hi natukage405, we are looking into it, and will get back to you.

7

Hi @natukage405,
can you save the image with cv2.imwrite()?
With below chnage, I can save the image as png and view it with display.
Would check if it’s your display issue or others.

Thanks!

@@ -123,6 +126,7 @@ def tiler_sink_pad_buffer_probe(pad,info,u_data):
                     frame_image=np.array(n_frame,copy=True,order='C')
                     #covert the array into cv2 default color format
                     frame_image=cv2.cvtColor(frame_image,cv2.COLOR_RGBA2BGRA)
+                    cv2.imwrite("test.png", frame_image)

                 save_image = True
                 frame_image=draw_bounding_boxes(frame_image,obj_meta,obj_meta.confidence)
8

I can confirm that the image can be saved with cv2.imwrite() and that I can reference the saved image.
But what I want to do is to display the image using cv2.imshow().
You need to investigate.

11

I have the same problem as natukage405, I want to display real-time frame by using cv2.imshow() but when I use it in the code (in deepstream_test1_usbcam in python examples), it does not work. I need the cv2.imshow() output for using it in another GUI.
I appreciate it if you could help me with this issue.

12

This is still on our list to investigate. Thanks for your patience.

14

Hi @zhliunycm2 @mchi , this is an issue for my use case too, kindly clarify if there is any resolution for the same.

15

Hi @rsnk96
Why can’t use the display DeepStream currently provides?
Can you elabrate your use case and why this is must-have ?

17

Because they might need to make some transformations. I think the point is showing a custom image (~custom overlay, shrink etc.). I was running into this same issue. Do you have any solution yet? Thank you.

18

Hi benkelaci,

Please help to open a new topic for your issue. Thanks

19

My problem is completely the same as the original topic, but you do not have any good solution (cv2.imshow with deepstream), just a workaround (that is sometimes ok, sometimes not).

20

Hi @benkelaci,
We have render plugin, may I know why imshow() is needed? What’s the use case?

21

e.g.: a custom drawing/texts on the screen. The default deepstream drawing is not optimal in many cases (especially if there are overlapped objects. E.g.: if I would add text that shows average of the last few object count etc. Can I use custom drawing on render plugin?

22

Hi, If you have to achieve cv2 imshow output, do something like

from multiprocessing import  Process, Pipe

 recv_cv2_frame, send_cv2_frame = Pipe()

def element_probe(pad, info, u_data):
    #probe code 
    #in prob getting access to required buffer see DS-imagedata-multistream
     send_cv2_frame.send(n_frame)

def show_video(recv_frame):
    while True:
        frame = recv_frame.recv()
        # display the frame to the screen
        cv2.imshow("cv2image", frame)
        cv2.waitKey(1)
    
def main():
    # DS Main Code ....
    p2 = Process(target=show_video, args=(recv_cv2_frame,))
    p2.start()

    print("Starting pipeline \n")
    pipeline.set_state(Gst.State.PLAYING)
    try:
        loop.run()
    except:
        pass
    # cleanup
   pipeline.set_state(Gst.State.NULL)

   p2.terminate()
   p2.join()
23

ozayrmoorad

I don’t quite understand how to use this. Can you describe in more detail how the program code using Deepstream and cv2.imshow should look like?
Multithreading, multiprocessing but how exactly?
Cv2.imshow as far as I know should be in the main thread.

Have there been any changes in Jetpack 4.6?

24

Hi i have edited my answer to be a bit more detailed about the code structure

25

ozayrmoorad
Thank you so much!
The way you presented works well.