Deepstream Face Recognition OpenCV

Hello Experts,

In the fresh Jetson nano dev kit and flashed SD card image. I tried to run the face_recognition using opencv in Python.

Loaded the test mp4 file (containing faces) using gstreamer as
cv2.Videocapture(‘filesrc location=test.mp4 ! qtdemux ! queue ! h264parse ! omxh264dec ! nvoverlaysink’, cv2.CAP_GSTREAMER’),
but the face is not recognized precisely as done in Google colab using GPU.

Few questions,

  1. Using gstreamer does really fastens up the way frames are read. How to ensure/verify/examine that ?
  2. Reading the stored file (cctv footage) vs rtsp streaming which one has better performance in Jetson Nano.
  3. Can I use deepstream library examples to receive the video frames, detect/recognize the face and label the encodings ?

There might be a problem here. Your pipeline has nvoverlaysink as sink. This pipeline makes no processing, it doesn’t use opencv, it is just lauched by opencv.
For reading from opencv videoio, you would have to convert into BGR and use appsink as sink:

cv2.Videocapture('filesrc location=test.mp4 ! qtdemux ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

# or
cv2.Videocapture('filesrc location=test.mp4 ! qtdemux ! queue ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

Then do your processing with opencv and display results. imshow may work for low resoution*fps.
For higher values, you may use a videowriter to nvoverlaysink or nveglglessink.
If you want to use GPU, you may also get RGB frames with jetson-utils such as this example.

For your questions:

  1. The pipeline in your post 1 uses omxh264dec that runs on HW decoder. You may check its activity as NVDEC running tegrastats. It outputs into NVMM memory so that nvoverlaysink can directly display.
  2. First one is limited by storage+filesystem speed. Second one is limited by network and adapter and drivers, plus some buffering options.
  3. I cannot advise, I have poor knowledge of these samples.

Thank @Honey_Patouceul. Similarly for RTSP streaming what is the gstreamer arguments to be used ?

For reading a RTSP stream, see this post (Note that case was requiring jpeg output, in your case your would output BGRx from nvvidconv then videoconvert into BGR as in my previous post and you wouldn’t have to further decode the image).
For streaming the output, it would be more complicated, but you would try this one.

Again thanks @Honey_Patouceul. My objective is to read the RTSP stream and detect the face in a frame using dlib library.

Hi @Honey_Patouceul the link provided for reading RTSP stream causes the pipeline failure. Is there any alternative ways to fix it ?

"rtspsrc location=rtspt://user:psw@192.168.1.2 ! decodebin ! nvvidconv interpolation-method=5 ! video/x-raw, width=1280, height=720 ! jpegenc ! image/jpeg ! appsink"

I even adjusted resolution to 1080 pixel, still no luck.

When I tried to directly open the rtsp link with cv2.Videocapture() then it works, but CPU utilization is 100% in core and not sure accelerators are used.

As said above you wouldn’t use JPEG format for your case.
Use:

cv2.Videocapture('rtspsrc location=rtspt://user:psw@192.168.1.2 ! decodebin ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

I think decodebin would use HW accelerated plugin for H264 decoding. If not, you may try the following, but you would need to know the encoding and framerate of your source (here assuming 30 fps encoded into H264) :

cv2.Videocapture('rtspsrc location=rtspt://user:psw@192.168.1.2 ! application/x-rtp, media=video, encoding-name=H264 ! rtph264depay ! video/x-h264, clock-rate=90000, framerate=30/1 ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

@Honey_Patouceul Still the above command line is not working.

Logs as follows,

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module rtspsrc0 reported: Could not read from resource.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Traceback (most recent call last): 

First try without opencv, just a gstreamer pipeline:

gst-launch-1.0 rtspsrc location=rtspt://user:psw@192.168.1.2 ! decodebin ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! xvimagsink

Note that rtspt: is for a RTSP server over TCP. In most cases, they use UDP, so you would just use rtsp:.

Hi @Honey_Patouceul

It throwing error as follows,

Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://test:test@192.168.1.2:554
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not open resource for reading.
Additional debug info:
gstrtspsrc.c(5829): gst_rtspsrc_setup_auth (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0:
No supported authentication protocol was found
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

First, sorry there was a typo in the pipeline I posted, the last plugin for X display is xvimagesink.

This doesn’t seem to be the root cause of failure, though.

If

cv2.VideoCapture('rtsp://test:test@192.168.1.2:554')

works with ffmeg backend, you should be able to get it with gstreamer.

You would first prototype your pipeline from terminal with gst-launch:

gst-launch-1.0 rtspsrc location=rtsp://test:test@192.168.1.2:554 ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

If it works, for opencv videoCapture you would try:

cv2.VideoCapture('rtspsrc location=rtsp://test:test@192.168.1.2:554 ! decodebin ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink', cv2.CAP_GSTREAMER)

Hi @Honey_Patouceul

Yeah, but the pipeline creation itself fails. But debug print shows it is due to

No supported authentication protocol was found

in the capture.

Below one

cv2.VideoCapture('rtsp://test:test@192.168.1.2:554')

still works fine.

Any other way to approach the problem

Sorry, I can see no reason why you can access the rtsp stream from ffmeg backend but it would fail from gstreamer backend.
Did you try my exact pipeline above ? (In previous posts you used rtpst as my example with an online sample requiring TCP, at least for me).

Hi @Honey_Patouceul,

Yeah, the camera supports both TCP/UDP based RTSP streaming. In both the case it works with ffmpeg but not with gstreamer. I just copy pasted your exact command.

Is it possible to inspect the gstreamer source code from Jetpack SDK ?

Yes, sources of gstreamer are public. rtpsrc is not nvidia plugin.
Furthermore, you could use something like:

GST_DEBUG=*:3 gst-launch-1.0 rtspsrc location=rtsp://test:test@192.168.1.2:554 ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

that would set debug level 3 for all plugins. You may also use something like:

GST_DEBUG=rtspsrc:4 gst-launch-1.0 rtspsrc location=rtsp://test:test@192.168.1.2:554 ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

For rtspsrc only with level 4, but be aware that high debug levels may generate huge amount of messages that can slow down as well.

What is the encoding of your stream and payload of RTP ?

shall i use this command to get input from the webcam