Using mediapipe gpu accelerated python solutions with cuda accelerated opencv

[ajinkyabhagat94.ab]

I built mediapipe facemesh solution with gpu support on Jetson Xavier AGX with reference to https://forums.developer.nvidia.com/t/mediapipe/121120/33
After obtaining desired landmarks I also used OpenCV with CUDA support to develop my own code to detect iris of the eye.

JetPack version: 4.5.1
OpenCV version: 4.1.1
Mediapipe version: 0.8
Python version: 3.6.9
Cuda: 10.2

However despite using GPU support and CUDA accelerated OpenCV, the maximum FPS I have attained is around 14. I am using an USB camera which can give me maximum 30 fps for image size 640*480. How do I optimise my code?

[AastaLLL]

Hi,

Could you monitor the GPU utilization to see if all the GPU resources are used?

$ sudo tegrastats

More, have you maximized the device performance first?

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

Could you also share how do you read the camera? Is it cv::VideoCapture()?

Thanks.

[ajinkyabhagat94.ab]

Thank you for the prompt reply!
After changing the power mode to maximize device performance, no change was observed.
This is the output of tegrastats while running the code. We can clearly see the GPU being utilized.

Yes I am reading the camera using cv2.VideoCapture(0).
I am calculating exponentially decaying average of fps using time module by python.
For reference -

current_fps = 1.0/(toc - tic)

fps = fps*0.95 + current_fps*0.05 

In the above code
tic = time at the start of the code
toc = time after processing (at the end of the code)
current_fps = fps for this frame
fps = decaying average of fps

I followed this method to give us an idea of the overall average, instead of just the current frame.

[AastaLLL]

Hi,

Based on the log, the GPU utilization is low.
You can try to read the camera with GStreamer to see if it helps.

Below is an example for your reference:

Optimizing opencv gstreamer with mipi and usb uvc cameras

Hi, For the USB camera, you may check if you can run VideoCapture cap("v4l2src device=/dev/video1 ! video/x-raw,width=1920,height=1080,format=UYVY,framerate=30/1 ! videoconvert ! video/x-raw,format=BGR ! appsink"); OpenCV is CPU based frameworks and please execute sudo jetson_clocks to get max performance on Jetson Nano. If OpenCV is not must-have in your implementation, you can consider to use gstreamer or jetson_multimedia_api, which use hardware DMA buffers to get optimal performance.

Thanks.

[ajinkyabhagat94.ab]

Hi,
So I built OpenCV from source with Cuda support as you can see below -

Still there was no significant improvement.
Do you mean using gstreamer with OpenCV? Because I even tried that, again no significant improvement.
It will be very helpful if you could provide links for implementing gstreamer or jetson_multimedia_api.
Thanks!

[DaneLLL]

Hi,
Could you try this sample and check if you can achieve 30fps:
V4l2src using OpenCV Gstreamer is not working in Jetson Xavier NX - #3 by DaneLLL

It is a sample for showing camera preview. would like to know it this achieves target frame rate.