[Python] How to convert OpenCV frame to CUDA memory

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hello,
I’m trying to modify detectnet-camera.py in order to get source from RTSP instead of local webcam.

so i changed:

camera = jetson.utils.gstCamera(opt.width, opt.height, opt.camera)

to

gst = "rtspsrc location=rtsp://admin:Password@192.168.0.204:554/ latency=200 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"
camera = cv2.VideoCapture(gst);

but before send frame to net.Detect is necessary to allocate it to CUDA memory and I was not able to find the way.
I found only the possibility to allocate to CUDA memory a local image using jetson.utils.loadImageRGBA(), but this means write to disk each frame read from RTSP and performance will drop down drammatically (and also SD life).
I tried using a ramdisk but is not a valid solution.

2

I found the way! But performance are very bad…

gst = "rtspsrc location=rtsp://admin:Password@192.168.0.200:554/ latency=200 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"
camera = cv2.VideoCapture(gst);

while camera.isOpened():
	ret, frame = camera.read()
	frame_rgba = cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA)
	cuda_frame = jetson.utils.cudaFromNumpy(frame_rgba)
	detections = net.Detect(cuda_frame, opt.width, opt.height)

In this way works, but I have very poor performance. In fact cudaFromNumpy() conversion takes 40-70ms and net.Detect() takes 50-60ms so the result is at least 100-120ms per frame.
For my project 9-10 FPS is not enought.

Any suggestion?

PS: RGBA conversion in Gstreamer doesn’t produce an improvment, in fact cv2.cvtColor(frame, cv2.COLOR_BGR2RGBA) takes only 1-2ms.

3

Hi Simone, if you want to avoid the extra overhead of using OpenCV capture and numpy conversion, see here in the jetson-utils code for where the GStreamer pipeline string is created in gstCamera:

https://github.com/dusty-nv/jetson-utils/blob/dbad83dc2f707e2e794640d7db4ad9977b2e26d6/camera/gstCamera.cpp#L422

You can try editing this and changing it over to your RTSP pipeline above. Since gstCamera expects to recieve NV12 format from GStreamer, you will want to remove these elements from your pipeline:

nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"

I don’t have RTSP IP camera so I can’t test it, but my guess is you could replace line 422 with something like the following:

ss << rtspsrc location=rtsp://admin:Password@192.168.0.200:554/ latency=200 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! ";

See the existing code for how it incorporates the desired video resolution, ideally that would be set in your RTSP pipeline somewhere or at least match the resolution coming from the camera.