Object detection and recording video

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Hello everyone,

Has anyone succeded to do object detection and good video recording at the same time on Jetson Nano?

What I would like to achieve is at least 30fps for recording.
The object detection could run slower (at 15Hz would be enought) - since the tracking target moves slow.

At the moment I’m using detection from :

Capturing the video with OpenCV and recording it.

The fraction of the code that I’m using is :

def gstreamer():
    return ('v4l2src device=/dev/video0 ! '
            'video/x-raw, format=YUY2, '
            'width=640,height=480,framerate=30/1 ! '
            'videoconvert ! video/x-raw, '
            'format=BGR ! appsink drop=1 ')
def main():
        net = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)
        cap = cv2.VideoCapture(gstreamer(), cv2.CAP_GSTREAMER)
        w = cap.get(cv2.CAP_PROP_FRAME_WIDTH)
        h = cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
        fps = cap.get(cv2.CAP_PROP_FPS)
        print("w,h,fps ",w,h,fps)

        gst_out = "appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=test.mkv "
        
        out = cv2.VideoWriter(gst_out, cv2.CAP_GSTREAMER, 0, float(fps), (int(w), int(h)))
        if not out.isOpened():
                print("Failed to open output")
                exit()
        if not cap.isOpened():
            print("Cannot open camera")
            exit()
        time.sleep(2.0)

        while True:
                ret, frame = self.cap.read()
                if not ret:
                    exit()
                self.out.write(frame)
                
                input_image = cv2.cvtColor(frame, cv2.COLOR_RGB2RGBA).astype(np.float32)
                input_image = jetson.utils.cudaFromNumpy(input_image)
                detections = self.net.Detect(input_image, frame.shape[1], frame.shape[0])

                key = cv2.waitKey(1) & 0xFF        
                if key == ord("q"):
                        break
        cap.release()
        out.release()
        cv2.destroyAllWindows()

What I have achieved so far was a recorded video, but the duration was approx. half of the time.

I also measured the time, which is needed for grabbing frame/ recording / object detection and are as follows:
(in seconds)
Grabbing took:
0.0004832744598388672
Recording took:
0.00046253204345703125
Detections took:
0.05196261405944824

So what I can see is that the object detection takes too long, so the cpu is unable to record video fast enough. Thats why some frames are skipped and the duration of the video is 1/2 of the real scene duration.

What could be a solution?

Do you suggest any other implementation?

Do you suggest the use of threads? How could I run object detection and recording video on separate processes?

p.s…I’m using the OV2710 camera, which for 30fps supports only 640x480 resolution :/

thank you for your help and time.

2

Hi @nejc.likar,

Can you try running this command from jetson-inference to see if the results are any better?

$ detectnet.py --input-width=640 --input-height=480 /dev/video1 test.mkv

For more command-line options, see here: https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md

4

Hi @dusty_nv ,

Thank you for your answer - very helpful!

So, I tried running your code “detectnet” from the terminal, with different resolutions (640x480, 1280x720, 1920x1080) and it looked that the video recording was with correct timing. And in the file details I can see they are 30fps. Hovewer, when playing the videos, they seem a bit lagged.
Maybe its a problem with the camera. I will try next week when I get a IMX447.

I wonder why my code above doesnt produce same results. Is it problem with OpenCV?

How can I use your detection / screening code in my program? … Is there a source code? So I dont need to run it from the console.

5

OK cool - I am not sure about OpenCV (maybe it doesn’t add timestamping to the output video frames or is running slow), but you can view the python source code in detectnet.py:

7

@dusty_nv

I tried running the code :

import jetson.inference
import jetson.utils
import time
import argparse
import sys

# create video output object 
output = jetson.utils.videoOutput(f"output{time.time()}.mp4","--headless")
	
# load the object detection network
net = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)

# create video sources
input = jetson.utils.videoSource("v4l2:///dev/video0")


# process frames until the user exits
while True:
	# capture the next image
	img = input.Capture()

	# detect objects in the image (with overlay)
	detections = net.Detect(img)

	# print the detections
	print("detected {:d} objects in image".format(len(detections)))

	for detection in detections:
		print(detection)

	# render the image
	output.Render(img)


	# exit on input/output EOS
	if not input.IsStreaming() or not output.IsStreaming():
		break

Which produced the 1280x720 video (default). The timing was correct.

But the video is only 13fps.

I guess some other code runs when executing ./detectnet from the terminal.

8

detectnet is the C++ version, and detectnet.py is the Python version. Does detectnet.py record the video as desired?

9

@dusty_nv

When I run “detectnet” or “detectnet.py”, the recorded video framerate is 30fps. There is no information about the bitrate.

When I run the code Object detection and recording video - #7 by nejc.likar, the framerate is only about 15fps. I can see the information about the bitrate.

Really not sure what I’m missing here :/

p.s. How is the video recorder in jetson-utils? You dont use OpenCV, right?

Thank you for any help…

10

I don’t use OpenCV - I use a GStreamer pipeline. This pipeline gets printed out to the terminal when the videoOutput object is created. I am also using some optimizations such as CUDA/GPU memory and the hardware encoders. I also timestamp the video frames.

The encoding bitrate you can set with the --bitrate command-line option. You can read more about this here: https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md#output-options

11

Ok, Thank you,

I still don’t know why the recorded videos have different FPS :/

13

@dusty_nv Is it possible to add custom overlay text & image to the recorded video?

thank you

14

Hi @nejc.likar, yes you can see these examples for drawing overlays: https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-image.md#overlay

And see the source code to imagenet.cpp / imagenet.py for example of drawing text.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.