Recording video from camera

It may be an issue with player, I don’t see that.
From the test_h264.mkv file recorded from gstreamer, what gives:

# I see framerate=30/1 from this
gst-launch-1.0 filesrc location= ./test_h264.mkv ! matroskademux ! h264parse ! fakesink -v

# I see a solid 30 fps when decoding on Jetson, even setting sync=true, and even with over video sinks.
gst-launch-1.0 filesrc location= ./test_h264.mkv ! matroskademux ! h264parse ! nvv4l2decoder ! fpsdisplaysink video-sink=fakesink text-overlay=0 sync=1 -v

yes it is playing back at 30fps, on the Jetson as well. I think the issue is that it has dropped every other frame so I am encoding 15fps video as 30fps video. Then when it plays back it is double speed.

If I set the FPS to 15 manually in the “out” statement then the video playback speed is correct.

How can I get it to record the full 30FPS without dropping frames?

On another note, I am noticing that the file size is quite small compared to the MP4 files I was recording before. Is it possible to reduce compression or record the video stream uncompressed?

You may try to find where is the bottleneck. First check what your V4L can provide:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

If this doesn’t show 30 fps, there is a problem with your USB cam. Be sure it is connected at least in USB2 (USB3 would be better) with lsusb -t. You may also try v4l2src property io-mode=2:

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

If the capture can run 30 fps, check the YUV to RGB conversion:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! videoconvert ! video/x-raw,format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

If this is where it slows down, you may try to perform the YUV → BGRx with HW:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

If this is ok, then the problem may be with opencv. Don’t use imshow and try lower modes provided by your camera.

first two are solid 30FPS.

Third one is ~17-18FPS

Fourth one is mostly 30FPS but not stable and sometimes drops to ~26FPS

So what do I need to do to perform the conversion from YUV with HW?

The HW conversion is done in 4th pipeline, however nvvidconv doesn’t provide BGR format but BGRx, so you have to use CPU videoconvert for removing the 4th byte in each 32 bits pixel.
As you seem close to 30 fps, you may try:

  • adding queue before appsink, so that opencv app may be scheduled to another CPU core.
  • adding another queue before videoconvert, or after v4l2src. May depend on what load all CPU already have.
  • use n-threads property of videoconvert instead, or in combination with the above, you may test various configurations.
  • try to do the BGRx conversion in the first nvvidconv copy to NVMM.

I tried adding in queues and n-thread as follows:

cap = cv2.VideoCapture("v4l2src device=/dev/video0 ! queue ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! queue ! videoconvert n-threads=4 ! video/x-raw,format=BGR ! queue ! appsink", cv2.CAP_GSTREAMER)

No change.

If I try and change format=BGR to BGRx in the cap statement it errors out.

I’d suggest trying just with gstreamer for now so that you get correct framerate if possible before involving opencv.

Sorry I made a typo in previous post (corrected now).
For moving the YUV->BGRx conversion in the first nvvidconv instance, you would try:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=BGRx' ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

You may also boost your Jetson if not already done:

# Max performance mode. With Xavier NX, alternatetively use -m2 for 6cores-15W
sudo nvpmodel -m0

# Boost clocks
sudo jetson_clocks

that command returns 30FPS now with a bit of variability to 29.96FPS at minimum.

Boosting the jetson does not show a difference, above command has same variability either way.

With the change all 4 of the commands of your previous message return 30FPS now.

So probably videoconvert was the bottleneck doing full YUV-> BGR conversion.
Now you’ll see if opencv is next bottleneck and let us know your best working solution.

How do I convert that gst-launch command into the opencv command?

My attempt just fails:

cap = cv2.VideoCapture("v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM), format=BGR' ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink", cv2.CAP_GSTREAMER)

Remove the single quotes from the string. These are only required from shell so that it doesn’t interpret the parentheses.

cap = cv2.VideoCapture("v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=YUY2,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=BGR ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink", cv2.CAP_GSTREAMER)

error with:

(python3:24182): GStreamer-CRITICAL **: 15:25:19.936: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed
Failed to open camera

Should be BGRx after the first nvvidconv.

Thanks! I just caught that myself and that solves it! 30FPS recording now.

Thank you again, I think that last one was just me trying to rush while I had your attention.

Is there any way I can maximize the output quality? I would like minimum or no compression if possible.

For H264 encoding, a quick solution may be to try main profile rather than baseline. Should be profile=2 in nvv4l2h264enc properties. Many other properties can change the quality, but there is no general answer, it depends on your case.
For nvv4l2h264enc properties, see:

gst-inspect-1.0 nvv4l2h264enc

Thank you, and hopefully one last question. I see a difference between the recorded video and the live stream video. The recorded video is all slightly washed out with lower contrast. It looks like there is an offset applied to the video data so blacks are no longer black?

Not sure I can really help further.
Check with various players.
If confirmed, you may have to adjust qp-range or other properties depending on your scene, encoding options and maybe more.
Someone else may further advise.

Ok Thank you again for all the help. I truly do appreciate it. I am attaching the working file here for anyone that comes across this thread with the same issue.

If anyone has an idea about the reduced contrast/black level issue I would appreciate any insight.

recordVideo.py (1.5 KB)

1 Like