Frame rate drops when accessing >4 V4L2 camera streams through a PCIe framegrabber

Hi,

I have an 8 channel PCIe framegrabber which can capture the 8 streams in parallel at 25 FPS (PAL). After much difficulty, I was able to finally get it working without any crashes/hangs. I am profiling the video output by running 6 gstreamer pipelines:

gst-launch-1.0 v4l2src device=/dev/video1 ! videoconvert ! videoscale ! "video/x-raw, width=704, height=576" ! videoconvert ! fpsdisplaysink -v

As I start running more pipelines, I start getting a drop in frame rate. It starts from around 25 for one camera, around 24-25 for 4 cameras and then starts taking a dip to 21-22 for 5 cameras and to 19-20 for 6 cameras.

I’ve tried this framegrabber on a desktop machine and it works perfectly without any frame drops. Any specific reason for this on TX1? Any pointers on how to debug this would be very much appreciated.

Thanks!

Hi

Since videoconvert, videoscale and fpsdisplaysink are all running on the CPU, the CPU load will probably be too much when capturing multiple sources. Take a look at your CPU load, it will probably be at 100% on at least one core, thus limiting frames that can be provided to the v4l2src and leading to reduced frame rate.
fpsdisplaysink creates a text overlay, which in my experience is a very expensive CPU operation.

Try replacing videoconvert and videoscale with nvvidconv (which is hardware-accelerated) and fpsdisplaysink with nvoverlaysink (take a look at the TX1 Multimedia User Guide from Nvidia).

It’s not 100% but it’s 80-90% on all cores. Even if I use a fpsdisplaysink with a fakesink, so that it just prints the fps in a terminal in 100 seconds, I get the same results. And I also have my own modified version of a Visionworks sample which displays multiple V4L2 streams at once. I don’t use videoscale element in the GStreamer wrapper used in nvxio for capturing frames, but still I’m getting the same results.

I’m using Gstreamer 1.8.0, which I built from source. Is there a way to use nvvidconv and nvoverlaysink with this Gstreamer version?

Thanks!

Alright, then i would think the videoconvert operations are what probably causes most of the load.
(If you want to be sure, you can try gst-top from https://github.com/kirushyk/gst-instruments which tells you which element uses the most CPU time)

Using nvvidconv and nvoverlaysink in newer GStreamer versions should be possible. Have you followed all the steps described in the Multimedia User Guide for building new GStreamer versions? The last step 13. Copy the nvidia gstreamer-1.0 libraries… should make the Nvidia-specific plugins available in the new GStreamer.
Edit:
If you built GStreamer from source without following the instructions, just copy the additional Nvidia files over as described.

Also since you are using GStreamer version 1.8.0, you could try the different io-mode parameters available in v4l2src. We had most success with the Userptr mode (io-mode=3). But this requires a suitable videosink like xvimagesink.

1 Like

I just found out and did that. Simple pipelines gave me an internal data flow error while playing. I tried replacing videoconvert, videoscale with nvvidconv and videosink with nvoverlaysink. Haven’t spent too much time on it yet, so not sure if I’m doing it right. I’ll spend some more time on it.

I’ll try gst-top and changing io-mode, but I’m not sure io-mode is gonna help much.

Thanks! I’ll update soon with the results.

Also, could it be a driver/system issue? Resource collision for buffers etc.?

Be aware that nvoverlaysink only works with the following formats: format=I420 or NV12. Also when using both nvvidconv and nvoverlaysink, I think it is necessary to convert to NVMM memory.

Examples:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=3840, height=2160, format=UYVY, framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=2160, format=I420, framerate=30/1' ! nvoverlaysink

(this works for me on GStreamer 1.2.4, but not on 1.8.0)

DISPLAY=:0 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=3 ! 'video/x-raw, width=3840, height=2160, format=UYVY, framerate=30/1' ! xvimagesink

(this works for me on GStreamer 1.8.0 with some patches [1])

For using videoconvert with io-mode=3 an additional patch was necessary [2] for us due to memory alignment.

I would not expect the driver to have a resource problem with such low datarates. But I dont have that much experience with the default v4l2 tegra driver, we have written our own driver for capturing 4K from HDMI.

Hope that helps,
Regards

[1]
https://github.com/InES-HPMM/linux-l4t/wiki/BuildGStreamer
[2]
https://github.com/InES-HPMM/linux-l4t/wiki/io-modes