Encoding annomalies on Jetson TX2

igal.kroyter · February 4, 2021, 12:01am

hi,

We try to encode 6 video channels 720p60. It works great on 2 unit but on the 3-rd unit the video(s) looks a little bit corrupted after 20 or so minutes of encoding, moreover it looks like the time encoded within the stream goes slower:
We record a running clock, then when we playback the recorded video on the VLC application (on a PC) the slide bar shows after 40 minutes a gap of 4 minutes, while the time on the video shows the correct time (to ensure it with an external stop watch). Above is beside the corrupt frames once per a few seconds.

We run the same software (equiped with GStreamer) and the units share the same kernel version.

When we reduce the number of encoded streams to 3 (on the “corrupted” unit), the recorded movies looks OK as on the “good” units.

How should we debug this issue?

DaneLLL · February 4, 2021, 2:00am

Hi,
Please share information about the cameras, such as vendoer and model ID. If the cameras are from our camera partners, we will check with them and do further investigation.

igal.kroyter · February 4, 2021, 6:50am

Hi, @DaneLLL,

our unit has an FPGA which converts HD-SDI to MIPI. The unit is fed with an HD-SDI from HDMI active converter , which video is fed from a laptop. So no cameras there.

We thought that the input video video might be an issue as a non stable provider of 60 fps, but we have used the same setup and only replaced the unit, another thing that we did was replacing only the Jetson module, so we have narrowed the problem to a specific Jetson module.

Please advise.

DaneLLL · February 4, 2021, 8:17am

Hi,
Do you capture through v4l2? If yes, please share information about the source:

$ v4l2-ctl -d /dev/videoX --list-formats-ext

igal.kroyter · February 4, 2021, 9:05am

@DaneLLL
a file with all formats per channel: v4l2-ctl (12.2 KB)

DaneLLL · February 5, 2021, 1:44am

Hi,
Please check if you can launch cameras at 60fps in the command:

$ gst-launch-1.0 v4l2src ! video/x-raw,format=UYVY,width=1280,height=720,framerate=60/1 ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

Check if the sources can steadily output in 720p60. If you run jetson_multimedia_api, you can try 12_camera_v4l2_cuda sample.

igal.kroyter · February 8, 2021, 8:29am

@DaneLLL, hi,

we’ve run the script above (included video source field) for 30 minutes on 6 independent input channels, where the average framerate was 60 and there were no frames drop.

please advise.

DaneLLL · February 8, 2021, 8:59am

Hi,
Please execute sudo nvpmodel -m 0 and sudo jetson-clocks. And run the pipeline:

$ gst-launch-1.0 v4l2src ! video/x-raw,format=UYVY,width=1280,height=720,framerate=60/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

If above command achieves performance, please run

$ gst-launch-1.0 v4l2src ! video/x-raw,format=UYVY,width=1280,height=720,framerate=60/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! omxh264enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

See if the bottleneck is in nvvidconv or omxh264enc. If you are on r32 release, please use nvv4l2h264enc instead of omxh264enc.

igal.kroyter · February 10, 2021, 7:11am

@DaneLLL, hi,

sudo nvpmodel -m 0 and sudo jetson-clocks is activate at all times upon unit boot - we activate it.
Though the connectivity between nvvidconv and omxh264enc per the following command line fails:

gst-launch-1.0 v4l2src ! video/x-raw,format=UYVY,width=1280,height=720,framerate=60/1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),width=1280,height=720,framerate=60/1,format=NV12’ ! omxh264enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

or

gst-launch-1.0 v4l2src ! video/x-raw,format=UYVY,width=1280,height=720,framerate=60/1 ! nvvidconv ! ‘video/x-raw(memory:NVMM),width=1280,height=720,framerate=60/1,format=I420’ ! omxh264enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

igal.kroyter · February 10, 2021, 7:11am

@DaneLLL, hi,

any advise?

DaneLLL · February 17, 2021, 5:20am

Hi,
Does it work with videotestsrc?

$ gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,format=UYVY,width=1280,height=720,framerate=60/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! omxh264enc ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

On r32 release, both v4l2src and videotestsrc commands work. We are not able to try r28 release due to WFH. Please give it a try.

igal.kroyter · February 21, 2021, 8:48am

@DaneLLL, hi,

We’ve run the above pipeline on six simultaneous terminals with no problems, though when we added the 7-th we have experienced a drop in average bit rate (no frames’ drop) (from 60 to 55), the more terminals we added the more the average dropped.

We understand that the Jetson TX2 supposed to support a single 4k channel encoding, or it can encode the following number of pixels:
3840 x 2160 x 60 x1= 497,664,000 [pixels]
while we utilize only:
1280 x 720 x 60 x 7 = 387,072,000 [pixels]
(1280 x 720 x 60 x 9 = 497,664,000 [pixels])
I guess that there is some overhead which we must reduce in order to handle the problem we are facing.
Any advise?

DaneLLL · February 22, 2021, 4:35am

Hi,
If videotestsrc works, v4l2src should also work fine. Somehow you cannot launch v4l2src ! nvvidconv ! omxh264enc ! …. You may check this first.

The bottleneck may not be in encoder but CPU. For running

v4l2src ! video/x-raw ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! ...

There is a memory copy of CPU buffers to NVMM buffers in nvvidconv. We have an optimal solution in jetson_multimedia_api and demonstrated in 12_camera_v4l2_cuda, but in gstreamer, need to have the memory copy and it may impact the performance. You can check system loading by executing sudo tegrastats.

igal.kroyter · February 22, 2021, 9:41am

Hi,

thanks for the reply.

the problem is that we need to synchronize the recorded video with audio and 12_camera_v4l2_cuda does not provide it while gstreamer does.

could you advise?

DaneLLL · February 22, 2021, 11:14am

Hi,
On r32 releases, we have implemented a plugin nvv4l2camerasrc to eliminate the memory copy. Since you are on r28 release, have to use v4l2src. One thing you can try is to build the plugin on r28. But the source code is open on r32, it’s not guaranteed to work. You can see the source code in JP4.4.1 or JP4.5.

igal.kroyter · March 1, 2021, 10:28am

@DaneLLL,

I finally have installed the JP4.5 on the evaluation board. and downloaded the Linux sources to both machines with r28 and r32. and tried to recompile the nvv4l2camerasrc:
r28:

nvidia@tegra-ubuntu:~/Linux_for_Tegra/source/public/gst-nvv4l2camera$ make clean
rm -rf gstnvv4l2camerasrc.o libgstnvv4l2camerasrc.so
nvidia@tegra-ubuntu:~/Linux_for_Tegra/source/public/gst-nvv4l2camera$ make
g++ -c gstnvv4l2camerasrc.cpp -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstnvv4l2camerasrc.o
Package libv4l2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libv4l2.pc’
to the PKG_CONFIG_PATH environment variable
No package ‘libv4l2’ found
gstnvv4l2camerasrc.cpp:40:21: fatal error: gst/gst.h: No such file or directory
compilation terminated.
Makefile:67: recipe for target ‘gstnvv4l2camerasrc.o’ failed
make: *** [gstnvv4l2camerasrc.o] Error 1

r32:

nvidia@nvidia:~/Linux_for_Tegra/source/public/gst-v4l2$ make clean
rm -rf gstv4l2videodec.o gstv4l2h265enc.o gstv4l2bufferpool.o gstv4l2vp9enc.o gstv4l2.o gstv4l2allocator.o v4l2-utils.o gstv4l2vp8enc.o gstv4l2videoenc.o gstv4l2object.o gstv4l2h264enc.o v4l2_calls.o libgstnvvideo4linux2.so
nvidia@nvidia:~/Linux_for_Tegra/source/public/gst-v4l2$ make
cc -c gstv4l2videodec.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2videodec.o
cc -c gstv4l2h264enc.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2h264enc.o
cc -c gstv4l2bufferpool.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2bufferpool.o
cc -c gstv4l2vp9enc.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2vp9enc.o
cc -c gstv4l2.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2.o
cc -c gstv4l2h265enc.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2h265enc.o
cc -c gstv4l2allocator.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2allocator.o
cc -c v4l2-utils.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o v4l2-utils.o
cc -c gstv4l2vp8enc.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2vp8enc.o
cc -c gstv4l2videoenc.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2videoenc.o
cc -c gstv4l2object.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o gstv4l2object.o
cc -c v4l2_calls.c -fPIC -DEXPLICITLY_ADDED=1 -DGETTEXT_PACKAGE=1 -DHAVE_LIBV4L2=1 -DUSE_V4L2_TARGET_NV=1 pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -I./ -I…/ -o v4l2_calls.o
cc -shared -o libgstnvvideo4linux2.so gstv4l2videodec.o gstv4l2h264enc.o gstv4l2bufferpool.o gstv4l2vp9enc.o gstv4l2.o gstv4l2h265enc.o gstv4l2allocator.o v4l2-utils.o gstv4l2vp8enc.o gstv4l2videoenc.o gstv4l2object.o v4l2_calls.o -lnvbuf_utils -lnvbufsurface pkg-config --libs gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 gstreamer-allocators-1.0 glib-2.0 libv4l2 -Wl,–no-undefined -L/usr/lib/aarch64-linux-gnu/tegra/ -Wl,-rpath,/usr/lib/aarch64-linux-gnu/tegra/

I can try and manually modify the make files but I am not sure if this was the intension.

any suggestions?