Corrupted Encoded Video via CSI

igal.kroyter · December 9, 2018, 9:23am

Hello,

We are trying to encode CSI video coming from an external video source which is not a camera with the following terminal Gstreamer command:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, width=1280, height=720, framerate=60/1, format=<b><u>UYVY</u></b>' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=60/1' ! omxh264enc profile=high bitrate=16000000 control-rate=constant ! 'video/x-h264, stream-format=(string)byte-stream'  ! h264parse ! mp4mux  ! queue ! filesink location=output2.mp4 –e

We are getting an upper portion of each frame is repeated (image is attached).
While when I utilize the following command the video is not corrupted, though I get CPU utilization which is not acceptable by our system:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, width=1280, height=720, framerate=60/1, format=<b><u>I420</u></b>' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=60/1' ! omxh264enc profile=high bitrate=16000000 control-rate=constant ! 'video/x-h264, stream-format=(string)byte-stream'  ! h264parse ! mp4mux  ! queue ! filesink location=output2.mp4 –e

Please advise.

DaneLLL · December 10, 2018, 2:49am

Hi igal,
Can you try to run the pipeline with jetson_clocks.sh?

Or try below pipeline?

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, width=1280, height=720, framerate=60/1, format=UYVY' ! xvimagesink

Are you on r28.2.1?

igal.kroyter · December 10, 2018, 7:37am

DaneLLL, hi.

Thanks for the reply.

We are using r28.2.1 and with whatever version is available in Jetpack 3.3.

The pipeline with the framerate=60/1 field did not work on the Jetpack 3.3 version, while on the 28.2.1 it did work.
When we removed the field, we got the same result the higher lines were multiplied with the lower lines exactly as in the image in message #1.

Regards.

JerryChang · December 10, 2018, 8:16am

hello igal.kroyter,

there’s known issue with l4t-r28.2.1’s VI driver.
could you please apply the kernel patches from Topic 1038067.

besides v4l2src, could you please also get the raw dumps by v4l2-ctl to check the capture images.
for example,

v4l2-ctl -d /dev/video0 --set-fmt-video=width=2592,height=1944,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.raw

you’re able to analysis that by 3rdparty tools, 7yuv.
thanks

igal.kroyter · December 11, 2018, 11:16am

Hi,

I have tried to added the patches, though the whole video becomes shaky every second.

I have printed some debug output with the trace capabilities, and I have notices that every 5,000,000 [nSec] (I guess) rtos_queue_peek_from_isr_failed message is printed out.

It looks like when the encoding is applied to frame rate of 60 Fps the lines issue appears, while when the frame rate drops below, the video is encoded with a satisfying quality.

It is worth noting that I also see the following error in the terminal “MW_ACK_DONE syncpoint time out!0” error every frame (16.66mSec).

Any suggestions?

CSI trace.txt (561 KB)

igal.kroyter · December 12, 2018, 11:33am

Anyone?

igal.kroyter · December 13, 2018, 4:23am

Hi,
Per message #5, if I have every frame a failure I guess that the frame size that I use is incorrect?
This is a portion of our device-tree which includes the frame definition:

pixel_t	= "uyvy";	
active_w = "1280";
active_h = "720";
readout_orientation = "90";
line_length = "1650";
inherent_gain = "1";
mclk_multiplier = "3.13";
pix_clk_hz = "74250000"; // 1650 x 750 x 60 = 74250000

Is our case the CSI video source provides the full frame to the Jetson via the CSI-2 port.
Does it mean that we need to set the encoding Gstreamer script line with the following parameters:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, <u>width=1650, height=750</u>, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), <u>width=1280, height=720</u>, format=NV12, framerate=60/1' ! omxh264enc profile=high bitrate=16000000 control-rate=constant ! 'video/x-h264, stream-format=byte-stream'  ! mp4mux ! queue ! filesink location=output2.mp4 –e

Do note the difference between the device video input frame size to the streaming frame size?
Regards.

DaneLLL · December 13, 2018, 7:51am

Hi,
Please run v4l2-ctl command to check the raw image and tune the clock.

v4l2-ctl -d /dev/video2 --set-fmt-video=width=1280,height=720,pixelformat=UYVY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100 --stream-to=test.raw

It should not be an issue in the camera source, not gstreamer commands.

igal.kroyter · December 13, 2018, 9:39am

DaneLLL,

Could you please explain what do you mean by “tune the clock”?

mode0 { //OV5693_MODE_1280X720
	mclk_khz		= "24000";
	num_lanes		= "2";
	tegra_sinterface 	= "serial_a";
	discontinuous_clk	= "no";
	dpcm_enable		= "false";
	cil_settletime		= "0";

	pixel_t			= "uyvy";	
	active_w		= "1280";
	active_h		= "720";
	readout_orientation 	= "90";
	line_length		= "1650";
	inherent_gain		= "1";
	mclk_multiplier		= "3.13";
	pix_clk_hz		= "74250000"; // 1650 x 750 x 60 = 74250000

	min_gain_val		= "1";
	max_gain_val		= "256";
	min_hdr_ratio		= "1";
	max_hdr_ratio		= "64";
	min_framerate		= "2";
	max_framerate		= "60";
	min_exp_time		= "34";
	max_exp_time		= "660000";
				
	horz_front_porch	= "74";
	horz_sync		= "80";
	horz_back_porch		= "216";
	vert_front_porch	= "23";
	vert_sync		= "2";
	vert_back_porch		= "5";				
};

Regards.

DaneLLL · December 13, 2018, 11:03am

Hi,
Please try different pix_clk_hz. And maybe try 30fps.

Also please try max clocks
[url]https://devtalk.nvidia.com/default/topic/1026128/jetson-tx2/tegra-multimedia-apis-could-get-raw-image-via-v4l2-only-at-15fps/post/5221392/#5221392[/url]

A user has shared patch for OV10635. You may also check your sensor driver.
[url]https://devtalk.nvidia.com/default/topic/1026128/jetson-tx2/tegra-multimedia-apis-could-get-raw-image-via-v4l2-only-at-15fps/post/5227866/#5227866[/url]

igal.kroyter · December 13, 2018, 11:39am

DaneLLL,

We cannot use 30 fps we only have 60fps as our video source can provide this frame rate.
The patch here is to modify the camera configuration (#5227866), while we are using a different video source. The problem that I do not understand how to figure out the reason for the failure. We see a good picture accept to the above mention issue.

How much do you propose to increase the clock by? set it to 90000000[Hz] ?

BTW, after running your command we got this message in the terminal: No vi channel is active

The clocks’ extension was already provided without any change.

Regards.

igal.kroyter · December 16, 2018, 10:08am

DaneLLL and JerryChang, hi,

I have run some comparison between L4T versions 31.1(Kernel 4.9) and 28.2.1(Kernel 4.4.38) and seems like some part of the patches that you proposed were implemented in 31.1 version, but not only.
I mean to changes which were applied to the vi4_fops.c file. Changes that deal with signal stability issues and etc.
Another thing that I’ve noticed which might confuse is the fact that in the new kernel the printout is:
“MW_ACK_DONE syncpoint time out!%d\n” while in the old one is “ATOMP_FE syncpt timeout! err = %d\n”, while both invoked in the tegra_channel_release_frame function.

Regards.

igal.kroyter · December 16, 2018, 3:21pm

DaneLLL, hi,

I have figured out the problem that I had the chan->timeout parameter in the function tegra_channel_release_frame was equal to 0, once I have set it to -1 I was able to encode a satisfying stream at constant 60fps. This value set the wait for the completion of the frame receive to forever (per the kernel code, I’ll fix it later to an acceptable value).

Now… the problem is that when I try to encode 2 streams from two different sources the frame rate drops after 15 seconds to ~52fps ; when I encode 3 streams the frame rate drops after 15 seconds to ~33fps. No error s are displayed.
I have also increased the clock rate, and decreased the bitrate down to 4Mbps, without any changes.

Any suggestions?

JerryChang · December 17, 2018, 3:17am

hello igal.kroyter,

to clarify, patches from Topic 1038067 is based-on l4t-r28.2.1.

may I have your confirmation which BSP release you’re currently working with?
were you still meet corrupted encoded video via CSI, what’s modification you had done besides above (comment #13)?
could you please also refer to Jetson_TX2_Camera_BringUp page to enable your latest VI tracing logs. thanks

igal.kroyter · December 17, 2018, 7:03am

Hi JerryChang,

I work with l4t-r28.2.1 (kernel 4.4.38)
No
Working on it.

We’ve managed to eliminate the FPS drop by adding the output-buffers=5(or above) flag to the nvvidconv plug-in.

Regards.

Topic		Replies	Views
Encoding annomalies on Jetson TX2 Jetson TX2 encoder	17	643	October 18, 2021
TX2 Camera not working? Jetson TX2	24	14532	October 18, 2021
Jetson L4T 35.1 - CSI camera's issue with gstreamer Jetson AGX Xavier camera	19	1911	November 2, 2022
Capture raw video through Gstreamer with CSI cameras Jetson TX2	38	17584	October 18, 2021
Frame rate drop during single node capture Jetson TX2 camera	38	2592	October 18, 2021
CSI latency is over 80 milliseconds...? Jetson TX2	27	11953	November 19, 2019
CSI camera is not working in Jetson tx2i Jetson TX2 camera	13	1187	October 18, 2021
CSI video input problem (PXL_SOF, trace, no rtcpu information) Jetson TX2	7	1317	October 18, 2021
TX2 H264 RTSP Stream decoding issues Jetson TX2	27	11586	October 18, 2021
About the frame rate reduction problem of capture processing using CSI of Jetson TX2 Jetson TX2 camera	54	2216	October 18, 2021

Corrupted Encoded Video via CSI

Related topics