Corrupted Encoded Video via CSI

Hello,

We are trying to encode CSI video coming from an external video source which is not a camera with the following terminal Gstreamer command:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, width=1280, height=720, framerate=60/1, format=<b><u>UYVY</u></b>' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=60/1' ! omxh264enc profile=high bitrate=16000000 control-rate=constant ! 'video/x-h264, stream-format=(string)byte-stream'  ! h264parse ! mp4mux  ! queue ! filesink location=output2.mp4 –e

We are getting an upper portion of each frame is repeated (image is attached).
While when I utilize the following command the video is not corrupted, though I get CPU utilization which is not acceptable by our system:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, width=1280, height=720, framerate=60/1, format=<b><u>I420</u></b>' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=60/1' ! omxh264enc profile=high bitrate=16000000 control-rate=constant ! 'video/x-h264, stream-format=(string)byte-stream'  ! h264parse ! mp4mux  ! queue ! filesink location=output2.mp4 –e

Please advise.

Hi igal,
Can you try to run the pipeline with jetson_clocks.sh?

Or try below pipeline?

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, width=1280, height=720, framerate=60/1, format=UYVY' ! xvimagesink

Are you on r28.2.1?

DaneLLL, hi.

Thanks for the reply.

We are using r28.2.1 and with whatever version is available in Jetpack 3.3.

The pipeline with the framerate=60/1 field did not work on the Jetpack 3.3 version, while on the 28.2.1 it did work.
When we removed the field, we got the same result the higher lines were multiplied with the lower lines exactly as in the image in message #1.

Regards.

hello igal.kroyter,

there’s known issue with l4t-r28.2.1’s VI driver.
could you please apply the kernel patches from Topic 1038067.

besides v4l2src, could you please also get the raw dumps by v4l2-ctl to check the capture images.
for example,

v4l2-ctl -d /dev/video0 --set-fmt-video=width=2592,height=1944,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.raw

you’re able to analysis that by 3rdparty tools, 7yuv.
thanks

Hi,

I have tried to added the patches, though the whole video becomes shaky every second.

I have printed some debug output with the trace capabilities, and I have notices that every 5,000,000 [nSec] (I guess) rtos_queue_peek_from_isr_failed message is printed out.

It looks like when the encoding is applied to frame rate of 60 Fps the lines issue appears, while when the frame rate drops below, the video is encoded with a satisfying quality.

It is worth noting that I also see the following error in the terminal “MW_ACK_DONE syncpoint time out!0” error every frame (16.66mSec).

Any suggestions?

CSI trace.txt (561 KB)

Anyone?

Hi,
Per message #5, if I have every frame a failure I guess that the frame size that I use is incorrect?
This is a portion of our device-tree which includes the frame definition:

pixel_t	= "uyvy";	
active_w = "1280";
active_h = "720";
readout_orientation = "90";
line_length = "1650";
inherent_gain = "1";
mclk_multiplier = "3.13";
pix_clk_hz = "74250000"; // 1650 x 750 x 60 = 74250000

Is our case the CSI video source provides the full frame to the Jetson via the CSI-2 port.
Does it mean that we need to set the encoding Gstreamer script line with the following parameters:

gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, <u>width=1650, height=750</u>, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), <u>width=1280, height=720</u>, format=NV12, framerate=60/1' ! omxh264enc profile=high bitrate=16000000 control-rate=constant ! 'video/x-h264, stream-format=byte-stream'  ! mp4mux ! queue ! filesink location=output2.mp4 –e

Do note the difference between the device video input frame size to the streaming frame size?
Regards.

Hi,
Please run v4l2-ctl command to check the raw image and tune the clock.

v4l2-ctl -d /dev/video2 --set-fmt-video=width=1280,height=720,pixelformat=UYVY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100 --stream-to=test.raw

It should not be an issue in the camera source, not gstreamer commands.

DaneLLL,

Could you please explain what do you mean by “tune the clock”?

mode0 { //OV5693_MODE_1280X720
	mclk_khz		= "24000";
	num_lanes		= "2";
	tegra_sinterface 	= "serial_a";
	discontinuous_clk	= "no";
	dpcm_enable		= "false";
	cil_settletime		= "0";

	pixel_t			= "uyvy";	
	active_w		= "1280";
	active_h		= "720";
	readout_orientation 	= "90";
	line_length		= "1650";
	inherent_gain		= "1";
	mclk_multiplier		= "3.13";
	pix_clk_hz		= "74250000"; // 1650 x 750 x 60 = 74250000

	min_gain_val		= "1";
	max_gain_val		= "256";
	min_hdr_ratio		= "1";
	max_hdr_ratio		= "64";
	min_framerate		= "2";
	max_framerate		= "60";
	min_exp_time		= "34";
	max_exp_time		= "660000";
				
	horz_front_porch	= "74";
	horz_sync		= "80";
	horz_back_porch		= "216";
	vert_front_porch	= "23";
	vert_sync		= "2";
	vert_back_porch		= "5";				
};

Regards.

Hi,
Please try different pix_clk_hz. And maybe try 30fps.

Also please try max clocks
https://devtalk.nvidia.com/default/topic/1026128/jetson-tx2/tegra-multimedia-apis-could-get-raw-image-via-v4l2-only-at-15fps/post/5221392/#5221392

A user has shared patch for OV10635. You may also check your sensor driver.
https://devtalk.nvidia.com/default/topic/1026128/jetson-tx2/tegra-multimedia-apis-could-get-raw-image-via-v4l2-only-at-15fps/post/5227866/#5227866

DaneLLL,

We cannot use 30 fps we only have 60fps as our video source can provide this frame rate.
The patch here is to modify the camera configuration (#5227866), while we are using a different video source. The problem that I do not understand how to figure out the reason for the failure. We see a good picture accept to the above mention issue.

How much do you propose to increase the clock by? set it to 90000000[Hz] ?

BTW, after running your command we got this message in the terminal: No vi channel is active

The clocks’ extension was already provided without any change.

Regards.

DaneLLL and JerryChang, hi,

I have run some comparison between L4T versions 31.1(Kernel 4.9) and 28.2.1(Kernel 4.4.38) and seems like some part of the patches that you proposed were implemented in 31.1 version, but not only.
I mean to changes which were applied to the vi4_fops.c file. Changes that deal with signal stability issues and etc.
Another thing that I’ve noticed which might confuse is the fact that in the new kernel the printout is:
MW_ACK_DONE syncpoint time out!%d\n” while in the old one is “ATOMP_FE syncpt timeout! err = %d\n”, while both invoked in the tegra_channel_release_frame function.

Regards.

DaneLLL, hi,

I have figured out the problem that I had the chan->timeout parameter in the function tegra_channel_release_frame was equal to 0, once I have set it to -1 I was able to encode a satisfying stream at constant 60fps. This value set the wait for the completion of the frame receive to forever (per the kernel code, I’ll fix it later to an acceptable value).

Now… the problem is that when I try to encode 2 streams from two different sources the frame rate drops after 15 seconds to ~52fps ; when I encode 3 streams the frame rate drops after 15 seconds to ~33fps. No error s are displayed.
I have also increased the clock rate, and decreased the bitrate down to 4Mbps, without any changes.

Any suggestions?

hello igal.kroyter,

to clarify, patches from Topic 1038067 is based-on l4t-r28.2.1.

  1. may I have your confirmation which BSP release you’re currently working with?
  2. were you still meet corrupted encoded video via CSI, what’s modification you had done besides above (comment #13)?
  3. could you please also refer to Jetson_TX2_Camera_BringUp page to enable your latest VI tracing logs. thanks

Hi JerryChang,

  1. I work with l4t-r28.2.1 (kernel 4.4.38)
  2. No
  3. Working on it.

We’ve managed to eliminate the FPS drop by adding the output-buffers=5(or above) flag to the nvvidconv plug-in.

Regards.