Xavier CSI-2 capture issue with IMX577 camera

hello Ax1,

cool, you’re having a huge step to verify camera sensor driver and also its basic functionality.
according to Camera Architecture Stack, nvarguscamerasrc plugin were using different path to access the sensor stream.

you have to define CSI ports/streams, VI channels, pixel parser streams, number of lanes etc… into device tree property. please also review Device Properties and revise those property settings to adapt your sensor device tree.
you may also gather the trace log while enable the gstreamer pipeline for more details.
thanks

Hi Jerry,

Given V4L2 works, do you still think the issue with gst-launch command is related to set_mode_delay_ms? Apparently, Xavier does see incoming CSI-2 frames.

What else do you think could be wrong in this setup?

Thanks.

hello Ax1,

yes.
please note that,
v4l2 standard controls and nvarguscamerasrc plugin were taking different path of code-flow.
nvarguscamerasrc depends-on mostly sensor properties definition from device tree, such as mode properties, signal properties, and image properties.

you may also gather VI tracing logs while enable the stream with nvarguscamerasrc for reference,
thanks

Hi Jerry,

Attached is the trace log with the following command:
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=30/1’ ! nvvidconv flip-method=2 ! ‘video/x-raw, format=(string)I420’ ! xvimagesink -e

Thanks.

debug trace gstreamer.txt (1.2 MB)

hello Ax1,

I did not see obvious errors according to your tracing logs. (debug trace gstreamer.txt (1.2 MB))
actually, there’re SOF/EOF signaling, frame-number and also VI timestamps has also reported. your camera stream should be works.
for example,

rtcpu_vinotify_event: tstamp:42616629151 tag:CHANSEL_PXL_SOF channel:0x23 frame:30 vi_tstamp:42616606736 data:0x00000001
rtcpu_vinotify_event: tstamp:42616788979 tag:CHANSEL_PXL_EOF channel:0x23 frame:30 vi_tstamp:42616787965 data:0x04370002
rtcpu_vinotify_event: tstamp:42616789124 tag:ATOMP_FRAME_DONE channel:0x23 frame:30 vi_tstamp:42616787986 data:0x00000000

could you please describe the scenario with nvarguscamerasrc plugin.
you may also share the messages reported below gstreamer pipeline.
thanks

Hi Jerry,

It’s good to hear that. However, the problem we are facing is no image/video at all when doing gst-launch command. We only see the new window popping up with no image/video, and it’s just hang there.

Is there any specific criteria the nvarguscamerasrc plugin requires from incoming csi-2 frames to display it correctly?

I’ll share the messages when launching gstreamer soon.

Thanks.

hello Ax1,

besides using xvimagesink, please also refer to below pipeline I’m usually test with nvarguscamerasrc plugin.
for example,

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvoverlaysink -ev

Hi Jerry,

There is a note in Gstreamer User Guide: “The nvoverlaysink plugin is deprecated in L4T Release 32.1. Use nvdrmvideosink and nv3dsink instead for render pipelines with gst-v4l2 decoder.”

Will it work with R32.3.1?

Anyway, I’ll give it a try to see if there is any difference.

Thanks.

Hi Jerry,

It happened that the previous trace log was recorded when doing gst-launch after a v4l2 command. Today I found that even after v4l2 command stops, the stream continues. So the previous trace log was actually for v4l2, which we knew working. Sorry for the inconvenience.

I dump the trace log again right after power on to avoid above situation. It doesn’t show any valid frames. I did it with both vximagesink and nvoverlaysink and result is the same.

Attached are trace log with terminal message for both cases. They are much smaller. Can you please help to review it and tell us what could be wrong?

debug trace gstreamer nvoverlaysink.txt (44.6 KB)
debug trace gstreamer xvimagesink.txt (43.0 KB)

Thanks.

hello Ax1,

there’s no SOF and EOF signaling coming to VI engine.
please examine your device tree property settings while you’d verify the sensor driver with v4l2 standard controls.

here’re several suggestions for you to investigate this further.

  1. you may double check Sensor Pixel Clock configuration. It must be set correctly to avoid potential issues.
  2. please Maximizing Jetson Xavier Performance by execute jetson_clocks.sh script file.
  3. please also have a try to add set_mode_delay_ms property into your DT, it’s configuration for adding wait time for the first frame after capture starts.
  4. please check Debugg Tips session for the steps to examine your sensor settings.

Hi Jerry,

For #3, I added set_mode_delay_ms=“3000” and gst-launch command shows error message: “no camera available”. The video window pops up and closes immediately.

I actually added set_mode_delay_ms to video0 only, but gst-launch on video1 also shows same message.

Is set_mode_delay_ms supposed to be under tegra-camera-platform {} as well?

===========================================================
imx477_a@10 {
compatible = “nvidia,imx477”;
/* I2C device address */
reg = <0x10>;

/* V4L2 device node location */
devnode = "video0";

/* Physical dimensions of sensor */
physical_w = "3.674";
physical_h = "2.738";

sensor_model = "imx477";

set_mode_delay_ms = "3000";

/* Define any required hw resources needed by driver */
/* ie. clocks, io pins, power sources */
avdd-reg = "vana";
iovdd-reg = "vif";

Thanks.

hello Ax1,

set_mode_delay_ms should be added in sensor mode property, is used to override the default capture timeout value.
you should review your DT properties for such “no camera available" error messages,
thanks

Hi Jerry,

Thanks. I assume the sensor mode is the following. I will give it a try and update the results.

mode2 { // IMX477_MODE_1920X1080
	mclk_khz = "24000";
	num_lanes = "4";
	tegra_sinterface = "serial_a";
	discontinuous_clk = "no";
	dpcm_enable = "false";
	cil_settletime = "0";
	dynamic_pixel_bit_depth = "10";
	csi_pixel_bit_depth = "10";
	mode_type = "bayer";
	pixel_phase = "rggb";

	active_w = "1920";
	active_h = "1080";
	pixel_t = "bayer_rggb";
	readout_orientation = "90";
	line_length = "4512";
	inherent_gain = "1";
	mclk_multiplier = "200";
	pix_clk_hz = "840000000";
	
	set_mode_delay_ms = "3000";

	min_gain_val = "1.0";
	max_gain_val = "354";
	min_hdr_ratio = "1";
	max_hdr_ratio = "160";
	min_framerate = "2.84";
	max_framerate = "60";
	min_exp_time = "5.373";
	max_exp_time = "352112";
	embedded_metadata_height = "2";
};

Thanks.

Hi Jerry,

I tried the following, but none of them helps:

  1. Set set_mode_delay_ms = 3000, no difference in behavior.
  2. Set post_crop_frame_drop = 1, no difference
  3. Set max_framerate = 30 (instead of 60), no difference
  4. Execute jetson_clocks script to maximize performance, no difference

I haven’t tried to experiment with the clock settings yet. However, I would like to reiterate that without the MIPI CSI-2 Repeater device, the video streaming using gst-launch works fine.

Just to be clear, here is the block diagram of our setup with Leopard Imaging camera module:

Without MIPI CSI-2 Repeater, the setup works fine. Can we conclude the clock setttings are correct?

With CSI-2 Repeater, only CSI-2 packets are going thru the repeater. We also knew that it works with v4l2-ctl. So it looks like something is not configured properly on Xavier side when receiving CSI-2 packets. Are there any particular patterns Xavier VI expects to see when performing video streaming? Or could it be any timing requirements for this setup?

Also note that the CSI-2 Repeater has continuous clock out on DPHY TX, and discontinuous_clk = “no” in sensor mode setting.

Can you please suggest where else to investigate further? Thanks.

Alternatively, since v4l2-ctl works, is there a way to use v4l2 commands to stream video from camera and display it on the screen?

Thanks.

hello Ax1,

may I know which JetPack and L4T release version you’re working with?
please check release tag for the info, i.e. # cat /etc/nv_tegra_release
thanks

Hi Jerry,

It is R32.3.1

Thanks.

hello Ax1,

here’s changes to configure CSI clock modes in case sensor start streaming early,
could you please have a try to replace SCF binary with the attachment, it’s based-on l4t-r32.3.1, Topic165343_Jan28_libnvscf.zip (2.7 MB)
you may overwrite it with /usr/lib/aarch64-linux-gnu/tegra/libnvscf.so, and perform a warm-reboot to take effects.
thanks

Hi Jerry,

That helps to get the video streaming back with gstreamer.

I could be wrong, but my guess is that Xavier VI sees continuous CSI-2 clock coming in and expects CSI-2 packets, but there is none. That causes timeout and stalls the video stream.

Can you please provide more details about what changes in the new libnvscf file?

Thanks.

hello Ax1,

glad to know the pre-built update in post #32 works.
it’s modification to configure LP-bypass mode for continuous clock sensors, setting CSI CIL configure in case sensor start streaming early.

FYI,
this change already complete code-review process,
you should expect it’ll include in the next public release, (i.e. JetPack-4.6)
thanks

Hi Jerry,

Thanks very much for your great support!

Best regards.