Xavier CSI-2 capture issue with IMX577 camera

Ax1 · January 7, 2021, 2:26am

Hello,

We’re trying to bring up camera system with Xavier and run into issue that Xavier doesn’t seem to receive valid CSI-2 frames (thus cannot display video stream). However, when connecting camera CSI-2 to MIPI analyzer, it can capture correct 1080p image. Probing the signals on clock lane and 4 data lanes on Xavier shows no anomaly.

Below is top part of the trace log. Can you please help to tell what’s wrong with our setup? Thanks.

 kworker/0:3-1733  [000] ....   271.107380: rtos_queue_peek_from_isr_failed: tstamp:8737239558 queue:0x0bcbcf78
 kworker/0:3-1733  [000] ....   271.107393: rtcpu_start: tstamp:8737243333
 kworker/0:3-1733  [000] ....   271.107397: rtos_queue_send_from_isr_failed: tstamp:8737356608 queue:0x0bcb41f8
 kworker/0:3-1733  [000] ....   271.107398: rtos_queue_send_from_isr_failed: tstamp:8737356831 queue:0x0bcb8a60
 kworker/0:3-1733  [000] ....   271.107400: rtos_queue_send_from_isr_failed: tstamp:8737357051 queue:0x0bcba5e0
 kworker/0:3-1733  [000] ....   271.107401: rtos_queue_send_from_isr_failed: tstamp:8737357271 queue:0x0bcbb3a0
 kworker/0:3-1733  [000] ....   271.107403: rtos_queue_send_from_isr_failed: tstamp:8737357491 queue:0x0bcbc160
 kworker/0:3-1733  [000] ....   271.107405: rtcpu_string: tstamp:8737358159 id:0x04010000 str:"Configuring ISP GoS.

"
kworker/0:3-1733 [000] … 271.107450: rtcpu_string: tstamp:8737358429 id:0x04010000 str:" VM GOS[#0] addr=0xe4900000
"
kworker/0:3-1733 [000] … 271.107462: rtcpu_string: tstamp:8737358759 id:0x04010000 str:" VM GOS[#1] addr=0xe4901000
"
kworker/0:3-1733 [000] … 271.107472: rtcpu_string: tstamp:8737359130 id:0x04010000 str:" VM GOS[#2] addr=0xe4902000
"
kworker/0:3-1733 [000] … 271.107481: rtcpu_string: tstamp:8737359437 id:0x04010000 str:" VM GOS[#3] addr=0xe4903000
"
kworker/0:3-1733 [000] … 271.107496: rtcpu_string: tstamp:8737359747 id:0x04010000 str:" VM GOS[#4] addr=0xe4904000
"
kworker/0:3-1733 [000] … 271.107506: rtcpu_string: tstamp:8737360057 id:0x04010000 str:" VM GOS[#5] addr=0xe4905000
"
kworker/0:3-1733 [000] … 271.107515: rtcpu_string: tstamp:8737378461 id:0x04010000 str:“WARNING: t194/isp5.c:901 [config_channel] “”
kworker/0:3-1733 [000] … 271.107517: rtcpu_string: tstamp:8737378947 id:0x04010000 str:“All error notifications not enabled: correctable”
kworker/0:3-1733 [000] … 271.107519: rtcpu_string: tstamp:8737379142 id:0x04010000 str:”=0x00 uncorrectable=0x00"
kworker/0:3-1733 [000] … 271.107521: rtcpu_string: tstamp:8737379427 id:0x04010000 str:""
"
kworker/0:1-1165 [000] … 271.275439: rtos_queue_peek_from_isr_failed: tstamp:8742240065 queue:0x0bcbcf78
nvargus-daemon-8053 [000] … 271.437822: tegra_channel_open: vi-output, imx477 30-0010
nvargus-daemon-8053 [000] … 271.437846: tegra_channel_set_power: imx477 30-0010 : 0x1
nvargus-daemon-8053 [000] … 271.437884: camera_common_s_power: status : 0x1
nvargus-daemon-8053 [000] … 271.438866: tegra_channel_set_power: 15a00000.nvcsi–4 : 0x1
nvargus-daemon-8053 [000] … 271.438872: csi_s_power: enable : 0x1
nvargus-daemon-8053 [000] … 271.439117: tegra_channel_close: vi-output, imx477 30-0010
nvargus-daemon-8053 [000] … 271.439124: tegra_channel_set_power: imx477 30-0010 : 0x0
nvargus-daemon-8053 [000] … 271.439145: camera_common_s_power: status : 0x0
nvargus-daemon-8053 [000] … 271.439382: tegra_channel_set_power: 15a00000.nvcsi–4 : 0x0
nvargus-daemon-8053 [000] … 271.439384: csi_s_power: enable : 0x0
nvargus-daemon-8053 [000] … 271.440717: tegra_channel_open: vi-output, imx477 31-0010
nvargus-daemon-8053 [000] … 271.440726: tegra_channel_set_power: imx477 31-0010 : 0x1
nvargus-daemon-8053 [000] … 271.440784: camera_common_s_power: status : 0x1
nvargus-daemon-8053 [000] … 271.441625: tegra_channel_set_power: 15a00000.nvcsi–3 : 0x1
nvargus-daemon-8053 [000] … 271.441629: csi_s_power: enable : 0x1
nvargus-daemon-8053 [000] … 271.441821: tegra_channel_close: vi-output, imx477 31-0010
nvargus-daemon-8053 [000] … 271.441825: tegra_channel_set_power: imx477 31-0010 : 0x0
nvargus-daemon-8053 [000] … 271.441872: camera_common_s_power: status : 0x0
nvargus-daemon-8053 [000] … 271.442055: tegra_channel_set_power: 15a00000.nvcsi–3 : 0x0
nvargus-daemon-8053 [000] … 271.442058: csi_s_power: enable : 0x0
nvargus-daemon-8053 [000] … 271.443111: tegra_channel_open: vi-output, imx477 32-0010
nvargus-daemon-8053 [000] … 271.443147: tegra_channel_set_power: imx477 32-0010 : 0x1
nvargus-daemon-8053 [000] … 271.443215: camera_common_s_power: status : 0x1
kworker/0:1-1165 [000] … 271.443427: rtos_queue_peek_from_isr_failed: tstamp:8747239991 queue:0x0bcbcf78

JerryChang · January 7, 2021, 3:05am

hello Ax1,

may I know what’s your commands to access the sensor stream.
could you please check Applications Using V4L2 IOCTL Directly to use V4L2 IOCTL to verify basic functionality during sensor bring-up.

according to your tracing logs, it indicates there’s no response from VI engine.
could you please review your Port Binding definition for VI, NvCSI, and sensor modules.
thanks

Ax1 · January 7, 2021, 3:09am

Hi Jerry,

Thanks for quick reply.

We’re using the following command:
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=30/1’ ! nvvidconv flip-method=2 ! ‘video/x-raw, format=(string)I420’ ! xvimagesink -e

JerryChang · January 7, 2021, 3:25am

hello Ax1,

please tried with V4L2 IOCTL to verify basic camera functionality and share the results.
thanks

Ax1 · January 7, 2021, 6:46am

Hi Jerry,

I’d like to clarify a bit more about our setup:

With the camera connected directly to Xavier, we can see video stream using gst-launch command. So, I assume there is nothing wrong with the camera.
When we put an intermediate device between camera and Xavier, we have the issue as described. The intermediate device is acting as CSI-2 repeater that replicates the CSI-2 packets received from camera and sends them to Xavier (In a way, it’s similar to setup using FPD-Link Serializer and Deserializer).

Is there any complication and check point we should pay extra attention to this special setup?

Thanks.

JerryChang · January 7, 2021, 7:15am

hello Ax1,

okay, had you complete the SerDes configuration?
please access Jetson Virtual Channel with GMSL Framework Guide via download center for reference,
thanks

Ax1 · January 7, 2021, 7:29am

Hi Jerry,

We’re not using SerDes. Our setup with intermediate device is like CSI-2 repeater:
Camera => Intermediate Device [CSI-2 RX -> CSI-2 TX] => Xavier

The I2C is connected directly from Xavier to Camera. So, we assume everything is transparent in this setup: whatever camera sends is replicated and sent to Xavier.

Again, when we connect intermediate device CSI-2 TX to MIPI Analyzer, it can capture the image correclty.

We’re not sure what else we might miss in this setup.

Thanks.

JerryChang · January 7, 2021, 7:43am

hello Ax1,

assume there’s timing difference, could you please add set_mode_delay_ms into your device tree property.
it could configure the maximum waiting time for the first frame after capture starts, the unit is in milliseconds.
please check Sensor Software Driver Programming Guide for reference,
thanks

Ax1 · January 8, 2021, 7:59am

Hi Jerry,

Thanks for your suggestion. We’re looking into adding set_mode_delay_ms

A couple of questions:

Assuming current setup does not have set_mode_delay_ms in place, what is the default wait time after capture starts?
What is the range of set_mode_delay_ms?

Thanks.

JerryChang · January 11, 2021, 3:57am

hello Ax1,

the default waiting time is 1500ms for the start-of-frame signaling.
there’s the unit in milliseconds to configure the wait time for the first frame after capture starts, it is used to override the default capture timeout value.
thanks

Ax1 · January 12, 2021, 2:49am

Hi Jerry,

We tried V4L2 and was able to capture raw image and view it correctly. That means the Xavier receives CSI-2 packets, but somehow it doesn’t work when using Gstreamer.

I also attached the trace log. debug trace.txt (369.7 KB)

I have not touched device tree which complicates things on our setup (also because V4L2 works). Can you please review the trace and pinpoint what else could be wrong?

:~$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1000 -d /dev/video0
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.39 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.19 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.13 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.02 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.02 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 60.02 fps

Thanks.

JerryChang · January 12, 2021, 3:22am

hello Ax1,

cool, you’re having a huge step to verify camera sensor driver and also its basic functionality.
according to Camera Architecture Stack, nvarguscamerasrc plugin were using different path to access the sensor stream.

you have to define CSI ports/streams, VI channels, pixel parser streams, number of lanes etc… into device tree property. please also review Device Properties and revise those property settings to adapt your sensor device tree.
you may also gather the trace log while enable the gstreamer pipeline for more details.
thanks

Ax1 · January 14, 2021, 12:43am

Hi Jerry,

Given V4L2 works, do you still think the issue with gst-launch command is related to set_mode_delay_ms? Apparently, Xavier does see incoming CSI-2 frames.

What else do you think could be wrong in this setup?

Thanks.

JerryChang · January 14, 2021, 2:52am

hello Ax1,

yes.
please note that,
v4l2 standard controls and nvarguscamerasrc plugin were taking different path of code-flow.
nvarguscamerasrc depends-on mostly sensor properties definition from device tree, such as mode properties, signal properties, and image properties.

you may also gather VI tracing logs while enable the stream with nvarguscamerasrc for reference,
thanks

Ax1 · January 14, 2021, 3:03am

Hi Jerry,

Attached is the trace log with the following command:
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=30/1’ ! nvvidconv flip-method=2 ! ‘video/x-raw, format=(string)I420’ ! xvimagesink -e

Thanks.

debug trace gstreamer.txt (1.2 MB)

JerryChang · January 21, 2021, 6:26am

hello Ax1,

I did not see obvious errors according to your tracing logs. (debug trace gstreamer.txt (1.2 MB))
actually, there’re SOF/EOF signaling, frame-number and also VI timestamps has also reported. your camera stream should be works.
for example,

rtcpu_vinotify_event: tstamp:42616629151 tag:CHANSEL_PXL_SOF channel:0x23 frame:30 vi_tstamp:42616606736 data:0x00000001
rtcpu_vinotify_event: tstamp:42616788979 tag:CHANSEL_PXL_EOF channel:0x23 frame:30 vi_tstamp:42616787965 data:0x04370002
rtcpu_vinotify_event: tstamp:42616789124 tag:ATOMP_FRAME_DONE channel:0x23 frame:30 vi_tstamp:42616787986 data:0x00000000

could you please describe the scenario with nvarguscamerasrc plugin.
you may also share the messages reported below gstreamer pipeline.
thanks

Ax1 · January 21, 2021, 6:49am

Hi Jerry,

It’s good to hear that. However, the problem we are facing is no image/video at all when doing gst-launch command. We only see the new window popping up with no image/video, and it’s just hang there.

Is there any specific criteria the nvarguscamerasrc plugin requires from incoming csi-2 frames to display it correctly?

I’ll share the messages when launching gstreamer soon.

Thanks.

JerryChang · January 21, 2021, 7:14am

hello Ax1,

besides using xvimagesink, please also refer to below pipeline I’m usually test with nvarguscamerasrc plugin.
for example,

$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvoverlaysink -ev

Ax1 · January 21, 2021, 7:44am

Hi Jerry,

There is a note in Gstreamer User Guide: “The nvoverlaysink plugin is deprecated in L4T Release 32.1. Use nvdrmvideosink and nv3dsink instead for render pipelines with gst-v4l2 decoder.”

Will it work with R32.3.1?

Anyway, I’ll give it a try to see if there is any difference.

Thanks.

Ax1 · January 22, 2021, 12:29am

Hi Jerry,

It happened that the previous trace log was recorded when doing gst-launch after a v4l2 command. Today I found that even after v4l2 command stops, the stream continues. So the previous trace log was actually for v4l2, which we knew working. Sorry for the inconvenience.

I dump the trace log again right after power on to avoid above situation. It doesn’t show any valid frames. I did it with both vximagesink and nvoverlaysink and result is the same.

Attached are trace log with terminal message for both cases. They are much smaller. Can you please help to review it and tell us what could be wrong?

debug trace gstreamer nvoverlaysink.txt (44.6 KB)
debug trace gstreamer xvimagesink.txt (43.0 KB)

Thanks.