tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:3607277994400 data:0x00000000000000a0

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
1

Hi Forum,

I would like to continue my bringing up of an FPGA based LVDS to MIPI-CSI2 bridge previously discussed here : Debug PHY_INTR0 phy:0 cil:0 st:0 vc:0 status

The FPGA designer did some modification and we did have some progress : fully visible frame but only one frame was captured :

IMG_5761
IMG_57612016×1512 494 KB

And we observed in the trace :

     kworker/2:2-87      [002] ....  3583.342579: rtcpu_vinotify_event: tstamp:112694368651 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:3606210266016 data:0x00000000000000a0
     kworker/2:2-87      [002] ....  3583.342580: rtcpu_vinotify_event: tstamp:112694368812 cch:0 vi:0 tag:FS channel:0x00 frame:1 vi_tstamp:3606210266016 data:0x0000000000000010
     kworker/2:2-87      [002] ....  3583.342581: rtcpu_vinotify_event: tstamp:112694368952 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:1 vi_tstamp:3606210270080 data:0x00000000000003c9
     kworker/2:2-87      [002] ....  3583.398540: rtcpu_vinotify_error: tstamp:112695114725 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:3606243632544 data:0x00000000000000a0
     kworker/2:2-87      [002] ....  3583.398542: rtcpu_vinotify_error: tstamp:112695115303 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:1 vi_tstamp:3606243636608

It looks similar to the issue in the following discussion : Boosting clocks is breaking camera stream

Do you think it is the same or could you point out something wrong with the full trace in attachment, please ? Was the above issue caused by the Tx side or Rx side ?

Also, how to know how many data lines that the sensor / TX side sent?

Update : We already ruled out the cause of boosting clocks : the issue continued happening.

trace.txt (2.5 MB)

Thanks in advance and best regards,
Khang

2

Hi @khang.l4es

From your log, it looks like the driver is expecting YUV data-type (0x1E), but the received data is not matching that. Are you sending metadata along with the image? If that’s the case, try to disable it or define the number of lines on the device-tree using embedded_metadata_height

Regards,

Enrique Ramirez
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com

3

Hi @enrique.ramirez,

Thanks for your suggestion, but I defined embedded_metadata_height = 0 in the device-tree, and the fpga should not send any meta data line :

mode0 {
                mclk_khz = "24000";
                set_mode_delay_ms = "5000";
                num_lanes = "4";
                tegra_sinterface = "serial_a";
                phy_mode = "DPHY";
                discontinuous_clk = "yes";
                dpcm_enable = "false";
                cil_settletime = "0";

                active_w = "1920";
                active_h = "1080";
                mode_type = "yuv";
                pixel_phase = "uyvy";
                pixel_t = "yuv_uyvy16";

                dynamic_pixel_bit_depth = "16";
                csi_pixel_bit_depth = "16";
                readout_orientation = "0";
                line_length = "2200"; // 2000 for EV9500M
                inherent_gain = "1";
                mclk_multiplier = "6.18";
                pix_clk_hz = "148500000";

				gain_factor = "10";
				min_gain_val = "10";/* 1DB*/
				max_gain_val = "160";/* 16DB*/
				step_gain_val = "1";
				default_gain = "10";
				min_hdr_ratio = "1";
				max_hdr_ratio = "1";
				framerate_factor = "1000000";
				min_framerate = "1816577";/*1.816577 */
				max_framerate = "30000000";/*30fps*/
				step_framerate = "1";
				default_framerate = "30000000";/*30fps*/
				exposure_factor = "1000000";
				min_exp_time = "34";/* us */
				max_exp_time = "550385";/* us */
				step_exp_time = "1";
				default_exp_time = "33334";/* us */
				embedded_metadata_height = "0";
			};

Best Regards,
Khang

5

hello khang.l4es,

may I double confirm your Jetpack release version, and your capture pipeline.

let’s also disable preview and shows frame-rate only for quick checking.
for instance,
$ gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=300 ! 'video/x-raw, width=1920, height=1080, framerate=30/1, format=YUY2' ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

6

Hi @JerryChang,

We use Jetpack-5.0.2. Using your gst command gave following message (no frame-rate show):

$ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! 'video/x-raw, width=1920, height=1080, framerate=30/1, format=UYVY' ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
New clock: GstSystemClock
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0.GstGhostPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, colorimetry=(string)2:4:7:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false

below trace :

trace2a.txt (4.7 MB)

7

hello khang.l4es,

it’s now shown a pair of CHANSEL_PXL_SOF/CHANSEL_PXL_EOF, which looks promising.
it’s ATOMP_FRAME_DONE to indicate a frame has write to memory,
however, the frame index doesn’t look correct here although each memory write with different VI timestamps.
could you please check from bridge driver side, please check whether there’re frame-index settings.
for instance,

tag:ATOMP_FRAME_DONE channel:0x23 frame:1 vi_tstamp:272024368000 data:0x0000000000000000
tag:ATOMP_FRAME_DONE channel:0x23 frame:1 vi_tstamp:272057734528 data:0x0000000000000000
tag:ATOMP_FRAME_DONE channel:0x23 frame:1 vi_tstamp:272091101056 data:0x0000000000000000
8

Hi @JerryChang,

The fpga designer asked :

Frame number is not mandatory in the CSI2 Specifications.
I’m checking with the IP provider if it is possible to enable it.
is it possible to disable in the nvidia system?

Do you think if it is possible to disable the frame number in the nvidia system?

Best Regards,
Khang

9

hello khang.l4es,

as you can see,
VI driver only added it incrementally for sending to user-space.

static void vi5_release_buffer(...)
{
...
        vbuf->sequence = chan->sequence++;
        vb2_buffer_done(&vbuf->vb2_buf, buf->vb2_state);
10

Hi @JerryChang,

Thanks for your reply but I couldn’t capture your idea. Could you elaborate more, please ?

11

hello khang.l4es,

I meant this is not necessary.

please add some debug logs to VI driver, you should see vi5_release_buffer() being called continuously.

12

Hi @JerryChang,

I meant this is not necessary.

I am confused: You said that the fpga side should check the frame indexing (numbering). The fpga designer asked back if we could disable the frame numbering on the jetson side. You said again that this was not necessary.

Can you confirm that the frame indexing/numbering on the fpga side (which is the current situation) is a MUST for the drivers of the Nvidia Jetson pipeline’s elements (NVCSI, VI) ?

Best regards,
Khang

13

hello khang.l4es,

ya, that frame index in the tracing logs could be ignored, (let me revise my comments)
I’ve checked with functional camera with tracing logs enabled, it works normally with consistent frame-index.

14

Hi @JerryChang,

What did you mean by consistent frame-index? Does it mean that it increases linearly (step of 1)? In my case, it stays 1 as you pointed out.

Also according to your revise, it seems that the frame-indexing on the fpga side is not necessary. If this is true, what could be the potential issue that cause us to receive just one frame ?

Regards,
Khang

15

Hi @JerryChang,

We are really being stuck and we need to provide more info to the FPGA designer to unblock the situation. Do you know if we can contact any of your partner that could help to debug the Nvidia Jetson side more deeply, please?

Best Regards,
Khang

16

hello khang.l4es,

it’s all report frame:0 with my functional camera.
let me attach tracing logs for your comparison, i.e. good_tracing.txt (256.1 KB)


I’ve re-visit your tracing logs in 1st comment.
it looks there’re only 4 ATOMP_FRAME_DONE has reported, such 4 is according to default buffer queue, it’s queue buffer allocation for 4-buffers, due to queue depth, VI driver side needs enqueue 4-times for sending 1st frame to user-space.
for instance, $public_sources/kernel_src/kernel/nvidia/drivers/media/platform/tegra/camera/vi/channel.c

int tegra_channel_alloc_buffer_queue(...)
{
...
	chan->capture_queue_depth = num_buffers;

so…
it’s more like an issue that FPGA did not output frames.
that’s why you see only one frame has captured.

1 Like
17

Hi @JerryChang,

The FPGA designer confirmed that the fpga sends frames constantly. However, re-visiting the logs I shared, I did not see the FE tags, except for the 4 ATOMP_FE belonging to the captured frame. Do you think that it could be an issue? I asked him to check the FE, but he needs to know how many rows and lines are being received :

I can check about the FE, what i need to know is how many rows and lines are being received an what is the Driver expecting!.

Is it possible to know the number of received rows and lines?

Best Regards,
Khang

18

hello khang.l4es,

it’s able to check via CHANSEL_PXL_EOF.
for instance,
kworker/2:2-87 [002] .... 3550.834613: rtcpu_vinotify_event: tstamp:111678482672 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:1 vi_tstamp:3573709942208 data:0x0000000004370002

it’s 0x437 = 1079-lines (start from 0) has received by VI engine.

1 Like
19

Hi @JerryChang,

Thanks for your finding. So it’s 1-line LESS than expected (1080). Similar issue Xavier_NX FS_FAULT error with lattice crosslink - #16 by alok.pawar ?

20

hello khang.l4es,

0x437 should be correct, it’s 0-1079, total 1080 lines for sending to VI engine.

IIRC, there’s a bug that set_mode_delay_ms did not applied correctly.
let’s have an alternative way to test with v4l2 IOCTL, you should also extend the CAPTURE_TIMEOUT_MS from VI driver to avoid capture timeout failure.
for instance,
$public_sources/kernel_src/kernel/nvidia/drivers/media/platform/tegra/camera/vi/vi5_fops.c
#define CAPTURE_TIMEOUT_MS 2500

here’s v4l pipeline for your reference,
$ v4l2-ctl -d /dev/video0 --list-formats-ext
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300

finger-crossed.

21

Hi @JerryChang,

0x437 should be correct, it’s 0-1079, total 1080 lines for sending to VI engine.

While I am still unable receive more than one frame, and do not see any FE tag in the trace, I reported that 1079 is less than expected 1080 however.

In my case, there’s already above definition.

I tried the v4l2 pipeline that you suggested :

nvidia@ubuntu:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'UYVY' (UYVY 4:2:2)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
	[1]: 'NV16' (Y/CbCr 4:2:2)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
	[2]: 'UYVY' (UYVY 4:2:2)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
nvidia@ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=UYVY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300

NO frame-rate progress (>>>) observed, unfortunately.

The trace :
trace_v4l2.txt (600.0 KB)