Xavier NX MIPI CSI-2 from FPGA

[d.glass]

I’m trying to receive CSI-2 data without I2C from an FPGA on a Xavier NX with L4T 32.7.1. I have followed several of the earlier topics on the subject, but I’m still having difficulty.
I’m using an IMX219 camera as a reference, and trying to duplicate the IMX219 format/timing with the FPGA (Xilinx MIPI CSI-2 Tx Subsystem on Ultrascale+ device). I have modified the device tree and imx219 driver files, and can see the /dev/videoN nodes as expected. I can also capture data on those nodes from an IMX219 camera, modified so that it’s power and I2C are controlled by a separate system. I’ve also set up another FPGA with a MIPI CSI-2 receiver and used that to verify that the IMX219 and FPGA -generated CSI-2 streams agree (start-of-frame, data type, line timing, number of pixels, number of lines, end-of-frame, etc.) The primary difference between the FPGA stream and the IMX219 is the presence of two lines of non-image data (datatype 0x12) preceeding each frame from the IMX219 - I don’t have those in place on the FGPA.
I’m using v4l2-ctl to try to capture the data on the Xavier NX. It works fine for the IMX219 camera, but I get nothing from the FPGA data.
When I look at the debug logs, I don’t see any indicators of activity from the NVCSI/VI when connected to the FPGA. Here is a log from the IMX219, then a log from the FPGA.
tracelog_imx219_1_camtrace_newt.log (623.9 KB)

With the FPGA:
tracelog_fpga_0_camtrace_newt.log (25.4 KB)

Again, I’ve verified that the MIPI CSI-2 streams appear very similar with another FPGA-based CSI-2 D-PHY receiver.
Also, I’ve been trying to access the NVCSI/VI registers from userspace using devmem2 and busybox devmem. With either of them , if I attempt to read from the NVCSI (0x15a00000), the Xavier crashes as follows:
crashlog_console.txt (5.7 KB)

Any help you can offer is greatly appreciated.
-Dan

[Trumany]

Hi, can you show the connection difference between FPGA to Xavier NX and IMX219 to Xavier NX that you have tested? We’d better to make sure if the connection is correct first.

[d.glass]

Please see the attached image. All of the connections are using 15-pin CSI cables, and the Xavier NX is on a seeed studio A205 carrier. The FPGA source is the AXU2CGA on the right, with two CSI interfaces, which you can see at the top. The CSI ports are duplicates, with identical data on both. The Zybo Z7 on the upper right is programmed with a Xilinx MIPI CSI-2 Rx Subsystem and ILA - the CSI connection is via the built-in 15-pin CSI connector. I have used that to verify that the two FPGA ports are identical, and to compare the FPGA signal with the IMX219. Note that the IMX219 with independent power/I2C is not shown connected in the picture, but I have verified it with the Zybo, and also verified that all 6 of the A205 CSI ports work with the IMX219. I’ve tried the FPGA signals on all of the A205 CSI ports, and none work. Remember that the FPGA signals are received just as expected on the Zybo: Frame starts, line length/timing, expected data, frame end, etc. compare with the IMX219. I have a 2GHz scope coming tomorrow for a better look at the signals electrically, but I can clearly see the CSI signal envelopes with a 200MHz scope, and IMX219 and FPGA match levels/excursions/timing pretty well (measured at A206 CSI connectors).

[Trumany]

@d.glass

The primary difference between the FPGA stream and the IMX219 is the presence of two lines of non-image data (datatype 0x12) preceeding each frame from the IMX219 - I don’t have those in place on the FGPA.

Can you share detail info of the difference? Which two lines of CSI port do you mean? Do you have the schematic of that?

[d.glass]

The schematics connections are identical. By lines, I mean the video horizontal sync period. The attached image shows the start of an IMX219 frame on the Zybo, with two video line periods of non-image data (type 0x12), followed by a gap, then the start of the first line of live video data (type 0x2B). The CSI data type is decoded on video_out_0_tdest[9:4]. The “noni” signals are non-image port on the CSI receiver, while the “video” signals are the image port.
I’ve tried sending both 0x12 (non-image) and 0x2B (RAW10) to the Xavier NX from the FPGA, with start of frame located as is it from the IMX219, but I don’t ever see an FS (Frame Start) indicated in the debug logs. I do see the FS for the IMX219.
The Xavier NX debug logs come out the same if the FPGA is connected or nothing is connected - but the Zybo-based CSI receiver receives and decodes the FPGA signals as expected.

[ShaneCCC]

The trace log from FPGA tell NVCSI/VI didn’t receive any validate package from MIPI bus.
Maybe need to probe the MIPI signal from FPGA to confirm it.

[d.glass]

Hi Shane -
I measured the signals at the Xavier Nx connector, comparing an IMX219 on CSI1 with the FPGA on CSI3. The image below shows where I was probing. Note that the log files from earlier are for the FPGA on CSI0 - I can’t probe those directly on the Xavier NX, as they’re on the underside, but I’m sending identical CSI signals from the FPGA to both CSI0 and CSI3, and I get the same log results for both.

With the oscilloscope, the primary difference I see between the FPGA and the IMX219 is that the clock lane stops during the vertical blanking interval for the IMX219, but it is continuous for the FPGA, shown below.

Other than that, the offsets are very similar, and the FPGA magnitude is a little higher. See below, IMX219 first, then FPGA:

The data lane also compare well. Here is a comparison of the D0 lane, single-ended measurement, with the IMX219 as the reference waveform and the FPGA on channel 1. The offset is 2 divisions for visibility.

Here’s a view over several hsync periods, IMX219 followed by FPGA:

The apparent structure on the FPGA data lane is consistent with the incrementing-pixel-value test pattern it is sending.
When I look at the CSI traffic from the FPGA using the Zybo Z7 (another, independent FPGA development board set up as a CSI receiver), the Frame Starts appear as expected (see image below), and the 32-bit RAW10 data stream decodes to the expected incrementing pixel pattern (0x001, 0x002, 0x003, 0x004, …). The datatype, pixels per line, bytes per line, line-end, frame end, etc. are all also as expected.

I’m currently putting together a D-PHY receiver to collect the traffic byte-by-byte, but things are getting pretty far in to the weeds. Any help you can offer is appreciated.

[d.glass]

I’ve updated the device tree so that mode0 on the video0 devnode has:
discontinuous_clk = “no”;
and now I see much more activity in the trace logs, but I still can’t capture the video data.
Here are the trace logs, first for a successful attempt with the IMX219 on /dev/video1.
The command generating the trace logs is:
%v4l2-ctl -d /dev/videoN --set-ctrl bypass_mode=0 --stream-mmap --stream-count=3
tracelog_imx219_noncontinuous_clk.log (29.5 KB)

And this is the tracelog for the failed attempt with the FPGA on /dev/video0.
tracelog_fpga_continuous_clk.log (145.1 KB)

This is device tree node:
dev_video0_node.txt (1.4 KB)

I’m wondering if A) the IMX219’s two lines of embedded non-image data at the top of each frame are necessary, and B) if each line of image data needs line start/end packets. Are there settings for these things that can be changed in the device tree or elsewhere?

[Trumany]

It’s hard to understand if both signals are same. There must be some difference between them. From HW side, have you checked with the carrier board vendor for this? Have you tried to connect FPGA port to the same CSI port of Xavier NX that IMX219 tested? Is there any design difference on FPGA board on CSI port? Can you share the schematic of CSI part of FPGA board and A205 carrier board?

[ShaneCCC]

Remove the “embedded_metadata_height” in device tree for FPGA didn’t send the 0x12 data type.

[d.glass]

Thank you, to @Trumany and @ShaneCCC - you’ve both been very helpful in solving the problems.
To summarize the solutions:

1. The mode0 description in the device tree needed
   discontinuous_clk=“no”
   rather than
   discontinuous_clk=“yes”
   The FPGA signals have a continuous clk, and without this change, the trace logs showed no difference between a CSI port with active signals and an unconnected port.
2. Removal of the embedded_metadata_height line from the mode in the device tree, as the FPGA was not including embedded non-image data (dataype 0x12).