Capturing from IMX568 via V4L2

We are trying to connect a IMX568 (Sony 5MP global shutter CSI-2 sensor) to a Xavier NX.

Our driver works fine using the ISP, so the sensor is configured correctly and sending the images as expected.

However:
When trying to capture raw data via V4L2, all images are discarded.

dmesg output:

[16054.614190] tegra194-vi5 15c10000.vi: corr_err: discarding frame 1, flags: 128, err_data 262144
[16054.638058] tegra194-vi5 15c10000.vi: corr_err: discarding frame 2, flags: 128, err_data 262144
[16054.661952] tegra194-vi5 15c10000.vi: corr_err: discarding frame 3, flags: 128, err_data 262144
[16054.685768] tegra194-vi5 15c10000.vi: corr_err: discarding frame 4, flags: 128, err_data 262144
[16054.709628] tegra194-vi5 15c10000.vi: corr_err: discarding frame 5, flags: 128, err_data 262144
[16054.733503] tegra194-vi5 15c10000.vi: corr_err: discarding frame 6, flags: 128, err_data 262144
[16054.757375] tegra194-vi5 15c10000.vi: corr_err: discarding frame 7, flags: 128, err_data 262144
[16054.781209] tegra194-vi5 15c10000.vi: corr_err: discarding frame 8, flags: 128, err_data 262144
[16054.805068] tegra194-vi5 15c10000.vi: corr_err: discarding frame 9, flags: 128, err_data 262144
[16054.828932] tegra194-vi5 15c10000.vi: corr_err: discarding frame 10, flags: 128, err_data 262144

(continuing like this forever)

Capture trace looks like this for every frame:

 kworker/0:0-11979 [000] .... 16054.691473: rtcpu_vinotify_error: tstamp:502123634822 tag:CHANSEL_NOMATCH channel:0x04 frame:3 vi_tstamp:502123631186 data:0x00000269
 kworker/0:0-11979 [000] .... 16054.691476: rtcpu_vinotify_event: tstamp:502124113087 tag:ATOMP_FRAME_DONE channel:0x23 frame:2 vi_tstamp:502123617168 data:0x00000000
 kworker/0:0-11979 [000] .... 16054.691478: rtcpu_vinotify_event: tstamp:502124113231 tag:RESERVED_19 channel:0x23 frame:2 vi_tstamp:502994331616 data:0x02020002
 kworker/0:0-11979 [000] .... 16054.691479: rtcpu_vinotify_event: tstamp:502124113395 tag:RESERVED_19 channel:0x23 frame:0 vi_tstamp:502994335456 data:0x07020003
 kworker/0:0-11979 [000] .... 16054.691480: rtcpu_vinotify_event: tstamp:502124113532 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:502994573728 data:0x10000000
 kworker/0:0-11979 [000] .... 16054.691481: rtcpu_vinotify_event: tstamp:502124113690 tag:RESERVED_18 channel:0x23 frame:0 vi_tstamp:502994603424 data:0x31000004
 kworker/0:0-11979 [000] .... 16054.691483: rtcpu_vinotify_event: tstamp:502124113826 tag:FS channel:0x00 frame:3 vi_tstamp:502123630472 data:0x00000012
 kworker/0:0-11979 [000] .... 16054.691511: rtcpu_vinotify_event: tstamp:502124113988 tag:ATOMP_FS channel:0x00 frame:3 vi_tstamp:502123630475 data:0x00000000
 kworker/0:0-11979 [000] .... 16054.691512: rtcpu_vinotify_event: tstamp:502124114126 tag:CHANSEL_EMBED_SOF channel:0x23 frame:3 vi_tstamp:502123630844 data:0x00000004
 kworker/0:0-11979 [000] .... 16054.691514: rtcpu_vinotify_event: tstamp:502124114281 tag:CHANSEL_NOMATCH channel:0x04 frame:3 vi_tstamp:502123631186 data:0x00000269
 kworker/0:0-11979 [000] .... 16054.691515: rtcpu_vinotify_event: tstamp:502124114419 tag:CHANSEL_PXL_SOF channel:0x23 frame:3 vi_tstamp:502123654104 data:0x00000001
 kworker/0:0-11979 [000] .... 16054.691516: rtcpu_vinotify_event: tstamp:502124114577 tag:RESERVED_19 channel:0x23 frame:3 vi_tstamp:502995454912 data:0x08020003
 kworker/0:0-11979 [000] .... 16054.691517: rtcpu_vinotify_event: tstamp:502124368148 tag:CHANSEL_PXL_EOF channel:0x23 frame:3 vi_tstamp:502124362833 data:0x080f0002
 kworker/0:0-11979 [000] .... 16054.691518: rtcpu_vinotify_event: tstamp:502124368305 tag:FE channel:0x00 frame:3 vi_tstamp:502124362845 data:0x00000022
 kworker/0:0-11979 [000] .... 16054.691520: rtcpu_vinotify_event: tstamp:502124368445 tag:CHANSEL_SHORT_FRAME channel:0x04 frame:3 vi_tstamp:502124362846 data:0x00000000
 kworker/0:0-11979 [000] .... 16054.691521: rtcpu_vinotify_event: tstamp:502124368606 tag:RESERVED_19 channel:0x23 frame:3 vi_tstamp:503018153152 data:0x01020003
 kworker/0:0-11979 [000] .... 16054.691522: rtcpu_vinotify_event: tstamp:502124368794 tag:ATOMP_FE channel:0x00 frame:3 vi_tstamp:502124362849 data:0x00000000
 kworker/0:0-11979 [000] .... 16054.691523: rtcpu_vinotify_error: tstamp:502124379686 tag:CHANSEL_NOMATCH channel:0x04 frame:4 vi_tstamp:502124376874 data:0x00000269

Looking at the CHANSEL_NOMATCH, is looks like it complains about a packet with header data type 0x13.

We contacted the sensor’s vendor (Sony) and found out that it is indeed sending packets (some embedded data lines) with header data type 0x13. Unfortunately they say it is impossible to change/disable this.

Two questions arise from this:
(1) Is the packet with the code 0x13 the reason for the frames being discarded? Is there a way to read from the logs to find out what exactly the reason is?
(2) If the data type 0x13 is indeed the reason for the packets being dropped: Is it possible to configure the Xavier NX in a way that it does not act as pedantic as it is in V4L2 mode? Since streaming through the ISP works fine, we know that the image data itself is OK.

Since the IMX568 is a global shutter sensor primarily meant for industrial applications, it would be very useful to to access the raw data from the sensor.

Add embedded_metadata_height to your device tree form embedded data.

https://elinux.org/Jetson/l4t/Camera_BringUp

Hi Shane,

I had it set to 4, since that was the number of supposed embedded data lines.

But that is apparently wrong; it has to be the number of 0x12 lines.

The 0x13 traces seem to be non-critical, only make the log harder to read!

While we are at that topic of embedded data: Is there any way for an application to actually receive the embedded data that was sent with an image?

Thank you!

You can use ISensorPrivateMetadata to get it by argus API.

https://docs.nvidia.com/jetson/l4t-multimedia/classArgus_1_1Ext_1_1ISensorPrivateMetadata.html

Any way to get that when streaming via V4L2?

Need modify the VI kernel driver to send it to user space.