Problem with embedded data

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

We are using SONY sensor which gives data in RAW12 format and each frame starts with 512 bytes of embedded data. This mipi data obtained from sensor is converted to parallel data using mipi to parallel converter bridge, which is again converted to mipi data by another bridge. We have given the data type ID for the output mipi data coming from parallel to mipi bridge as RAW12 (0x2C - as in MIPI CSI2 spec. document). So, the packet header of all the data including embedded data comes with ID of 0x2C (RAW12 ID) instead of 0x12 (Embedded metadata ID). We don’t have complete control over the parallel to mipi toshiba bridge to differentaite between the embedded metadata and image data. So, we were unable to give embedded metadata its datatype ID.

When using TX1 L4T-28.2, we were able to grab the frame using v4l2-ctl even if the kernel throws MW_ACK_DONE exception. The grabbed frame has 512 bytes of embedded data in the first line and the remaining length is appended with zeros. From the next line,we could get normal image data and were also able to stream using libargus application.

This is the exception thrown when trying to capture using v4l2-ctl in tx1.

[  387.970236] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.102199] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.238037] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.238038] video4linux video0: frame start syncpt timeout!0
[  388.370188] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.506017] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.638075] video4linux video0: frame start syncpt timeout!0
[  388.638087] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.738069] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  388.869940] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.002049] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.138109] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.144260] video4linux video0: frame start syncpt timeout!0
[  389.237881] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.371302] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.502009] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.638202] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.770054] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  389.902046] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.042460] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.170044] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.302440] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.438224] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.570048] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.702128] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.838028] video4linux video0: MW_ACK_DONE syncpoint time out!0
[  390.970013] video4linux video0: frame start syncpt timeout!0

But, when doing the same in TX2 L4T-28.2.1, we were unable to grab the frame and the kernel throws PXL_SOF syncpt error and occasionally ATOM_FE syncpt error.

[  128.194394] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  128.200801] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  129.326403] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  129.332761] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  129.398453] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000008
[  129.406788] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000008
[  129.414976] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000044
[  129.423317] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000044
[  130.494409] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  130.500776] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  130.566461] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000004
[  130.574450] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000004
[  130.582645] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000044
[  130.590622] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000044
[  131.658412] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  131.664781] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  131.730476] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000004
[  131.738334] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000004
[  131.747136] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000044
[  131.755402] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000044
[  132.826416] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[  132.832858] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
[  132.898439] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00000004
[  132.906312] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00000004
[  132.915066] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_INTR_STATUS 0x00000044
[  132.923536] nvcsi 150c0000.nvcsi: csi4_cil_check_status (0) CIL_ERR_INTR_STATUS 0x00000044
[  133.986416] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout!

As far as our understanding goes, when the line length received in the frame data is less than the specified one

  • TX1 throws a warning message and appends zeros to the remaining length of the line and gives the data to the upper layer.
  • TX2 throws exception and discards the frame data in the vi layer itself.

Is there any way to append zeroes to the first line as in TX1 or to discard only the first line of the frame which has line length less than the prescribed one?

NOTE: Embedded data coming from the sensor cannot be disabled.

2

hello vimalan.93,

please download the L4T documentation,
you may refer to [NVIDIA Tegra Linux Driver Package]-> [Release 28.2 Development Guide]-> [Sensor Driver Programming Guide]
please also refer to [url]https://elinux.org/Jetson_TX2/28.1_Camera_BringUp[/url] for some debug tips.
thanks

3

Hi JerryChang,

Thanks for the quick response,

This is the trace message we got while trying to capture the frames using v4l2-ctl

kworker/5:2-231   [005] ...1    64.110792: rtos_queue_peek_from_isr_failed: tstamp:2356138501 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    64.266801: rtos_queue_peek_from_isr_failed: tstamp:2361139030 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    64.422771: rtos_queue_peek_from_isr_failed: tstamp:2366139509 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    64.474790: rtcpu_vinotify_handle_msg: tstamp:2367626793 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2367626245 data:0x00000001
     kworker/5:2-231   [005] ...1    64.474794: rtcpu_vinotify_handle_msg: tstamp:2367626950 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2367626256 data:0x00000000
     kworker/5:2-231   [005] ...1    64.474795: rtcpu_vinotify_handle_msg: tstamp:2367628778 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2367628367 data:0x08000000
     kworker/5:2-231   [005] ...1    64.526807: rtcpu_vinotify_handle_msg: tstamp:2368625812 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2368625188 data:0x04370002
     kworker/5:2-231   [005] ...1    64.526811: rtcpu_vinotify_handle_msg: tstamp:2368625922 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:2368625318 data:0x04380040
     kworker/5:2-231   [005] ...1    64.526813: rtcpu_vinotify_handle_msg: tstamp:2368626726 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2368626067 data:0x08000000
     kworker/5:2-231   [005] ...1    64.526815: rtcpu_vinotify_handle_msg: tstamp:2368626824 tag:CHANSEL_FAULT_FE channel:0x01 frame:0 vi_tstamp:2368626069 data:0x00000001
     kworker/5:2-231   [005] ...1    64.526817: rtcpu_vinotify_handle_msg: tstamp:2368626953 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2368626072 data:0x00000000
     kworker/5:2-231   [005] ...1    64.578822: rtos_queue_peek_from_isr_failed: tstamp:2371140045 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    64.786841: rtos_queue_peek_from_isr_failed: tstamp:2376140750 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    64.942828: rtos_queue_peek_from_isr_failed: tstamp:2381141258 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.098961: rtos_queue_peek_from_isr_failed: tstamp:2386141763 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.254834: rtos_queue_peek_from_isr_failed: tstamp:2391142271 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.410845: rtos_queue_peek_from_isr_failed: tstamp:2396142779 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.566819: rtos_queue_peek_from_isr_failed: tstamp:2401057448 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.566826: rtos_queue_peek_from_isr_failed: tstamp:2401150836 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.566828: rtcpu_start: tstamp:2401151974
     kworker/5:2-231   [005] ...1    65.566831: rtcpu_vinotify_handle_msg: tstamp:2401971029 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:2401970605 data:0x00800060
     kworker/5:2-231   [005] ...1    65.618836: rtcpu_vinotify_handle_msg: tstamp:2403043416 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2403042872 data:0x00000001
     kworker/5:2-231   [005] ...1    65.618841: rtcpu_vinotify_handle_msg: tstamp:2403043571 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2403042883 data:0x00000000
     kworker/5:2-231   [005] ...1    65.618843: rtcpu_vinotify_handle_msg: tstamp:2403047111 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2403046703 data:0x08000000
     kworker/5:2-231   [005] ...1    65.670844: rtcpu_vinotify_handle_msg: tstamp:2404039660 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2404039037 data:0x04370002
     kworker/5:2-231   [005] ...1    65.670848: rtcpu_vinotify_handle_msg: tstamp:2404039765 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:2404039167 data:0x04380040
     kworker/5:2-231   [005] ...1    65.670850: rtcpu_vinotify_handle_msg: tstamp:2404040648 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2404039895 data:0x08000000
     kworker/5:2-231   [005] ...1    65.670853: rtcpu_vinotify_handle_msg: tstamp:2404040746 tag:CHANSEL_FAULT_FE channel:0x01 frame:0 vi_tstamp:2404039896 data:0x00000001
     kworker/5:2-231   [005] ...1    65.670855: rtcpu_vinotify_handle_msg: tstamp:2404040876 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2404039899 data:0x00000000
     kworker/5:2-231   [005] ...1    65.722845: rtos_queue_peek_from_isr_failed: tstamp:2406151751 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    65.878849: rtos_queue_peek_from_isr_failed: tstamp:2411152458 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.034832: rtos_queue_peek_from_isr_failed: tstamp:2416152967 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.190832: rtos_queue_peek_from_isr_failed: tstamp:2421153470 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.346811: rtos_queue_peek_from_isr_failed: tstamp:2426153787 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.502808: rtos_queue_peek_from_isr_failed: tstamp:2431154296 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.662831: rtos_queue_peek_from_isr_failed: tstamp:2436155059 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.714809: rtos_queue_peek_from_isr_failed: tstamp:2436430873 queue:0x0b4a3c58
     kworker/5:2-231   [005] ...1    66.714815: rtos_queue_peek_from_isr_failed: tstamp:2437481570 queue:0x0b4a3c58

In normal case, without the toshiba bridges’ setup we were able to see the streaming in argus application and could save the raw frames using v4l2-ctl without any errors.

We have referred the Driver Programming Guide and couldn’t find any ways to discard the first line of the frame.

Could you please help us out on this problem! It could be really helpful for us.

Thanks in advance
Vimal.

4

hello vimalan.93,

you got a CHANSEL_FAULT at frame-end, it reports pixel run away failure.
could you please check the line_length in your sensor device tree, according to the logs, your line_number is not identical, 0x0437 vs 0x0438.
you may also refer to [Sensor Driver Programming Guide] for device tree properties configurations.
thanks