Metadata Capture - TX2

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi,

I developed a V4l2 driver (L4T32.2) that creates 3 video devices and I’m able to capturing images from video0 (RAW10) and video1 (RAW8) using v4l2-ctl.

The third video device was created in order to capture Metadata ONLY. For this case, the data is coming with 0x30 Data Type (User-defined format), but this data type is changed to 0x2A in order to trick the TX2 to receive the data as RAW8 pixel format.

The data coming with DT=0x30 goes through a GSML stage, where is serialized and then, deserialized. The deserializer changes the DT to 0x2A and the TX2 should be have to capture RAW8 data, however it’s not working.

We verified that GSML stage it’s working correctly (serializer and deserializer are receiving and redirecting the data), so the problem is in when the TX2 capture.

The metadata stream is coming in 1024x4 resolution, but we also try configuring camera_common formats to 4096x1 and 2048x2, below are some logs from capture tests:

1) 1024x4:

v4l2-ctl -d /dev/video2 --set-fmt-video=width=1024,height=4 --set-ctrl bypass_mode=0 --stream-mmap

kworker/0:2-3127  [000] ....   128.735339: rtcpu_vinotify_event: tstamp:4153392913 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:4153392532 data:0x00010000
     kworker/0:2-3127  [000] ....   128.735346: rtcpu_vinotify_event: tstamp:4153738400 tag:CSIMUX_STREAM channel:0xff frame:512 vi_tstamp:4153738022 data:0x00010000
     kworker/0:2-3127  [000] ....   128.735348: rtcpu_vinotify_event: tstamp:4153739406 tag:CSIMUX_STREAM channel:0xff frame:256 vi_tstamp:4153739044 data:0x00010000
     kworker/0:2-3127  [000] ....   128.847200: rtos_queue_peek_from_isr_failed: tstamp:4157600321 queue:0x0b4b4500
     kworker/0:2-3127  [000] ....   129.014823: rtos_queue_peek_from_isr_failed: tstamp:4162600331 queue:0x0b4b4500
     kworker/0:2-3127  [000] ....   129.182513: rtos_queue_peek_from_isr_failed: tstamp:4167600336 queue:0x0b4b4500
     kworker/0:2-3127  [000] ....   129.350172: rtos_queue_peek_from_isr_failed: tstamp:4172600342 queue:0x0b4b4500
     kworker/0:2-3127  [000] ....   129.517810: rtos_queue_peek_from_isr_failed: tstamp:4177600349 queue:0x0b4b4500
     kworker/0:2-3127  [000] ....   129.685548: rtos_queue_peek_from_isr_failed: tstamp:4182600355 queue:0x0b4b4500
     kworker/0:2-3127  [000] ....   129.797246: rtcpu_vinotify_event: tstamp:4187100300 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:4187099351 data:0x00000001
     kworker/0:2-3127  [000] ....   129.797256: rtcpu_vinotify_event: tstamp:4187100699 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:4187099356 data:0x00000000
     kworker/0:2-3127  [000] ....   129.797259: rtcpu_vinotify_event: tstamp:4187100979 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:4187099368 data:0x00000200
     kworker/0:2-3127  [000] ....   129.797264: rtcpu_vinotify_event: tstamp:4187101652 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:4187099917 data:0x00030202
     kworker/0:2-3127  [000] ....   129.797266: rtcpu_vinotify_event: tstamp:4187101936 tag:CHANSEL_LOAD_FRAMED channel:0x90 frame:0 vi_tstamp:4187101192 data:0x08000000
     kworker/0:2-3127  [000] ....   129.797269: rtcpu_vinotify_event: tstamp:4187106746 tag:CHANSEL_LOAD_FRAMED channel:0x90 frame:0 vi_tstamp:4187106293 data:0x08000000
     kworker/0:2-3127  [000] ....   129.797272: rtcpu_vinotify_event: tstamp:4187401544 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:4187400981 data:0x00000000

Results:

CHANSEL_FAULT: data:0x00000202 → PIXEL_SHORT_LINE, PIXEL_EOF
CHANSEL_FAULT: data:0x00030202 → LINE_NUMBER = 3, PIXEL_SHORT_LINE, PIXEL_EOF

2) 4096x1:

v4l2-ctl -d /dev/video2 --set-fmt-video=width=4096,height=1 --set-ctrl bypass_mode=0 --stream-mmap

kworker/0:1-735   [000] ....   337.870367: rtcpu_vinotify_event: tstamp:10688390989 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:10688390175 data:0x00000001
     kworker/0:1-735   [000] ....   337.870373: rtcpu_vinotify_event: tstamp:10688391566 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:10688390180 data:0x00000000
     kworker/0:1-735   [000] ....   337.870376: rtcpu_vinotify_event: tstamp:10688391851 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:10688390193 data:0x00000202
     kworker/0:1-735   [000] ....   337.870378: rtcpu_vinotify_event: tstamp:10688392564 tag:CSIMUX_STREAM channel:0xff frame:256 vi_tstamp:10688391200 data:0x00010000
     kworker/0:1-735   [000] ....   337.870381: rtcpu_vinotify_event: tstamp:10688392954 tag:CHANSEL_LOAD_FRAMED channel:0x90 frame:0 vi_tstamp:10688392065 data:0x08000000
     kworker/0:1-735   [000] ....   337.870383: rtcpu_vinotify_event: tstamp:10688397003 tag:CHANSEL_LOAD_FRAMED channel:0x90 frame:0 vi_tstamp:10688396553 data:0x08000000
     kworker/0:1-735   [000] ....   337.870386: rtcpu_vinotify_event: tstamp:10688692362 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:10688691798 data:0x00000000

Results:

CHANSEL_FAULT: data:0x00000202 → PIXEL_SHORT_LINE, PIXEL_EOF

3) 2048x2:

v4l2-ctl -d /dev/video2 --set-fmt-video=width=2048,height=2 --set-ctrl bypass_mode=0 --stream-mmap

kworker/0:1-735   [000] ....   447.025919: rtcpu_vinotify_event: tstamp:14099594474 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:14099593521 data:0x00000001
     kworker/0:1-735   [000] ....   447.025927: rtcpu_vinotify_event: tstamp:14099595080 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:14099593526 data:0x00000000
     kworker/0:1-735   [000] ....   447.025930: rtcpu_vinotify_event: tstamp:14099595363 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:14099593538 data:0x00000200
     kworker/0:1-735   [000] ....   447.025934: rtcpu_vinotify_event: tstamp:14099595982 tag:CHANSEL_FAULT channel:0x00 frame:0 vi_tstamp:14099593721 data:0x00010202
     kworker/0:1-735   [000] ....   447.025937: rtcpu_vinotify_event: tstamp:14099596269 tag:CSIMUX_STREAM channel:0xff frame:256 vi_tstamp:14099594560 data:0x00010000
     kworker/0:1-735   [000] ....   447.025941: rtcpu_vinotify_event: tstamp:14099596720 tag:CHANSEL_LOAD_FRAMED channel:0x90 frame:0 vi_tstamp:14099595578 data:0x08000000
     kworker/0:1-735   [000] ....   447.025943: rtcpu_vinotify_event: tstamp:14099600473 tag:CHANSEL_LOAD_FRAMED channel:0x90 frame:0 vi_tstamp:14099600020 data:0x08000000
     kworker/0:1-735   [000] ....   447.025946: rtcpu_vinotify_event: tstamp:14099895720 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:14099895157 data:0x00000000

Results:

CHANSEL_FAULT: data:0x00000202 → PIXEL_SHORT_LINE, PIXEL_EOF
CHANSEL_FAULT: data:0x00010202 → LINE_NUMBER = 1, PIXEL_SHORT_LINE, PIXEL_EOF

Questions
1) What is the meaning of these errors? What does LINE_NUMBER notification mean?
2) I wonder if the Metadata can be threaded as RAW8 data just by changing the DT to 0x2A?
3) Is there a way to capture Metadata as DT=0x30?
4) What else could be causing this issue?

Regards,
-Enrique

2

hello EnriqueR,

our reference drivers working with VI engine to parse metadata, which usually in the beginning of the sensor streaming.
hence, you’ll need to configure embedded_metadata_height property in device tree for that.
you may also refer to Tegra X2 (Parker Series SoC) Technical Reference Manual,
please check [Chapter-27 Video input (VI)], and you might search “embedded data datatype” for more detail descriptions.

may I know is your 3rd device only produce metadata?
could you please also share some details of the metadata? is it only 1-line?
thanks

3

Hi JerryChang,

Yes, we need to capture metadata only from the 3rd device. That device only produce the metadata.

We are expecting 4K of metadata in 1024x4 format, so it should be 4-lines.

Thanks,
-Enrique

4

hello EnriqueR,

we had change add for l4t-r32.2 to read embedded_metadata_width for some specific sensors.
could you please have a try to update device tree as below,

embedded_metadata_width = "1024";
embedded_metadata_height = "4";

also, those v4l2 commands of --set-fmt-video=width=X,height=Y specify the effective pixels.
please try to exclude these configuration since your sensor only produce the metadata.

finger-crossed.

5

JerryChang,

I didn’t find support for embedded_metadata_width on the Kernel Sources for TX2. Actually there’s no information in the L4T32.2 release:
https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-322/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fcamera_sensor_prog.html%23wwpID0E0Y40HA

I tried:

embedded_metadata_height = "4";

However it doesn’t work. Is this only used when capturing through the ISP?

I’m setting the output frame as width=1024 and height=4 as only metadata is expected, but it’s not working.

Do you have any other suggestion?

Regards,
-Enrique

6

hello EnriqueR,

we’re having internal discussion about how to support this,
will update this forum discussion thread after we have some conclusions,
thanks

7

JerryChang,

Thank you for your support!

Let me know if you have an update on this.

8

Hi JerryChang,

Do you have news on metadata support?

-Enrique

9

EnriqueR,

Nope, I’m still waiting for internal team to have conclusions.

10

hello EnriqueR,

I have couple of questions need your comments,

  1. is this sensor outputting 4K frame but the data type is all embedded data?
  2. were you tried to work with Argus or simply with v4l2 standard controls?
11

Hi JerryChang,

  1. We have an ISP module that receives RAW images from 2 sensors. Then, this ISP module has 3 output streams: 1 RAW10, 1 RAW8 and the last one is for only metadata. We are able to capture the first 2, but not the metadata stream.

This metadata streams has a 0x30 data-type, but it goes through a GSML stage where the data-type is changed to 0x2A (RAW8). So, on the TX2 side we are trying to capture RAW8 pixel format because there’s no support for metadata or user defined formats.

  1. We have tried v4l2 capture only.

At this point we want to make sure if this conversion from DT=0x30 to DT=0x2A is valid to capture the metadata as a RAW8 stream or if we can capture the pure metadata stream as DT=0x30 on the TX2.

Thanks.

12

Hi JerryChang,

We were able to capture metadata as RAW8 pixel format. The problem was in the GMSL stage, that we had to defined the correct BPP and fixed the resolution using a lower width.

Thank you for your help.

-Enrique

13

hello EnriqueR,

FYI,
our current stack is expecting a normal data frame + optional embedded data frame.
If we only return embedded data frame, our pipeline will probably timeout, even if embedded data is received properly.

however,
converting data-type from 0x30 to 0x2A (RAW8) is the right thing.
please confirm you had receive metadata frame correctly with RAW8 configuration.
thanks

14

Hi JerryChang,

We were able to capture metadata by changing from DT=0x30 to DT=0x2A. However, the data was not complete, there were missing bytes and we are not sure if this problem was caused by the configuration used in the GMSL stage to convert the DT.

We managed to captured the correct data by changed the approach. Now we are receiving the metadata stream with DT=0x2A into the GMSL stage which is managing the data as a RAW8 without doing any DT conversion.

We confirm that we received the metadata correctly as RAW8 into the TX2.

Thank you!