Access to a capture frame end embedded data

hello authors

I have 3 questions.
my enviroment is below

module: Jetson orin nano 8GB (P3703)
carrier board : Jetson orin nano dev kit(P3768)
camera : custom csi camera (mipi 2lane , w=640, height=480, embedded=2)

now, i accomplished developing custom kernel driver for my custom cam.
I can get capture frames using “v4l2-ctl” utility.

$ v4l2-ctl -d /dev/video0 --set-ctrl=bypass_mode=0 --stream-mmap --stream-count=5
<<<<<

① my cutom camera have 2 lines of embedded data at frame tail position.
Can my module parse that emmbedded data? According to tracing log data,It looks success to parse embedded datas.

kworker/2:5-138 [002] … 1847.613770: rtcpu_vinotify_event: tstamp:58713806062 cch:0 vi:0 tag:FS channel:0x00 frame:1 vi_tstamp:1878838564512 data:0x0000000100000011
kworker/2:5-138 [002] … 1847.613772: rtcpu_vinotify_event: tstamp:58713806320 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:1 vi_tstamp:1878838564544 data:0x0000000800000000
kworker/2:5-138 [002] … 1847.613772: rtcpu_vinotify_event: tstamp:58713806609 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:1 vi_tstamp:1878839282400 data:0x0000000000000001
kworker/2:5-138 [002] … 1847.613772: rtcpu_vinotify_event: tstamp:58713806856 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:1 vi_tstamp:1878839284736 data:0x0000000008020001
kworker/2:5-138 [002] … 1847.613772: rtcpu_vinotify_event: tstamp:58713807141 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:1 vi_tstamp:1878841645248 data:0x0000000001df0002
kworker/2:5-138 [002] … 1847.613773: rtcpu_vinotify_event: tstamp:58713807386 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:1 vi_tstamp:1878841646144 data:0x0000000000000000
kworker/2:5-138 [002] … 1847.613773: rtcpu_vinotify_event: tstamp:58713807661 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:1 vi_tstamp:1878841649888 data:0x0000000002020001
kworker/2:5-138 [002] … 1847.613773: rtcpu_vinotify_event: tstamp:58713807905 cch:0 vi:0 tag:CHANSEL_EMBED_SOF channel:0x23 frame:1 vi_tstamp:1878841670752 data:0x0000000000000004
kworker/2:5-138 [002] … 1847.613773: rtcpu_vinotify_event: tstamp:58713808183 cch:0 vi:0 tag:CHANSEL_EMBED_EOF channel:0x23 frame:1 vi_tstamp:1878841672992 data:0x0000000000010008
kworker/2:5-138 [002] … 1847.613774: rtcpu_vinotify_event: tstamp:58713808431 cch:0 vi:0 tag:FE channel:0x00 frame:1 vi_tstamp:1878841673568 data:0x0000000100000021
kworker/2:5-138 [002] … 1847.613774: rtcpu_vinotify_event: tstamp:58713811285 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:1 vi_tstamp:1878841673600 data:0x0000000800000000
kworker/2:5-138 [002] … 1847.613774: rtcpu_vinotify_event: tstamp:58713811542 cch:0 vi:0 tag:ATOMP_EMB_DATA_DONE channel:0x23 frame:1 vi_tstamp:1878841673792 data:0x0000000000000000
kworker/2:5-138 [002] … 1847.613774: rtcpu_vinotify_event: tstamp:58713904611 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:1 vi_tstamp:1878841686048 data:0x0000000003020001
kworker/2:5-138 [002] … 1847.613774: rtcpu_vinotify_event: tstamp:58713904865 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:1 vi_tstamp:1878841689632 data:0x0000000000020001

② If my module can parse embedded data, I attemptiing to caputure embedded data (data type = 0x12) . How I access to emmbedded meta data?
According to the previous post topic, embedded data is buffered at "chan → emb_buf” (vi5_fops.c).
But I can’t find v4l2 API name which is “chan → emb_buf”. I want to access it via v4l2 API.

③ if “chan-> emb_buf” API is not exist, do I need to create a cutom API to access embedded data ?

betst regards.

hello Orinbeginner_XXX,

there’s chan->emb_buf; to allocates private buffer for embedded data storage. you may retrieved via user-space.

Hello Jerry Chang,

Thanks for quick replay!

Good news for me that orin nano can capture frame tail embedded data.

but, I don’t know how to allocate mamery from kermnel space to user space.
Could you please provide me a way to move it to user space, manuals or link to past discussion?

best regards.

hello Orinbeginner_XXX,

you may see-also VI-5 driver,
for instance, $public_sources/kernel_src/kernel/nvidia/drivers/media/platform/tegra/camera/vi/vi5_fops.c
for your use-case, embedded data at the bottom of each frame.
image_buf_offset = x
emb_buf_offset = x + size of image data

hello Jerry Chang, thanks for your reply.

you may see-also VI-5 driver,

yes, I refer from VI-5 driver.

image_buf_offset = x
emb_buf_offset = x + size of image data

That is good idea. So, I change pix height value in “vi5_setup_surface” descrepter , but an error occured.
I think that if I expand the height line length in the vi-5 driver, Atomp is not be able to capture embedded data
because of other data type. (pix=raw12 , Emb=emb8) embedded data may be streamed in other vi port.
I tried to expand buffering size by modifying vi-5 diver as shown below.

So, I have to custom another driver to execute you pointed out. Couled you please tell me which driver I shouled change?

best regards.

modifyied at “source/public/kernel/nvidia/drivers/media/platform/tegra/camera/vi/vi5_fops.c”

+ desc->ch_cfg.frame.frame_y = height + chan->embedded_data_height;
+ desc_memoryinfo->surface[0].size = chan->format.bytesperline * ( height + chan->embedded_data_height);

error “dmesg”

[ 141.056949] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 1, flags: 0, err_data 131072
[ 141.067186] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 2, flags: 0, err_data 131072
[ 141.077403] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 3, flags: 0, err_data 131072
[ 141.087616] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 4, flags: 0, err_data 131072

error “tracing log”

 kworker/1:3-126     [001] ....   193.792300: rtcpu_vinotify_event: tstamp:7054028623 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225723499552 data:0x399d580010000000
 kworker/1:3-126     [001] ....   193.792303: rtcpu_vinotify_event: tstamp:7054028923 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225723509280 data:0x0000000031000001
 kworker/1:3-126     [001] ....   193.792304: rtcpu_vinotify_event: tstamp:7054029265 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225727080288 data:0x399d550010000000
 kworker/1:3-126     [001] ....   193.792304: rtcpu_vinotify_event: tstamp:7054029554 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225727090176 data:0x0000000031000002
 kworker/1:3-126     [001] ....   193.855605: rtcpu_vinotify_event: tstamp:7057221319 cch:0 vi:0 tag:FS channel:0x00 frame:1 vi_tstamp:225827823904 data:0x0000000100000011
 kworker/1:3-126     [001] ....   193.855608: rtcpu_vinotify_event: tstamp:7057221582 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:1 vi_tstamp:225827823936 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855608: rtcpu_vinotify_event: tstamp:7057221869 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:1 vi_tstamp:225828541792 data:0x0000000000000001
 kworker/1:3-126     [001] ....   193.855608: rtcpu_vinotify_event: tstamp:7057222114 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:1 vi_tstamp:225828544096 data:0x0000000008020001
 kworker/1:3-126     [001] ....   193.855608: rtcpu_vinotify_event: tstamp:7057222393 cch:0 vi:0 tag:CHANSEL_EMBED_SOF channel:0x23 frame:1 vi_tstamp:225830930112 data:0x0000000000000004
 kworker/1:3-126     [001] ....   193.855609: rtcpu_vinotify_event: tstamp:7057222636 cch:0 vi:0 tag:CHANSEL_EMBED_EOF channel:0x23 frame:1 vi_tstamp:225830932384 data:0x0000000000010008
 kworker/1:3-126     [001] ....   193.855609: rtcpu_vinotify_event: tstamp:7057222913 cch:0 vi:0 tag:FE channel:0x00 frame:1 vi_tstamp:225830932960 data:0x0000000100000021
 kworker/1:3-126     [001] ....   193.855609: rtcpu_vinotify_event: tstamp:7057223160 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x02 frame:1 vi_tstamp:225830932960 data:0x01e0200001000000
 kworker/1:3-126     [001] ....   193.855609: rtcpu_vinotify_event: tstamp:7057223435 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:1 vi_tstamp:225830949056 data:0x0000000001020001
 kworker/1:3-126     [001] ....   193.855610: rtcpu_vinotify_event: tstamp:7057223684 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:1 vi_tstamp:225830932992 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855610: rtcpu_vinotify_event: tstamp:7057223964 cch:0 vi:0 tag:ATOMP_EMB_DATA_DONE channel:0x23 frame:1 vi_tstamp:225830933152 data:0x0000000000000000
 kworker/1:3-126     [001] ....   193.855610: rtcpu_vinotify_event: tstamp:7057224207 cch:0 vi:0 tag:FS channel:0x00 frame:2 vi_tstamp:225830975776 data:0x0000000200000011
 kworker/1:3-126     [001] ....   193.855610: rtcpu_vinotify_event: tstamp:7057320734 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:2 vi_tstamp:225830975808 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855611: rtcpu_vinotify_event: tstamp:7057320991 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225831085920 data:0x399d580010000000
 kworker/1:3-126     [001] ....   193.855611: rtcpu_vinotify_event: tstamp:7057321279 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225831095808 data:0x0000000031000003
 kworker/1:3-126     [001] ....   193.855611: rtcpu_vinotify_event: tstamp:7057321526 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:2 vi_tstamp:225831693632 data:0x0000000000000001
 kworker/1:3-126     [001] ....   193.855611: rtcpu_vinotify_event: tstamp:7057321806 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:2 vi_tstamp:225831695968 data:0x0000000008020002
 kworker/1:3-126     [001] ....   193.855611: rtcpu_vinotify_event: tstamp:7057322050 cch:0 vi:0 tag:CHANSEL_EMBED_SOF channel:0x23 frame:2 vi_tstamp:225834081984 data:0x0000000000000004
 kworker/1:3-126     [001] ....   193.855612: rtcpu_vinotify_event: tstamp:7057322325 cch:0 vi:0 tag:CHANSEL_EMBED_EOF channel:0x23 frame:2 vi_tstamp:225834084256 data:0x0000000000010008
 kworker/1:3-126     [001] ....   193.855612: rtcpu_vinotify_event: tstamp:7057322568 cch:0 vi:0 tag:FE channel:0x00 frame:2 vi_tstamp:225834084800 data:0x0000000200000021
 kworker/1:3-126     [001] ....   193.855612: rtcpu_vinotify_event: tstamp:7057322850 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x02 frame:2 vi_tstamp:225834084832 data:0x01e0200001000000
 kworker/1:3-126     [001] ....   193.855612: rtcpu_vinotify_event: tstamp:7057323095 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:2 vi_tstamp:225834100832 data:0x0000000001020002
 kworker/1:3-126     [001] ....   193.855612: rtcpu_vinotify_event: tstamp:7057323375 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:2 vi_tstamp:225834084832 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855613: rtcpu_vinotify_event: tstamp:7057323621 cch:0 vi:0 tag:ATOMP_EMB_DATA_DONE channel:0x23 frame:2 vi_tstamp:225834085056 data:0x0000000000000000
 kworker/1:3-126     [001] ....   193.855613: rtcpu_vinotify_event: tstamp:7057323899 cch:0 vi:0 tag:FS channel:0x00 frame:3 vi_tstamp:225834127648 data:0x0000000300000011
 kworker/1:3-126     [001] ....   193.855613: rtcpu_vinotify_event: tstamp:7057324212 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:3 vi_tstamp:225834127680 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855613: rtcpu_vinotify_event: tstamp:7057324492 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225834219520 data:0x399d550010000000
 kworker/1:3-126     [001] ....   193.855613: rtcpu_vinotify_event: tstamp:7057324733 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:225834229408 data:0x0000000031000004
 kworker/1:3-126     [001] ....   193.855614: rtcpu_vinotify_event: tstamp:7057515799 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:3 vi_tstamp:225834845472 data:0x0000000000000001
 kworker/1:3-126     [001] ....   193.855614: rtcpu_vinotify_event: tstamp:7057516053 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:3 vi_tstamp:225834847744 data:0x0000000008020003
 kworker/1:3-126     [001] ....   193.855614: rtcpu_vinotify_event: tstamp:7057516337 cch:0 vi:0 tag:CHANSEL_EMBED_SOF channel:0x23 frame:3 vi_tstamp:225837233856 data:0x0000000000000004
 kworker/1:3-126     [001] ....   193.855614: rtcpu_vinotify_event: tstamp:7057516582 cch:0 vi:0 tag:CHANSEL_EMBED_EOF channel:0x23 frame:3 vi_tstamp:225837236096 data:0x0000000000010008
 kworker/1:3-126     [001] ....   193.855614: rtcpu_vinotify_event: tstamp:7057516857 cch:0 vi:0 tag:FE channel:0x00 frame:3 vi_tstamp:225837236672 data:0x0000000300000021
 kworker/1:3-126     [001] ....   193.855615: rtcpu_vinotify_event: tstamp:7057517105 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x02 frame:3 vi_tstamp:225837236672 data:0x01e0200001000000
 kworker/1:3-126     [001] ....   193.855615: rtcpu_vinotify_event: tstamp:7057517382 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:3 vi_tstamp:225837252704 data:0x0000000001020003
 kworker/1:3-126     [001] ....   193.855615: rtcpu_vinotify_event: tstamp:7057517631 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:3 vi_tstamp:225837236704 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855615: rtcpu_vinotify_event: tstamp:7057517906 cch:0 vi:0 tag:ATOMP_EMB_DATA_DONE channel:0x23 frame:3 vi_tstamp:225837236928 data:0x0000000000000000
 kworker/1:3-126     [001] ....   193.855615: rtcpu_vinotify_event: tstamp:7057518147 cch:0 vi:0 tag:FS channel:0x00 frame:4 vi_tstamp:225837279520 data:0x0000000400000011
 kworker/1:3-126     [001] ....   193.855616: rtcpu_vinotify_event: tstamp:7057518426 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:4 vi_tstamp:225837279520 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.855616: rtcpu_vinotify_event: tstamp:7057518685 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:4 vi_tstamp:225837997312 data:0x0000000000000001
 kworker/1:3-126     [001] ....   193.855616: rtcpu_vinotify_event: tstamp:7057518962 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:4 vi_tstamp:225837999616 data:0x0000000008020004
 kworker/1:3-126     [001] ....   193.855616: rtcpu_vinotify_event: tstamp:7057519205 cch:0 vi:0 tag:CHANSEL_EMBED_SOF channel:0x23 frame:4 vi_tstamp:225840385728 data:0x0000000000000004
 kworker/1:3-126     [001] ....   193.855616: rtcpu_vinotify_event: tstamp:7057519480 cch:0 vi:0 tag:CHANSEL_EMBED_EOF channel:0x23 frame:4 vi_tstamp:225840387968 data:0x0000000000010008
 kworker/1:3-126     [001] ....   193.855617: rtcpu_vinotify_event: tstamp:7057519720 cch:0 vi:0 tag:FE channel:0x00 frame:4 vi_tstamp:225840388544 data:0x0000000400000021
 kworker/1:3-126     [001] ....   193.908290: rtcpu_vinotify_error: tstamp:7057663914 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x02 frame:5 vi_tstamp:225845207200 data:0x0000000000000589
 kworker/1:3-126     [001] ....   193.908292: rtcpu_vinotify_error: tstamp:7057762466 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x02 frame:6 vi_tstamp:225848359168 data:0x0000000000000589
 kworker/1:3-126     [001] ....   193.908294: rtcpu_vinotify_event: tstamp:7057844112 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x02 frame:4 vi_tstamp:225840388544 data:0x01e0200001000000
 kworker/1:3-126     [001] ....   193.908294: rtcpu_vinotify_event: tstamp:7057844362 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:4 vi_tstamp:225840406240 data:0x0000000001020004
 kworker/1:3-126     [001] ....   193.908294: rtcpu_vinotify_event: tstamp:7057844649 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:4 vi_tstamp:225840388576 data:0x0000000800000000
 kworker/1:3-126     [001] ....   193.908295: rtcpu_vinotify_event: tstamp:7057844896 cch:0 vi:0 tag:ATOMP_EMB_DATA_DONE channel:0x23 frame:4 vi_tstamp:225840388768 data:0x0000000000000000

hello Orinbeginner_xyz,

you should not revise the code, embedded data has already allocated at the bottom of each frame.

BTW, is it mandatory for using v4l to fetch metadata?
there’s Argus API. i.e. Argus::Ext::ISensorPrivateMetadata::getMetadata to copy back the metadata to the provided memory location.

hello Jerry Chang

Thanks for your replay!

you should not revise the code, embedded data has already allocated at the bottom of each frame.

Oh, It seems I misunderstood. I understand embedded data already allocated at the end of frame.

is it mandatory for using v4l to fetch metadata?

The reason why I used v4l2src is to show Grayscale (GRAY16_LE) and used it to get raw image data. Then I tried to use “nvargus_nvraw”, I can get raw image data formed by pix data and embedded data, by changing data type (GRAY16_LE → NV12 ) and using it.

Does “nvargus_nvraw” tool use “getMetadata” API?
And why “v4l2” can only capture raw image data formed by pix data only?
Is embedded data lost any part way to application?

there’s Argus API. i.e. Argus::Ext::ISensorPrivateMetadata::getMetadata to copy back the metadata to the provided memory location.

If the “getMetadata” API can allocate embedded data at the end of frame data, I’ll also use it for developing to my custom viewer software.

best regards.

hello Orinbeginner_xyz,

please see-also Camera Architecture Stack. v4l2src and libargus they went through difference pipelines.
as mentioned in previous comment #5, embedded data has already allocated.
you may use the Argus API to retrieve metadata, or calculating the offset to fetch it via user-space.

hello JerryChang, thanks for your reply.

you may use the Argus API to retrieve metadata, or calculating the offset to fetch it via user-space.

I’ll also use Argus API first. And after, If It’s not suitable for my software, I try to allocate metadata offset to image data.Thank a lot.

best regards.