While working with MIPI sensors on both the Jetson Nano and Xavier NX platforms, we noticed that VI2 and VI5 handle the RAW data from CSI differently, as @tim.cassens also observed and well described in this post. After checking with the platform-respective TRMs (Tegra X1 TRM for Nano and Xavier TRM for NX), we got even more confused. So, could someone help us clarify our confusion? That would be very much appreciated! Here are the questions that we have:
According to the TRM, RAW10 data from CSI are converted via VI into
(on NX)MSB-aligned16-bit data with the15th bit being 0:
(on Nano)MSB-aligned16-bit data with the14&15th bit being 0:
But NONE of which matches the data byte that we got from v4l2-ctl. What we observed:
(on NX) the 16-bit data is indeed MSB-aligned, but the 15th bit is NOT always 0;
(on Nano) the 16-bit data is however LSB-aligned, which means that top 6 bits (15th-10th) are 0 and there is no MSB’s replication on the lower bits.
So, is the description in the TRM about the raw data formatting actually wrong, especially for Nano?
If the description about the raw data formatting for NX is correct, what does the 15th bit mean? Can it be 1 instead 0?
According to vi2_formats.h and vi5_formats.h, RAW10 is converted to T_R16 on NX and T_R16_I on Nano. But in NX’s TRM, only T_R16_I is documented and it looks obviously different than the T_R16_I documented in Nano’s TRM. Could it be that it is documented wrong in NX’s TRM? The T_R16_I in NX’s TRM is actually meant to be T_R16?
(very important to us) The MSB-aligned 16-bit data is not easy for us to handle in the user space. Besides, we don’t want the replication of MSB bits at all. From user’s perspective, RAW10 should simply be LSB-aligned 16-bit data with the top 6 bits being 0. Why doesn’t VI convert the raw data this way? Since we need the data directly from v4l2-ctl (not through ISP), is there a way to disable VI’s raw data formatting?
FYI,
instead of T_R16_I, Xavier is used T_R16 to handle Raw10, it is a 16-bit unsigned word format, packing two per 32-bit word.
there’re some MSBs replicated. please see [Table 7.21 RAW Memory Formats] and also [Figure 7.17 T_R16_I Format with Examples for RAW6, RAW10, RAW12, and RAW14] for details.
here’s an example that you can capture into CUDA buffer and implement CUDA code for debayering.
for capturing frame date into CUDA buffer, please take a look at… /usr/src/jetson_multimedia_api/samples/v4l2cuda/
Hi @jerrychang, thank you for your reply! But the answer has unfortunately nothing to do with my questions. I’ve been through the TRM regarding data formatting many times and the Figure 7.17 that you mentioned is exactly the screenshot that I posted above in my question. So could you please go through my questions and provide some guidance? Thank you very much!
The v4l2cuda sample code is made for cameras that output YUYV format. Our camera currently outputs RAW format and we don’t work with CUDA. So the CUDA sample code is not much of a help to us, but thanks for mentioning that. What we need on the user end is the real RAW data format without any bit-shift or MSB replication, which means per pixel:
RAW10 should be 0 0 0 0 0 0 D9 D8 D7 D6 D5 D4 D3 D2 D1 D0
RAW12 should be 0 0 0 0 D11 D10 D9 D8 D7 D6 D5 D4 D3 D2 D1 D0
please check this Argus sample, /usr/src/jetson_multimedia_api/argus/samples/cudaBayerDemosaic.
this sample demonstrate CUDA engine to demosaics RAW16 into RGBA.
Thank you @JerryChang. So it means that it is not possible to disable VI’s pixel conversion, right? And how about questions 1-3 above? We still didn’t get any resolutions on that. Thanks!
on Xavier,
T_R16 format is used. This is an unsigned 16 bit integer. The data is in MSBs and LSBs are replicated from MSBs. For example, for RAW12, data is in bits 15:4 and bits 15:12 are replicated in bits 3:0.
that’s something like this packing formats… d11 d10 d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 x x x x
Hi @JerryChang, what you wrote about the packing format is also what we observed. But T_R16 is not documented, right? In Xavier TRM we could only find the T_R16_I format which looks like (for RAW12): 0 d11 d10 d9 d8 d7 d6 d5 d4 d3 d2 d1 d0 x x x. It has a zero at the MSB (15th bit). It is not what you wrote or what we saw from our raw image data. Thanks!
