The rawBayerOutput sample appears to have captured incorrect RAW image data

Robotics & Edge Computing Jetson Systems Jetson Orin NX
1

Our team has integrated a MIPI CSI-2 4-lane ToF camera with the NVIDIA Orin NX. The camera outputs Bayer RAW12 format data at a resolution of 640x480. Currently, I want to use Argus to capture several consecutive frames in RAW format to verify data integrity and perform some basic testing (if I can obtain four consecutive RAW images, I can use the 4-phase method to generate a preliminary depth map).

However, I am currently encountering some issues:

I first configured the sensor registers to output a fixed value of 0x800 for every pixel. Using the command nvgstcapture-1.0 --sensor-id=2, I confirmed that each pixel indeed outputs the same value, as expected.

图片
图片1383×516 73.1 KB

Next, I used the nvargus tool to capture a single RAW image and parsed it into a CSV file. The output values for each pixel were consistently 0x800 (displayed as 0x8008, which I assume aligns with the T_R16 Format used on the Orin NX).

图片
图片733×200 26.6 KB

图片
图片742×428 21.2 KB

I then proceeded to use the Argus API sample rawBayerOutput to capture a RAW image, since I plan to later retrieve and process the raw data stream . This generated a file named argus_bayerWithOutIsp.raw. However, upon parsing this file, I found that the pixel values were all different and appeared to be random, suggesting that the data might be incorrect or not what I expected.

图片
图片780×179 16.6 KB

图片
图片1040×622 46.2 KB

My questions are:

  • What could be the reason for this discrepancy in pixel values?
  • Is it actually possible to capture RAW image data using the Argus API samples?
3

How do you configure the sensor output fixed value.
Do you confirm the setting while using nvargus_nvraw capture. Suppose the sensor CID function like xxx_set_exposure/xxx_set_framerate will set the sensor REG to impact the result compare with v4l2-ctl capture.

5

I used a register in the sensor called “pixel statistics,” which monitors the value of each sampling point. When any sampling point exceeds the defined maximum or minimum threshold, the sensor collects statistical information and forces the output value of the erroneous sampling point (pixel) to 0x800.
I set the threshold range very narrowly, ensuring that every pixel would be marked as an error sampling point and would output a fixed value of 0x800.

Additionally, since this ToF camera is still in the debugging phase, I have left functions such as xxx_set_exposure and xxx_set_framerate (CID functions) as empty implementations.

6

The latest update is that the data captured through rawBayerOutput now appears to be correct. When I configured the output so that all pixel values were set to 0x800, the raw format data indeed reflected that setting. This suggests that the previous errors may have been due to a misconfiguration.

I’ve now reconfigured the sensor for normal output mode. My current question is this: after I call setEnableIspStage(false) to skip the ISP (Image Signal Processor) stage, am I truly receiving unprocessed raw sensor data? I need to obtain the original sensor output in order to calculate distance information.

Is the file argus_bayerWithOutIsp.raw produced by rawBayerOutput actually the unprocessed data from the sensor? This point is extremely important to me, and I’m looking forward to your response.

7

Suppose have mini ISP process like demosaic.

8

Even if it’s a very slight modification, it still alters the original output of the sensor, which would lead to inaccurate depth information in my calculations — is that correct? How can this be avoided?

9

No way to avoid. Suggest using v4l2-ctl instead of argus for this kind of case.

10

Does this mean that if we want to obtain truly unmodified raw sensor data, it’s not possible through libargus, and we can only achieve this via the V4L2 interface?

11

Yes, suppose you can use MMAPI but argus API for your software implementation.

Thanks

12

We have also attempted another approach to obtain raw images using libargus by accessing the iImage’s mapbuffer, copying its content, and saving it in raw format. Will the raw data obtained through this method still be affected by the ISP’s demosaicing operation?

Argus::UniqueObj<EGLStream::Frame> frame(iFrameConsumer->acquireFrame(FIVE_SECONDS_IN_NANOSECONDS, &status));
EGLStream::IFrame *iFrame = Argus::interface_cast<EGLStream::IFrame>(frame);
EGLStream::Image *image = iFrame->getImage();
EGLStream::IImage *iImage = Argus::interface_cast<EGLStream::IImage>(image);   
uint16_t *p_s = (uint16_t*)iImage->mapBuffer();
uint16_t* output_data = new uint16_t[640*480];
size_t destIndex = 0;
for(int y=0;y<480;y++){
for(int x=0;x<640;x++)
            {
                // output_data[destIndex++] = *p_s>>4;
                output_data[destIndex++] = *p_s;
                    p_s++;
            }
        }
std::ostringstream filename;
        filename << "oneShot_"  << ".raw";
std::ofstream file (filename.str(),std::ios::out | std::ios::binary);
file.write(reinterpret_cast<const char*>(output_data),sizeof(uint16_t)*640*480);
file.close();
13

Yes, this still involve ISP.