Hello, our team is planning to develop a driver for an Mipi CSI-2 iTOF camera on the Orin NX platform. Currently, both the device tree and the driver are in an initial working state. Using the gst+nvarguscamerasrc command, we can see that the camera is outputting a data stream.
Since a ToF camera does not require Argus ISP processing like an RGB camera, our goal is to obtain the raw data from the camera and process it to generate depth frames. This camera uses a four-phase method to acquire depth information, meaning that it outputs four sub-frames within one frame period. We need to use these four sub-frames to construct a single depth frame.
To achieve this, we wrote a simple piece of code based on some examples from the libargus library (such as the oneShot example). This code can capture continuous frame data from the sensor. In theory, once we obtain this data, we should be able to proceed with the depth calculations.
Unfortunately, in a previous post I was informed that the raw images acquired via the Argus method are not the actual original output from the sensor. Even when using setEnableIspStage(false) to skip the ISP post-processing stage, the raw data still undergoes de-mosaicing. This suggests the data may be altered, potentially compromising the accuracy of the depth frame.
https://forums.developer.nvidia.com/t/the-rawbayeroutput-sample-appears-to-have-captured-incorrect-raw-image-data/331306/7?u=star_sea
To investigate this further, we conducted a simple test. According to the camera sensor’s manual, there is a MIPI test mode where each pixel sequentially outputs a regular pseudo-random value. In 640x480 mode, the visualization of this pattern looks like this.
We configured the camera into this test mode and compared the raw data obtained through the following two methods:
- We wrote code to generate 640x480 data following the rules specified in the sensor manual, where each value is generated according to the documented pseudo-random pattern.
- We used an Argus example to capture the raw data output from the sensor.
If the Argus example applied a de-mosaicing operation to the raw data, the results from method (1) and method (2) should differ, right? However, after comparison, we found that the data obtained using both methods was identical.
Does this mean that the Argus example we are currently using is indeed capturing the sensor’s original output? And if so, what does the previously mentioned de-mosaicing operation refer to in this context?