Record rgba data from ISP2 using nvsipl_camera and get all zero output

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.9.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Issue Description
I try to dump rgba image using nvsipl_camera tool and enable ISP2.

sudo ./nvsipl_camera -c AR0823_REV2_96717_CPHY_x2 -m “0 0 0x0010 0” -0 -1 --skipFrames 10 --writeFrames 10 -f AR0823_1029_test

The process didn’t show any error, log is down in log section. However, when I check the .rgba data, all pixels are zeros. Since the data is between [0, 1.0], I multiplied the data by 2^16 but still return all zeros.

rgba_image : max_value: 0 , min_value: 0 size: (2168, 3848, 3)

**The camera is supported since Raw data and YUV data from ISP0 are all correct

Logs

dev@tegra-ubuntu:~$ sudo ./nvsipl_camera -c AR0823_REV2_96717_CPHY_x2 -m "0 0 0x0010 0" -0 -1 --skipFrames 10 --writeFrames 10 -f AR0823_1029_test22
****GetAR0823ConfigSet: Set Debug Log Level to 1****
Pipeline: 9 ISP Output: 2 is using RGBA PACKED FLOAT16 PL SENSOR_RGBA
MAX96712: Revision 5 detected
MAX96712 Link 1: PHY optimization was enabled
MAX96717: Revision 4 detected!
AR0823CheckPresence: Chip id is 0xe54
AR0823 Rev readbuff: 0x12, majorRev: 2, cfa: 1, minorRev: 0
Sensor AR0823 GRBG Rev 2.0 detected!
AR0823: /home/nvidia/ar0823_rev2_ofl_calib.txt file not found, running with default settings.
Enabling Streaming...
nvsipl_camera: Opened NITO file for module "AR0823_REV2_96717", file name: "/usr/share/camera/AR0823_REV2_96717.nito"

DEBUG :: ReCalculate3ExpTime :: START
---- sensor & clock settings ------------------
[       pCLK ]  148500000 Hz
[  llc (HTS) ]       2132 pCLK  # line  length in pCLK
[  line_rate ] 69652.906250 Hz  # pCLK / llc
[     fit[0] ]       1997
[     fit[1] ]       2386
---- requested --------------------------------
[     exp_t1 ] 0.01002300 sec
[     exp_t2 ] 0.00022100 sec
[      T1/T2 ]         45
---- converted register values ----------------
[     cit_t1 ]        698 (0x02ba)
[     cit_t2 ]         16 (0x0010)
[     T1/T2 ]         46
---- register to time achieved ----------------
[     exp_t1 ] 0.01002021 sec
[     exp_t2 ] 0.00021706 sec
DEBUG :: ReCalculate3ExpTime :: END

Enter 'les' followed by sensor ID to enable LED
Enter 'lds' followed by sensor ID to disable LED
Enter 'dl' followed by sensor ID to disable the link
Enter 'el' followed by sensor ID to enable the link without module reset
Enter 'elr' followed by sensor ID to enable the link with module reset
Enter 'cm' to check the module availability
Enter 'q' to quit the application
-
Sensor9         Frame captured: 44              Frame drops: 14         Frame discontinuities: 0
SUCCESS
dev@tegra-ubuntu:~$
dev@tegra-ubuntu:~$ ll AR0823_1029_test22_cam_9_out_3.rgba
-rw-r--r--. 1 root root 667397120 Oct 29 21:14 AR0823_1029_test22_cam_9_out_3.rgba
dev@tegra-ubuntu:~$


Hi Jason,

It might be the reason that the camera that you have doesn’t support ISP2. Have you checked with your supplier that ISP2 is supported? What is the reason for using ISP2 instead of ISP0? Are you trying to use both?

Hi @extern.ray.xie
ISP2 should be generated by Orin hardware not the camera module right?
My goal is to record ISP2 RGBA data. ISP0 and raw pipeline are used to verify the camera is working.

I don’t know which side provides support for ISP0/1/2 (ISP0 and ISP1 seems to be the same?). Or maybe both sides need to be compatible with each other. For our cameras, I have tried to use ISP2 to serialize camera frames into h264 streams but it would always return an error. I can’t remember the exact error code but it does seem like our cameras don’t support ISP2. I don’t know about your camera modules specs so I can’t say for sure. I just want to share with you my experience.

Dear @jason.fu ,
What is the camera module? Is it tested on 6.0.10 as well as 6.0.9 is not supported release via forum.
Also, how did you access the RGBA values. Could you share code snippet?

Hi @SivaRamaKrishnaNV
the camera is onsemi 0823 REV2 with 96717

dev@tegra-ubuntu:~$ sudo ./nvsipl_camera -l | grep 823
AR0823_REV2_96717_CPHY_x2 :AR0823 REV2 RGGB MARS module with MAX96717 Serializer in 2 lane CPHY mode

I didn’t check on 6.0.10, but I do check on 6.0.6 with onsemi 0823 REV1.1 with 96717 which is Working.

The code I use to access RGBA data is following.

width = 3848   # Replace with the actual width of your image
height = 2168  # Replace with the actual height of your image
rgba_image_2 = np.zeros((height, width, 4), dtype=np.float16)
# Load the .rgba file (this file is assumed to contain raw RGBA data)
with open('color_test/ori_raw3/AR0823_1029_test22_cam_9_out_3.rgba', 'rb') as f:
    rgba_data = np.frombuffer(f.read(), dtype=np.float16)
stepsize = width * height * 4
# Read first frame
rgba_image1 = rgba_data[0:stepsize]
# Reshape the data to the correct image dimensions (height, width, 4 channels for RGBA)
rgba_image = rgba_image1.reshape((height, width, 4))
# Find rgb channel
rgba_image_2[:, :, 3] = rgba_image[:, :, 3]
# Increase rgb channel from fp16 to uint16
rgba_image_unit16 = (rgb_image_2 * 65504.0).astype(np.uint16)
print("rgb_image: max_value:", np.max(rgba_image_unit16), ", min_value:", np.min(rgba_image_unit16), ", size: ", rgba_image_unit16.shape)


@extern.ray.xie
In my points of view, camera module provide more like raw images itself and Orin do the all ISP parts and output each processed images. Maybe the nito file from module company may affects the ISP output process, but need to be confirmed.

Hi @SivaRamaKrishnaNV
Any thoughts you have for this issue?

Dear @jason.fu,
Is it Entron F009A120RM0A? We support only the camera modules DRIVE AGX Orin Sensors & Accessories | NVIDIA Developer
ISP pipeline uses NITO files provided by vendor. You need to reach out to respective sensor vendor to get the files to bring up the camera module.