Frame drop Notification when execute camera sample

[Natsuha_Shishido]

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.2.10884
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Camera A: 1876 x 2896 = 5,432,896 [pix] 30fps
Camera B: 2160 x 3840 = 8,294,400 [pix] 30fps

When I connected four Camera A to GMSL Group A and one Camera B to GMSL Group D (The throughput is 0.9 Gpix/sec.), and executed the Camera Sample app provided by DW, a message indicating frame drops from SIPL was output.

[13-04-2023 09:19:14] CameraClient: Notification received from pipeline index:4 of type: NOTIF_WARN_ICP_FRAME_DROP
[13-04-2023 09:19:14] CameraClient: Notification received from pipeline index:0 of type: NOTIF_WARN_ICP_FRAME_DROP
[13-04-2023 09:19:14] CameraClient: Notification received from pipeline index:3 of type: NOTIF_WARN_ICP_FRAME_DROP
[13-04-2023 09:19:14] CameraClient: Notification received from pipeline index:1 of type: NOTIF_WARN_ICP_FRAME_DROP
[13-04-2023 09:19:14] CameraClient: Notification received from pipeline index:2 of type: NOTIF_WARN_ICP_FRAME_DROP

Is it exceeding the limits of Orin’s hardware performance or the driver’s performance for the camera input?

In the case of only four Camera A are connected to GMSL Group A (The throughput is 0.65 Gpix/sec.), this message will not be outputted.

[SivaRamaKrishnaNV]

Dear @Natsuha_Shishido,
So when 4 cameras connected to port A, you don’t see frame drops. When you connect an additional camera to port D, you see frame drop.
Did you check with nvsipl_camera NVmedia sample as well? Could you share the rigfile used with sample_Camera DW app?

Is the used camera modules are part of DRIVE AGX Orin Sensors & Accessories | NVIDIA Developer ?

[Natsuha_Shishido]

Hi @SivaRamaKrishnaNV

Thank you for suggesting to run nvsipl_camera! As a result of running it, there were no frame drops observed.
This means that there is a problem with sample_camera.
Furthermore, because the main thread’s cycle is not keeping up with 30fps, the function dwSensorCamera_returnFrame() is also not keeping up, resulting in frame drops due to the queue becoming full. Since the cause has been identified, my question has been resolved.
Thank you!

[SivaRamaKrishnaNV]

Dear @Natsuha_Shishido,
Glad to hear the issue is resolved. Did you make any changes in DW camera sample code to fix this? If so, could you share the change done in the DW sample?

[Natsuha_Shishido]

@SivaRamaKrishnaNV

Due to the confidentiality of the work, I cannot show the source code, but the implementation details are as follows:

1. Generate streamer thread for each camera to do CUDA to GL type conversion.
2. Send CUDA image handles for each camera to the streamer conversion thread.
3. Send GL image handles from the streamer conversion thread to the main thread.
4. Render in the main thread.

When multiple cameras were connected, the type conversion using the streamer was taking too much time. Therefore, we shortened the overall processing time by separating the processing from the main thread.