Driveworks sample_camera issue with certain GMSL camera setups

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.10.0
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
2.1.0
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello Nvidia Team,

our plan is to run 4 imx623 in one camera group (csi-ef) and 1 imx728 camera in another group (csi-ab) on the DRIVE Agx Orin via GMSL. When doing so we observed some behavior that is not explainable to us, so hopefully you can support and clarify.

To simplify and discard possible error sources, we tested our rig files with your provided sample_camera application. Our .rig file with 4 imx623 and 1 imx728 camera failed to run and returned the error ( log_nvidia_sensor_fails_4_plus_1.txt (116.9 KB)):

Main: Starting master.
[03-12-2025 13:02:54] CameraClient: startImpl: thread id: 33132
[03-12-2025 13:02:54] CameraClient: initializeAndStartCameras: thread id: 33132
MAX96712: Revision 5 detected
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX96712 Link 0: PHY optimization was enabled
MAX96712 Link 1: PHY optimization was enabled
MAX96712 Link 2: PHY optimization was enabled
MAX96712 Link 3: PHY optimization was enabled
MAX96717: Revision 4 detected!
MAX96717F: Revision 4 detected!
Sensor IMX728 Rev 4 RGGB detected!
Sensor IMX623 Rev 8 RGGB detected!
MAX96717F: Revision 4 detected!
Sensor IMX623 Rev 8 RGGB detected!
MAX96717F: Revision 4 detected!
[03-12-2025 13:02:55] SIPLClient: ImageQueue timeout
[03-12-2025 13:02:55] SIPLNotifier: Event notification queue timeout
[03-12-2025 13:02:55] eventQueuePollCustomInterface: Custom interface is not available. This is expected if bootstrap is not complete
[03-12-2025 13:02:55] SIPLClient: ImageQueue timeout
[03-12-2025 13:02:55] SIPLNotifier: Event notification queue timeout
[03-12-2025 13:02:55] SIPLClient: ImageQueue timeout
[03-12-2025 13:02:55] SIPLNotifier: Event notification queue timeout
[03-12-2025 13:02:55] SIPLClient: ImageQueue timeout
[03-12-2025 13:02:55] SIPLNotifier: Event notification queue timeout
[03-12-2025 13:02:55] SIPLClient: ImageQueue timeout
[03-12-2025 13:02:55] SIPLNotifier: Event notification queue timeout
Sensor IMX623 Rev 8 RGGB detected!
[03-12-2025 13:02:56] CameraMaster: master init failed with status = 10
[03-12-2025 13:02:56] Driveworks exception thrown: DW_UNEXPECTED_EVENT: CameraMaster::resume failed with sipl status 10

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2025-12-03 14:02:56] DW Error DW_UNEXPECTED_EVENT executing DW function:
 dwSensor_start(m_camera[i])
 at /usr/local/driveworks/samples/src/sensors/camera/camera/main.cpp:911
Aborted (core dumped)

However, when running the setup with 4 imx623 and 4 imx728 sensors, the sensors start and we can see the 8 video streams. The rig files are identical, despite having the 3 imx728 removed for the non working case.

rig_4+1cams_imx728_imx623.txt (5.0 KB)

rig_4+4cams_imx728_imx623.txt (8.0 KB)

Can you reproduce the issue? Can you tell us, why the requested setup of 4 + 1 camera is not working, but if you add three sensors to the rig file, it is?

For further clarification we have conducted additional tests with different setups with surprising results. E.g. we tried to split the imx623 between the different interfaces and found out that this only works in combination with the imx728 sensors when exactly 2 imx728 sensors are used. If you only use 1 imx728 sensor, the application fails. If it is helpful we can also share the whole list of permutations we have tested on the different ci interfaces.

Thanks and regards!

3

Moving the topic to DRIVE AGX Orin General - NVIDIA Developer Forums

4

Dear @kristian.fischer ,
Could you verify running two instances of nvsipl_Camera(One for → 4 imx623 in one camera group (csi-ef) and another one ->1 imx728 camera in group (csi-ab)) and share logs?

5

Dear SivaRamaKrishnaNV,

please find attached the rig files and the corresponding logs for my experiments with different camera configurations. From the hardware perspective I had attached 4 imx728 cameras at csi-ab and 4 imx623 at csi-ef for all tests.

The following tests are covered:

1 Pinhole => worked

4 SV cameras => worked

4 Pinhole + 4 SV cameras => worked

1 Pinhole + 4 SV cameras => did not start.

test_cameras_drive_orin.zip (35.7 KB)

6

Dear SivaRamaKrishnaNV,

do you have any updates on this issue?

7

Do you observe this behavior consistently or in some runs it is observed?

Do you see
I notice you have shared sample_camera logs. Could you share the nvsipl_camera logs for this use case.
Also, for this issue please set env varaible DW_SIPL_NVMEDIA_VERBOSITY=4 and share /var/log/syslog , dmesg along with sample_camera logs when we hit error

8

Dear SivaRamaKrishnaNV,

we observe the non-starting cases every time. With the permutations that are actually supposed to work, sometimes (maybe every tenth start), they are not starting as well with the same error.

Regarding your request with the nvispl camera application: I have conducted two tests. The hardware setup was 2x2 imx623 sensors at interface csi-ef and csi-gh, respectively (see this rig file rig_2+2cams_imx623.txt (4.0 KB)).

  1. Test with camera_sample app: running the camera sample app with this rig file does not start. These are the console output log_rig_2+2cams_imx623.txt (1.3 MB) and the corresponding syslog syslog_camera_sample.txt (46.4 KB).
  2. Testing the same permutation of 2x2 imx623 sensors works with the nvsipl app call
    sudo ./nvsipl_camera --platform-config “V1SIM623S4RU5195NB3_CPHY_x4” --link-enable-masks “0x0000 0x0000 0x0011 0x0011”
    Here the resulting console output is log_nvsipl_2x2_imx623.txt (2.9 KB).