Multiple GMSL camera setup failure, gmsl-slave mode

aswb · August 9, 2021, 11:26pm

I have created a program that can successfully capture GMSL imagery from a connected GMSL camera(SF3324).
But when I try to run multiple instances of the program in different windows the following error occurs:

[D] [13:24:14.255] CameraMaster: master initiation
DEVBLK_WORKER_0: pwr_utils.cpp: 196: PowerControlRequestOwnership: PowerControlRequestOwnership: nvccp_request_ownership failed with status (1005)
DEVBLK_WORKER_0: CNvMDeviceBlockMgr.hpp: 147: ThreadFunc: Device block init failed(7f)
ld-node-camera-: CNvMDeviceBlockMgr.cpp: 129: WorkersDoAction: Device block action:1 failed for block:0
ld-node-camera-: CNvMDeviceBlockMgr.cpp: 163: Init: Failed to initialize device blocks
ld-node-camera-: CNvMCamera.cpp: 436: Init: DeviceBlock manager init failed

The scenario is like this : Port A0 - GMSL camera SF3324, A1 - GMSL camera SF3324, A2 - GMSL camera SF3324.
I have given the parameters accordingly in the three instances : csi-a link-0, csi-a link-1, csi-a link-2. But only the camera started 1st is ran successfully. The other two are failing with the above error.

Protocol used is camera.gmsl, Do I need to use some other protocol for this?

I have not specified a rig file, so I think the default one is used.

I tried to launch a single camera in slave mode , but ran into the same issue.

std::string parameterString = std::string("output-format=yuv420-uint8,camera-name=SF3324");
    parameterString             += std::string(",slave=1");
    parameterString             += std::string(",interface=")+ std::string(settings.interface);
    parameterString             += std::string(",link=") + std::to_string(settings.link);
    parameterString             += std::string(",fifo-size=") + std::to_string(settings.fifoSize);
    params.parameters           = parameterString.c_str();
    params.protocol             = "camera.gmsl";

The complete Log for reference:

[D] [13:24:13.166] Configurations read - Interface : csi-a, Link : 1 FiFoSize : 4, FullResolution : 0, Compressed: true, LogLevel: trace
[T] [13:24:13.166] Platform: Detected DDPX - Tegra A
[D] [13:24:13.166] TimeSource: monotonic epoch time offset is 1628502824278816
[W] [13:24:13.166] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[E] [13:24:13.166] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[W] [13:24:13.167] TimeSource: Could not detect valid PTP time source at 'eth0'. Fallback to CLOCK_MONOTONIC.
[T] [13:24:13.901] Platform: number of GPU devices detected 1
[T] [13:24:13.901] Platform: currently selected GPU device integrated ID 0
[T] [13:24:13.904] Context::getDataPathFromSelfLocation DATA_ROOT found at: /usr/local/driveworks-3.5/data
[D] [13:24:13.905] SDK: No resources(.pak) mounted, some modules will not function properly
[D] [13:24:13.905] SDK: Create NvMediaDevice
[D] [13:24:13.905] SDK: Create NvMedia2D
[D] [13:24:13.916] egl::Display: found 1 EGL devices
[D] [13:24:13.917] egl::Display: use drm device: /dev/dri/card0
[D] [13:24:14.058] TimeSource: monotonic epoch time offset is 1628502824278816
[W] [13:24:14.058] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to eth0
[E] [13:24:14.058] TimeSource Eth: Lost PTP time synchronizaton. Synchronized time will not be available from this timesource.
[W] [13:24:14.058] TimeSource: Could not detect valid PTP time source at 'eth0'. Fallback to CLOCK_MONOTONIC.
[D] [13:24:14.059] Initialize DriveWorks SDK v3.5.75
[T] [13:24:14.059] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gc61a9a35bd0 against Drive PDK v5.2.0.0
[I] [13:24:14.059] Initialized SDK
[I] [13:24:14.060] Initialized Sensor Abstraction Layer
[D] [13:24:14.060] SensorFactory::createSensor() -> camera.gmsl, output-format=yuv420-uint8,camera-name=SF3324,interface=csi-a,link=1,fifo-size=4
[T] [13:24:14.060] CameraBase: pool size set to 8
[D] [13:24:14.060] SensorFactory::createSensor() -> camera.gmsl.master, 
[D] [13:24:14.094] CameraMaster::parseDevBlock Getting device info list.
[D] [13:24:14.094] devBlock: 1 Slave = 0 Interface = csi-a Camera_name = SF3324 Link = 1
[D] [13:24:14.095] Camera Match Name: SF3324 Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705 linkIndex: 4294967295 serInfo.Name: MAX96705
[D] [13:24:14.095] Client, Setting up information for camera ID 0
[D] [13:24:14.095] Client, successfully found info for camera ID 0 bound to id 0
[D] [13:24:14.247] CameraClient: no NITO found at /opt/nvidia/nvmedia/nit/SF3324.nito
[D] [13:24:14.247] CameraClient: using NITO found at /opt/nvidia/nvmedia/nit/sf3324.nito
[T] [13:24:14.247] CameraGSMLMaster: starting...
[D] [13:24:14.247] SIPLMaster::SIPLMaster: Setting up master camera
[T] [13:24:14.254] Platform: 
[T] [13:24:14.254] Platform Config: 
[T] [13:24:14.254] Description: 
[T] [13:24:14.254] Number of device blocks: 1
[T] [13:24:14.254] Device Block : 0
[T] [13:24:14.254]      csiPort: 0
[T] [13:24:14.254]      i2cDevice: 0
[T] [13:24:14.254]      Deserializer Name: MAX96712
[T] [13:24:14.254]      Deserializer Description: Maxim 96712 Aggregator
[T] [13:24:14.254]      Deserializer i2cAddress: 41
[T] [13:24:14.254]      Simulator Mode: 0
[T] [13:24:14.254]      Slave Mode: 0
[T] [13:24:14.254]      Phy Mode: 0
[T] [13:24:14.254]      Number of camera modules: 1
[T] [13:24:14.254]      CameraModule index: 1
[T] [13:24:14.254]              Name :SF3324
[T] [13:24:14.254]              Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705
[T] [13:24:14.255]              Serializer name: MAX96705
[T] [13:24:14.255]              Serializer description: Maxim 96705 Serializer
[T] [13:24:14.255]              Serializer i2cAdress: 64
[T] [13:24:14.255]                      Sensor ID: 0
[T] [13:24:14.255]                      Sensor name: AR0231
[T] [13:24:14.255]                      Sensor description: OnSemi AR0231 Sensor
[T] [13:24:14.255]                      Sensor i2cAddress: 16
[T] [13:24:14.255]                      Sensor isTPGEnabled: 0
[T] [13:24:14.255]                      Sensor isTriggerMode: 1
[T] [13:24:14.255]                               cfa: 39
[T] [13:24:14.255]                               embeddedTopLines: 24
[T] [13:24:14.255]                               embeddedBottomLines: 4
[T] [13:24:14.255]                               inputFormat: 8
[T] [13:24:14.255]                               height: 1208
[T] [13:24:14.255]                               width: 1920
[T] [13:24:14.255]                               fps: 30.0000000
[T] [13:24:14.255]                               Embedded Data: 0
[D] [13:24:14.255] CameraMaster::setOutputDescription Setting output consumer descriptors for sensor:  OutputType: 1
[D] [13:24:14.255] Client, setting pipeline config for camera ID 0
[D] [13:24:14.255] CameraMaster: master initiation
DEVBLK_WORKER_0: pwr_utils.cpp: 196: PowerControlRequestOwnership: PowerControlRequestOwnership: nvccp_request_ownership failed with status (1005)
DEVBLK_WORKER_0: CNvMDeviceBlockMgr.hpp: 147: ThreadFunc: Device block init failed(7f)
ld-node-camera-: CNvMDeviceBlockMgr.cpp: 129: WorkersDoAction: Device block action:1 failed for block:0
ld-node-camera-: CNvMDeviceBlockMgr.cpp: 163: Init: Failed to initialize device blocks
ld-node-camera-: CNvMCamera.cpp: 436: Init: DeviceBlock manager init failed
[E] [13:24:19.269] CameraMaster: master init failed127
ld-node-camera-: CNvMCamera.cpp: 568: GetImageAttributes: Incorrect pipeline state:1 for pipeline:0
[D] [13:24:19.270] CameraMaster: GetImageAttributes failed6
[D] [13:24:19.270] CameraClient: GetImageAttributes failed6
[D] [13:24:19.270] CameraClient: Failed to allocate images groups!6
[E] [13:24:19.270] Driveworks exception thrown: DW_FAILURE: CameraClient: Failed to allocate images pool!
[E] [13:24:19.270] 
[C] [13:24:19.270] [2021-08-09 13:24:19] DW Error DW_FAILURE executing DW function:
 dwSensor_start(mCamera)
 at /home/nvidia/TCs/ld-node-camera-gmsl/builds-aarch64/ld-node-camera-gmsl_1628163020589/work/src/CameraGmsl.cpp:122
[D] [13:24:19.270] Deinit master camera
ld-node-camera-: CNvMCamera.cpp: 1088: Deinit: Incorrect pipeline state:1 for pipeline:0
[D] [13:24:19.270] Failed to Deinit master camera!6
[D] [13:24:21.270] CameraClient: Stopping client
[I] [13:24:21.287] Camera stopped and released
[D] [13:24:21.287] Releasing camera master
ld-node-camera-: CNvMCamera.cpp: 1088: Deinit: Incorrect pipeline state:1 for pipeline:0
ld-node-camera-: CNvMCamera.cpp: 48: ~CNvMCamera: CNvMCamera object fails to deinit
nvmedia: ERROR: MAX96712: Bad parameter: Null ptr
ld-node-camera-: CNvMMax96712.cpp: 244: Deinit: CDI_CONFIG_MAX96712_DISABLE_REPLICATION failed! with NvMedia error: 1
ld-node-camera-: CNvMDeviceBlock.cpp: 353: Deinit: Deserializer Deinit failed[I] [13:24:21.288] Sensor Abstaraction Layer released
[D] [13:24:21.289] Releasing Driveworks SDK Context
[D] [13:24:21.289] SDK: Release NvMediaDevice
[D] [13:24:22.000] SDK: Release NvMedia2D
[I] [13:24:22.001] SDK released
In Main: Initialization failed

Please provide the following info (check/uncheck the boxes after clicking “+ Create Topic”):
Software Version
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.6.0.8170
1.5.1.7815
1.5.0.7774
other

Host Machine Version
native Ubuntu 18.04
other

VickNV · August 10, 2021, 5:19pm

Hi @aswb,

Please check Context document. This may limit running multiple camera applications.

aswb · August 10, 2021, 5:34pm

Hi @VickNV , Thanks for the response.

I have not created a single application to handle multiple cameras.
The application controls only one GMSL camera.

The issue happens only when I run multiple instances of the same program.
OR
When I run a single instance of the program with slave mode = true.

The camera workflow is successful upto dwSAL_createSensor.

The error is happening at dwSensor_start(),
I have followed the workflow as mentioned at DriveWorks SDK Reference: Camera Workflow (nvidia.com)

VickNV · August 10, 2021, 6:02pm

For slave mode, you may need to have a primary process already programming the camera.
“Secondary Capture” section in DRIVE OS document has some explanation about it.

aswb · August 10, 2021, 6:08pm

Okay, other than using slave mode and secondary capture.

Can you help figure out why this issue is happening when multiple instances of the GMSL capture program is ran simultaneously.

The 1st error/failure that is observed from the log is :

 DEVBLK_WORKER_0: pwr_utils.cpp: 196: PowerControlRequestOwnership: PowerControlRequestOwnership: nvccp_request_ownership failed with status (1005)
DEVBLK_WORKER_0: CNvMDeviceBlockMgr.hpp: 147: ThreadFunc: Device block init failed(7f)

VickNV · August 10, 2021, 6:23pm

It is because NVIDIA Camera Control Protocol (NVCCP) failed to connect to MCU.
The details are in “SoC to Microcontroller Communications” document.

VickNV · August 10, 2021, 6:25pm

Below is from the document. FYI.

2.Camera Group Ownership
•Any camera application MUST request a camera group ownership using request_ownership() API before power control and release the ownership using release_ownership()API at application exit.
•Each Xavier can capture independently as primary if they capture on camera groups exclusive to each other.
•A camera group can ONLY have one primary and one secondary for dual Xavier capture.
•For all dual Xavier cases, Xavier A is ALWAYS the Primary and Xavier B is ALWAYS the secondary.

VickNV · August 10, 2021, 8:21pm

In further releases, DriveWorks will support controlling the same camera from two applications with new frameworks. FYI.

aswb · August 10, 2021, 9:11pm

Okay, thanks for the info.

FYI : I am not trying control the same camera from two applications. The two cameras are connected to separate ports on the DevKit, Camera 1 on Port A-0 and Camera 2 on port A-1.

VickNV · August 10, 2021, 9:38pm

Thanks for clarifying your use case.

aswb · August 11, 2021, 7:46am

@VickNV

With current release of driveworks, Is there any way to run two GMSL image capture applications (both using protocol-camera.gmsl working on separate cameras) to run simultaneously ?

I observed similar error/failure when I ran two instances of the sample application - DriveWorks SDK Reference: Camera Sample (nvidia.com). to control two separate cameras connected on port A (separate rig.json files) , with 1st instance on csi-a link-0 and 2nd instance on csi-a link-b.

VickNV · August 11, 2021, 2:21pm

As you can see in Multiple GMSL camera setup failure, gmsl-slave mode - #11 by VickNV, it’s not covered in the current design.