Hyperion 7.1 - Cameras failing to start

jpvans · July 12, 2022, 10:11am

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.8.0.10363
other

Host Machine Version
native Ubuntu 18.04
other

Our Hyperion 7.1 Hardware suddenly stopped working and currently I am debugging the camera side of things. When running the roadrunner app, a Camera no Detected message appears on screen so I am checking the sample_camera_gsml_custom output to start with:

xavierA:nvidia@tegra-ubuntu:/usr/local/driveworks-3.0/bin$ ./sample_camera_gmsl_custom
[11-07-2022 18:31:46] Platform: Detected DDPX - Tegra A
[11-07-2022 18:31:46] TimeSource: monotonic epoch time offset is 1657563123869338
[11-07-2022 18:31:46] PTP Time is available from NVPPS Driver
[11-07-2022 18:31:46] Platform: number of GPU devices detected 2
[11-07-2022 18:31:46] Platform: currently selected GPU device discrete ID 0
[11-07-2022 18:31:46] Context::getDataPathFromSelfLocation DATA_ROOT found at: /usr/local/driveworks-3.0/data
[11-07-2022 18:31:46] SDK: Resources mounted from /usr/local/driveworks-3.0/data/
[11-07-2022 18:31:46] SDK: Create NvMediaDevice
[11-07-2022 18:31:46] SDK: Create NvMedia2D
[11-07-2022 18:31:46] SDK: use EGL display as provided
[11-07-2022 18:31:46] TimeSource: monotonic epoch time offset is 1657563123869338
[11-07-2022 18:31:46] PTP Time is available from NVPPS Driver
[11-07-2022 18:31:46] Initialize DriveWorks SDK v3.0.5819-experimental
[11-07-2022 18:31:46] Experimental Release build with GNU 7.3.1 from heads/buildbrain-branch-0-ge470818d679 against Drive PDK v5.1.15.2
[11-07-2022 18:31:46] Rig::fromFile: Loading rig file: /usr/local/driveworks-3.0/data/samples/sensors/camera/camera_gmsl_custom/rig.json
[11-07-2022 18:31:46] rig::DatabaseOverlayer: no valid VIN loaded - skipping overlay
[11-07-2022 18:31:46] rig::DatabaseOverlayer: no valid VIN loaded - skipping overlay
onInitialize: creating camera.gmsl with params: camera-name=SF3324,interface=csi-a,link=0,output-format=processed
[11-07-2022 18:31:46] SensorFactory::createSensor() → camera.gmsl, camera-name=SF3324,interface=csi-a,link=0,output-format=processed
[11-07-2022 18:31:46] SensorFactory::createSensor() → camera.gmsl.master,
[11-07-2022 18:31:46] CameraMaster::parseDevBlock Getting device info list.
[11-07-2022 18:31:46] devBlock: 1 Slave = 0 Interface = csi-a Camera_name = SF3324 Link = 0
[11-07-2022 18:31:46] Camera Match Name: SF3324 Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705 linkIndex: 4294967295 serInfo.Name: MAX96705
[11-07-2022 18:31:46] Client, Setting up information for camera ID 0
[11-07-2022 18:31:46] Client, successfully found info for camera ID 0 bound to id 0
[11-07-2022 18:31:46] Initialize DriveWorks VisualizationSDK v3.0.5819-experimental
[11-07-2022 18:31:46] Initialize DriveWorksGL SDK v3.0.5819-experimental
onInitialize: Total cameras 1
onInitialize: getting image props 0
onInitialize: initilizing stream: 0
[11-07-2022 18:31:47] StreamConsumerGL: successfully initialized
[11-07-2022 18:31:47] StreamProducerCUDA: successfully initialized
Main: Starting master.
[11-07-2022 18:31:47] CameraGSMLMaster: starting…
[11-07-2022 18:31:47] SIPLMaster::SIPLMaster: Setting up master camera
[11-07-2022 18:31:47] Platform:
[11-07-2022 18:31:47] Platform Config:
[11-07-2022 18:31:47] Description:
[11-07-2022 18:31:47] Number of device blocks: 1
[11-07-2022 18:31:47] Device Block : 0
[11-07-2022 18:31:47] csiPort: 0
[11-07-2022 18:31:47] i2cDevice: 0
[11-07-2022 18:31:47] Deserializer Name: MAX96712
[11-07-2022 18:31:47] Deserializer Description: Maxim 96712 Aggregator
[11-07-2022 18:31:47] Deserializer i2cAddress: 41
[11-07-2022 18:31:47] Simulator Mode: 0
[11-07-2022 18:31:47] Slave Mode: 0
[11-07-2022 18:31:47] Phy Mode: 0
[11-07-2022 18:31:47] Number of camera modules: 1
[11-07-2022 18:31:47] CameraModule index: 0
[11-07-2022 18:31:47] Name :SF3324
[11-07-2022 18:31:47] Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705
[11-07-2022 18:31:47] Serializer name: MAX96705
[11-07-2022 18:31:47] Serializer description: Maxim 96705 Serializer
[11-07-2022 18:31:47] Serializer i2cAdress: 64
[11-07-2022 18:31:47] Number of EEPROMs: 0
[11-07-2022 18:31:47] Number of Sensors: 1
[11-07-2022 18:31:47] Sensor index: 0
[11-07-2022 18:31:47] Sensor ID: 0
[11-07-2022 18:31:47] Sensor name: AR0231
[11-07-2022 18:31:47] Sensor description: OnSemi AR0231 Sensor
[11-07-2022 18:31:47] Sensor i2cAddress: 16
[11-07-2022 18:31:47] Sensor isTPGEnabled: 0
[11-07-2022 18:31:47] Sensor isTriggerMode: 1
[11-07-2022 18:31:47] Sensor Number of Virt. Chan.: 0
[11-07-2022 18:31:47] Virt. Chan. Index: 0
[11-07-2022 18:31:47] cfa: 39
[11-07-2022 18:31:47] embeddedTopLines: 24
[11-07-2022 18:31:47] embeddedBottomLines: 4
[11-07-2022 18:31:47] inputFormat: raw12
[11-07-2022 18:31:47] height: 1208
[11-07-2022 18:31:47] width: 1920
[11-07-2022 18:31:47] fps: 30.0000000
[11-07-2022 18:31:47] Embedded Data: 0
[11-07-2022 18:31:47] CameraMaster::setOutputDescription Setting output consumer descriptors for sensor: OutputType: 1
[11-07-2022 18:31:47] Client, setting pipeline config for camera ID 0
[11-07-2022 18:31:47] CameraMaster: master initiation
DEVBLK_WORKER_0: pwr_utils.cpp: 271: PowerControlSetPower: PowerControlSetPower: nvccp_set_cam_pwr_on failed with status (1)
DEVBLK_WORKER_0: CNvMDeviceBlockMgr.hpp: 121: ThreadFunc: Device block init failed(7f)
sample_camera_g: CNvMDeviceBlockMgr.cpp: 97: WorkersDoAction: Device block action:1 failed for block:0
sample_camera_g: CNvMDeviceBlockMgr.cpp: 131: Init: Failed to initialize device blocks
sample_camera_g: CNvMCamera.cpp: 239: Init: DeviceBlock manager init failed
[11-07-2022 18:31:47] CameraMaster: master init failed127
sample_camera_g: CNvMCamera.cpp: 457: GetImageAttributes: Pipeline 0 not found
[11-07-2022 18:31:47] CameraMaster: GetImageAttributes failed1
[11-07-2022 18:31:47] CameraClient: GetImageAttributes failed1
[11-07-2022 18:31:47] CameraClient: Failed to allocate images groups!1
[11-07-2022 18:31:47] Driveworks exception thrown: DW_FAILURE: CameraClient: Failed to allocate images pool!

**** Stack trace ****
/usr/local/driveworks-3.0/bin/…/targets/aarch64-Linux/lib/libdriveworks_visualization.so.3: dw_5ed3c2be0a4fc47847a935953bba7432caaeb8d8()+0x90 [0x7fb0787c80]
/usr/local/driveworks-3.0/bin/…/targets/aarch64-Linux/lib/libdriveworks.so.3: dw_26c551493b354091ceaf8fa0c5c7791e29aa601e()+0xd98 [0x7f9a559610]
/usr/local/driveworks-3.0/bin/…/targets/aarch64-Linux/lib/libdriveworks.so.3: dw_53ece75864c25a07b06e64b3c49d8a04bfcb81d9()+0x64 [0x7f9a54aa9c]
/usr/local/driveworks-3.0/bin/…/targets/aarch64-Linux/lib/libdriveworks.so.3: dw_8db4123cdee707e134ea5c01b34d68cfce0c661f()+0x9c8 [0x7f9a54f048]
/usr/local/driveworks-3.0/bin/…/targets/aarch64-Linux/lib/libdriveworks.so.3: dw_ec58f74e8f4b130a25d923f2f29ba5f19206ad3a()+0x254 [0x7f9a54fbd4]
/usr/local/driveworks-3.0/bin/…/targets/aarch64-Linux/lib/libdriveworks.so.3: dwSensor_start()+0x214 [0x7f9a646974]
./sample_camera_gmsl_custom: CameraCustomSimpleApp::onInitialize()+0x8e8 [0x416cc8]
./sample_camera_gmsl_custom: dw_samples::common::DriveWorksSample::run()+0x30 [0x41bd10]
./sample_camera_gmsl_custom: main()+0x310 [0x411080]
/lib/aarch64-linux-gnu/libc.so.6: __libc_start_main()+0xe0 [0x7f61e0d6e0]
./sample_camera_gmsl_custom() [0x411528]

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2022-07-11 18:31:47] DW Error DW_FAILURE executing DW function:
dwSensor_start(m_camera[0])
at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/sensors/camera/camera_gmsl_custom/main.cpp:361
Aborted (core dumped)
xavierA:nvidia@tegra-ubuntu:/usr/local/driveworks-3.0/bin$ packet_write_wait: Connection to 192.168.0.10 port 22: Broken pipe

Questions:

Is it an I2C problem? It seems the power command is not successfully transmitted for camera power.
How all the cameras can be affected and not be able to stream frames? Is the deserializer engine hardware bad?
Any suggestions to debug further about this issue?

SivaRamaKrishnaNV · July 12, 2022, 11:05am

Dear @jpvans,
I am assuming the same camera is working earlier. Please check below things

Restart board
Change camera or connect these camera to other boards if you have.
Reflash the target to fix SW related issue.

Let us know the feedback.

jpvans · July 12, 2022, 11:11am

@SivaRamaKrishnaNV

Yes, Our setup had been working for more than a year. The working status was confirmed couple of weeks ago and now we are facing this issue.

Because this is a Hyperion 7.1 kit, We don’t have additional cameras but the problem is that all the cameras are not detected so what could be causing this? I can understand if one camera might fail but not all.

Reflashing Xavier A/B will take an additional effort so trying to understand if this is a hardware or software issue. How can we confirm this? I don’t want to reflash both Xaviers and still have the same issue present.

jpvans · July 12, 2022, 11:16am

Also, Can you advise if we can run the sample_camera_gsml_custom app from Xavier B. It is my understanding both Xavier shared a couple of camera feeds correct?

Xavier A, dmesg shows the following:

[48837.590085] tegra-i2c c240000.i2c: no acknowledge from address 0x30
[48837.590398] tegra-i2c c240000.i2c: no acknowledge from address 0x31
[48837.590660] tegra-i2c c240000.i2c: no acknowledge from address 0x32
[48837.590923] tegra-i2c c240000.i2c: no acknowledge from address 0x33
[48837.591181] tegra-i2c c240000.i2c: no acknowledge from address 0x34
[48837.591437] tegra-i2c c240000.i2c: no acknowledge from address 0x35
[48837.591693] tegra-i2c c240000.i2c: no acknowledge from address 0x36
[48837.591978] tegra-i2c c240000.i2c: no acknowledge from address 0x37
[48837.592428] tegra-i2c c240000.i2c: no acknowledge from address 0x50
[48837.592726] tegra-i2c c240000.i2c: no acknowledge from address 0x51
[48837.592997] tegra-i2c c240000.i2c: no acknowledge from address 0x52
[48837.593259] tegra-i2c c240000.i2c: no acknowledge from address 0x53
[48837.593523] tegra-i2c c240000.i2c: no acknowledge from address 0x54
[48837.593791] tegra-i2c c240000.i2c: no acknowledge from address 0x55
[48837.594072] tegra-i2c c240000.i2c: no acknowledge from address 0x56
[48837.594333] tegra-i2c c240000.i2c: no acknowledge from address 0x57
[48837.594598] tegra-i2c c240000.i2c: no acknowledge from address 0x58
[48837.594887] tegra-i2c c240000.i2c: no acknowledge from address 0x59
[48837.595162] tegra-i2c c240000.i2c: no acknowledge from address 0x5a
[48837.595421] tegra-i2c c240000.i2c: no acknowledge from address 0x5b
[48837.595697] tegra-i2c c240000.i2c: no acknowledge from address 0x5c
[48837.595954] tegra-i2c c240000.i2c: no acknowledge from address 0x5d
[48837.596211] tegra-i2c c240000.i2c: no acknowledge from address 0x5e
[48837.596463] tegra-i2c c240000.i2c: no acknowledge from address 0x5f
[48844.672058] tegra-i2c 3160000.i2c: no acknowledge from address 0x30
[48844.672257] tegra-i2c 3160000.i2c: no acknowledge from address 0x31
[48844.672411] tegra-i2c 3160000.i2c: no acknowledge from address 0x32
[48844.672554] tegra-i2c 3160000.i2c: no acknowledge from address 0x33
[48844.672797] tegra-i2c 3160000.i2c: no acknowledge from address 0x34
[48844.672951] tegra-i2c 3160000.i2c: no acknowledge from address 0x35
[48844.673099] tegra-i2c 3160000.i2c: no acknowledge from address 0x36
[48844.673242] tegra-i2c 3160000.i2c: no acknowledge from address 0x37
[48844.673604] tegra-i2c 3160000.i2c: no acknowledge from address 0x50
[48844.673744] tegra-i2c 3160000.i2c: no acknowledge from address 0x51
[48844.673921] tegra-i2c 3160000.i2c: no acknowledge from address 0x52
[48844.674071] tegra-i2c 3160000.i2c: no acknowledge from address 0x53
[48844.674225] tegra-i2c 3160000.i2c: no acknowledge from address 0x54
[48844.674365] tegra-i2c 3160000.i2c: no acknowledge from address 0x55
[48844.674513] tegra-i2c 3160000.i2c: no acknowledge from address 0x56
[48844.674655] tegra-i2c 3160000.i2c: no acknowledge from address 0x57
[48844.674798] tegra-i2c 3160000.i2c: no acknowledge from address 0x58
[48844.674962] tegra-i2c 3160000.i2c: no acknowledge from address 0x59
[48844.675114] tegra-i2c 3160000.i2c: no acknowledge from address 0x5a
[48844.675266] tegra-i2c 3160000.i2c: no acknowledge from address 0x5b
[48844.675423] tegra-i2c 3160000.i2c: no acknowledge from address 0x5c
[48844.675577] tegra-i2c 3160000.i2c: no acknowledge from address 0x5d
[48844.675738] tegra-i2c 3160000.i2c: no acknowledge from address 0x5e
[48844.675890] tegra-i2c 3160000.i2c: no acknowledge from address 0x5f
[48847.936899] tegra-i2c c240000.i2c: no acknowledge from address 0x30
[48847.937175] tegra-i2c c240000.i2c: no acknowledge from address 0x31
[48847.937431] tegra-i2c c240000.i2c: no acknowledge from address 0x32
[48847.937684] tegra-i2c c240000.i2c: no acknowledge from address 0x33
[48847.937957] tegra-i2c c240000.i2c: no acknowledge from address 0x34
[48847.938207] tegra-i2c c240000.i2c: no acknowledge from address 0x35
[48847.938475] tegra-i2c c240000.i2c: no acknowledge from address 0x36
[48847.938737] tegra-i2c c240000.i2c: no acknowledge from address 0x37
[48847.939190] tegra-i2c c240000.i2c: no acknowledge from address 0x50
[48847.939444] tegra-i2c c240000.i2c: no acknowledge from address 0x51
[48847.939696] tegra-i2c c240000.i2c: no acknowledge from address 0x52
[48847.939949] tegra-i2c c240000.i2c: no acknowledge from address 0x53
[48847.940214] tegra-i2c c240000.i2c: no acknowledge from address 0x54
[48847.940470] tegra-i2c c240000.i2c: no acknowledge from address 0x55
[48847.940746] tegra-i2c c240000.i2c: no acknowledge from address 0x56
[48847.941023] tegra-i2c c240000.i2c: no acknowledge from address 0x57
[48847.941275] tegra-i2c c240000.i2c: no acknowledge from address 0x58
[48847.941536] tegra-i2c c240000.i2c: no acknowledge from address 0x59
[48847.941805] tegra-i2c c240000.i2c: no acknowledge from address 0x5a
[48847.942089] tegra-i2c c240000.i2c: no acknowledge from address 0x5b
[48847.942339] tegra-i2c c240000.i2c: no acknowledge from address 0x5c
[48847.942595] tegra-i2c c240000.i2c: no acknowledge from address 0x5d
[48847.942863] tegra-i2c c240000.i2c: no acknowledge from address 0x5e
[48847.943136] tegra-i2c c240000.i2c: no acknowledge from address 0x5f
[48850.288282] tegra-i2c 3180000.i2c: no acknowledge from address 0x30
[48850.288457] tegra-i2c 3180000.i2c: no acknowledge from address 0x31
[48850.288677] tegra-i2c 3180000.i2c: no acknowledge from address 0x32
[48850.288843] tegra-i2c 3180000.i2c: no acknowledge from address 0x33
[48850.289008] tegra-i2c 3180000.i2c: no acknowledge from address 0x34
[48850.289162] tegra-i2c 3180000.i2c: no acknowledge from address 0x35
[48850.289333] tegra-i2c 3180000.i2c: no acknowledge from address 0x36
[48850.289500] tegra-i2c 3180000.i2c: no acknowledge from address 0x37
[48850.289998] tegra-i2c 3180000.i2c: no acknowledge from address 0x51
[48850.290163] tegra-i2c 3180000.i2c: no acknowledge from address 0x52
[48850.290324] tegra-i2c 3180000.i2c: no acknowledge from address 0x53
[48850.290477] tegra-i2c 3180000.i2c: no acknowledge from address 0x54
[48850.290644] tegra-i2c 3180000.i2c: no acknowledge from address 0x55
[48850.290797] tegra-i2c 3180000.i2c: no acknowledge from address 0x56
[48850.290968] tegra-i2c 3180000.i2c: no acknowledge from address 0x57
[48850.291260] tegra-i2c 3180000.i2c: no acknowledge from address 0x59
[48850.291410] tegra-i2c 3180000.i2c: no acknowledge from address 0x5a
[48850.291554] tegra-i2c 3180000.i2c: no acknowledge from address 0x5b
[48850.291719] tegra-i2c 3180000.i2c: no acknowledge from address 0x5c
[48850.291864] tegra-i2c 3180000.i2c: no acknowledge from address 0x5d
[48850.292012] tegra-i2c 3180000.i2c: no acknowledge from address 0x5e
[48850.292159] tegra-i2c 3180000.i2c: no acknowledge from address 0x5f
[48871.188699] tegra-i2c 3190000.i2c: pio timed out addr: 0x30 tlen:12 rlen:4
[48871.193281] tegra-i2c 3190000.i2c: — register dump for debugging ----
[48871.197370] tegra-i2c 3190000.i2c: I2C_CNFG - 0x22c00
[48871.200514] tegra-i2c 3190000.i2c: I2C_PACKET_TRANSFER_STATUS - 0x10001
[48871.204650] tegra-i2c 3190000.i2c: I2C_FIFO_CONTROL - 0x0
[48871.207936] tegra-i2c 3190000.i2c: I2C_FIFO_STATUS - 0x800080
[48871.211478] tegra-i2c 3190000.i2c: I2C_MST_FIFO_CONTROL - 0x70000
[48871.215246] tegra-i2c 3190000.i2c: I2C_MST_FIFO_STATUS - 0x800000
[48871.219006] tegra-i2c 3190000.i2c: I2C_MST_PACKET_TRANSFER_CNT - 0x0
[48871.222913] tegra-i2c 3190000.i2c: I2C_INT_MASK - 0x7d
[48871.226092] tegra-i2c 3190000.i2c: I2C_INT_STATUS - 0x2
[48871.229335] tegra-i2c 3190000.i2c: i2c transfer timed out addr: 0x30

I believe there is an I2C issue on my setup don’t you think?

SivaRamaKrishnaNV · July 12, 2022, 1:02pm

Dear @jpvans,
Yes. For some reason power on command is not working.
You may check running camera sample for camera B as well.
If flashing does not fix it, then it can be concluded as HW issue.
May I know where board was kept? Is it in office or car before you see this issue.

jpvans · July 12, 2022, 2:22pm

@SivaRamaKrishnaNV

We have the Hyperion 7.1 kit installed on a vehicle and I advised, it had been working for more than a year.

Any insight on what could impact the power command to the cameras?

VickNV · July 12, 2022, 5:11pm

Have you observed any suspicious messages in Aurix console?

jpvans · July 12, 2022, 11:53pm

Connected to Aurix port but nothing critical shown during Xavier A bootup and roadrunner execution

VickNV · July 13, 2022, 12:40am

As @SivaRamaKrishnaNV suggested, you may want to clarify if it’s a hardware issue or not by reflashing the system.

jpvans · July 16, 2022, 4:32am

We flashed our Pegasus hardware, Tried roadrunner and PFC with the same behavior: No Camera/Radar detected.

Wanted to check about the next steps, Is it possible to provide an extensive log or how can we get to the bottom of this issue?

VickNV · July 18, 2022, 4:05pm

Which version did you flash? Why does it show DRIVE OS 5.1.15.2 and DriveWorks 3.0? They are not releases supported via the forum. You will need to get support from another channel.