Cameras not detected after hard boot sometimes

I have 2 sony imx sensors connected via MIPI cables to my jetson xavier. I don’t understand the sequence with which cameras are detected and mounted as devices. Sometimes when I hard reboot the system, both the cameras will not be detected and mounted, sometimes only one camera will be detected and mounted and sometimes both of them will be detected. When they are not detected, I usually do a soft reboot of my xavier and 100% of the time, both the cameras will be detected and mounted.
In short with hard boot, sometimes cameras are not detected but they will be after a soft boot.

I usually check my dmesg to see if they were detected while system was booting up or I check if they are mounted as /dev/video0 or /dev/video1.

There is an interesting thing I have noticed. Even if both the cameras are not detected ( dmesg or /dev/video* is not there ), I check my i2cdump to get further information and I can see my camera ids there. Which means it’s not the connection issue and cameras are responding.
My i2cdump status are here…

i2cdump -y 30 0x56
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff cc 66 72 61 6d 6f 73 2d 69 6d 78 33    ....?framos-imx3
20: 30 34 2d 30 00 ff ff ff ff ff ff ff ff ff ff ff    04-0............
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................

and for 2nd camera

i2cdump -y 32 0x56
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
10: ff ff ff ff cc 66 72 61 6d 6f 73 2d 69 6d 78 33    ....?framos-imx3
20: 30 34 2d 32 00 ff ff ff ff ff ff ff ff ff ff ff    04-2............
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
50: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
60: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
70: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
80: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
90: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
c0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
d0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
e0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
f0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................

Can you give some more insight about what this could be happening?
Is there anyway or some script I can run after my camera have booted up, so that I can mount those camera devices as /dev/video* ??

hello saumil.shah,

are you using the camera sensor from the framos team, is it IMX304 camera sensor?

Yes Jerry. I am using imx304 camera sensors from framos.
All the patches are already in the place and they are working as well. It’s just the matter of reliability, cause sometimes they are not being detected on hard boot.

hello saumil.shah,

please also share the JetPack release version you’re currently working with,
you may also check release tag for confirmation, i.e. $ cat /etc/nv_tegra_release

# R32 (release), REVISION: 4.4, GCID: 23942405, BOARD: t186ref, EABI: aarch64, DATE: Fri Oct 16 19:37:08 UTC 2020

This is the version of my jetpack.

Hi @saumil.shah,

where do you connect image sensors on FPA (which J# connector). From i2cdump it seems they are on J5 and J6. Are they always there?

In the case when devices are not detected could you dump this cmds:
ls /proc/device-tree/chosen/plugin-manager/ids/
uname -r

and attach the dmesg log after boot.

If this is a binding ids problem you could set up a static device tree for imx304, compile and flash the device tree.

Thanks

uname -r  
 4.9.140-tegra

And

When it detected just one camera

ls /proc/device-tree/chosen/plugin-manager/ids/
2888-0004-400  framos-adapter  framos-imx304-0  name

When it detected both the cameras

2888-0004-400  framos-adapter  framos-imx304-0  framos-imx304-2  name

Attaching dmesg output after boot for both the scenarios.
one_camera_failed.txt (83.1 KB)
camera_found.txt (83.2 KB)

Can you explain a bit how can I setup a static device tree for this cameras. I have source files for device trees and it is being used by the kernel right now.

I did try to add devices manually when one camera was not detected, but got the error that prob failed.


Adding some information here. framos generic drivers was mounted as i2cdevice 30-0022 and 32-0022 was the one I added manually
One camera was mounted as 30-001a and 32-001a was the one I tried to add manually.

Hi @saumil.shah,

from your logs plugin manager and driver probe is working as expected. In your ids dump when you only have one sensor detected there is only framos-imx304-0 id. framos-imx304-2 id is missing.

This reading of ids in EEPROM is happening in cboot. It looks main cam i2c bus on adress 0x54 (framos-adapter id) where you have specified i2c mux address (0x70). Then it loops trough i2c mux channels ch0-ch7 on specific range of i2c adresses and reads FSM EEPROM(framos-imx304-0). Why this fails in certain cases is maybe related with EEPROM power supply not active (disabled on FSA dip switch 1 pin(it’s not the case if the sensor is detected in other cases), eeprom power is not stable). Do you have custom carrier?

If you want to enable a static device tree you need to do the following (included in user guide for source package).
a. Locate and edit the .dtsi file
sources/hardware/nvidia/platform/t19x/galen/kernel-dts/tegra194-p2888-0001-p2822-0000.dtsi
b. Remove the following lines from the file
#include “common/tegra194-p2822-camera-modules.dtsi”
#include “t19x-common-modules/tegra194-camera-plugin-manager.dtsi”
c. Include a specific file. For example include imx304 sensor
#include “common/tegra194-p2822-0000-camera-imx304.dtsi”

  1. Buid the device tree (source the helper script (check the paragraph 4. Compile custom kernel) and execute build_dtree)
  2. Flash the device tree
    sudo ./flash.sh -k kernel-dtb jetson-xavier mmcblk0p1

Thanks

Yes we do have a custom carrier board from connectec.
Can you suggest any idea about how can we check if eeprom power is stable or not? Any pins where we can connect a scope?

Hi @saumil.shah,

you should check the 1V8_VDD signal that has TP on the next boards:

  • FPA test point pin TP70
  • FSA test point pin TP10

Could you also please attach cboot log? You can check here for the reference.

Thanks