I2C issues on TX2 - Random no acknowledge from address

Hi ,

We are currently working with a TX2 that has 6 cameras attached, there are two cameras per i2c bus on the following buses:

  • c240000
  • 3180000
  • c250000

We are unable to communicate with the cameras randomly, getting the following message on kernel ring buffer :

[  144.693153] tegra-i2c c250000.i2c: no acknowledge from address 0x10

To be a little bit more specific, the TX2 is mounted on a carrier board were we could use the TX1 and it work properly with the same 6 cameras; we switched to the TX2 using the version of Jetpack that is intended for that use and we updated it to include the driver. We also updated the device tree and are able to capture frames, however there seems to exhibit a random I2c communication issue. Currently we suspect that is a software issue, since this is what we checked so far:

1- We tried changing the pull up registers to optimize the waveform on the i2c bus and it didn’t made any difference. So it seems like is not a hardware issue ( also ,it was working on TX1).

2 - The issue seems related to the amount of devices on the i2c bus. When more than one cam is connected, the error occurs most of the time (two cameras on the same bus don’t share the same address, and as I said it was working on the TX1).

3 - The issue seems related to the i2c port. At some combinations of cams the error occurs more, than on others .

We are now guessing that this issue might be related to:

  • i2c speed
  • i2c settings
  • OS i2c resource Management

Is there any difference to the i2c bus configuration on TX2 from TX1? any pointer will be gladly appreciated.

Regards,
JJ

josejich
So have you try to reduce the i2c speed?

  1. Hook up an oscilloscope (or combined scope / logic analyzer) to verify signal integrity and pull-up strength.

  2. Try slower I2C speed.

Hi,

Thanks for the recomendations, I solve my stability issues by lowering the i2c speed to 100khz. I actually found a way to modify the device-tree at run time, I documented it here:

Thanks again!.

@josejich
Thanks for your information. That’s really helpful information for debugging.

Hi ShaneCCC,

Just in case. Reducing the i2c bus frequency to 100kHz was a big improvement in the stability, however there still seems to be an issue because if I start all the cameras with one single pipeline sometimes I don’t get the 6 cameras to work but if I start the 6 cameras one by one, with small sleep periods between pipelines I do get all the cameras to start properly.

Any idea?

Thanks for your help.

Regards,
Edison

How do you start all cameras with one single pipeline? Did you run the argus_camera with multi session?

Hi ShaneCCC,

I’m using gstreamer for testing. This is the pipeline I use:

gst-launch-1.0 -v -e \
v4l2src io-mode=1 device=/dev/video0 ! video/x-bayer, format=rggb, width=1640, height=1232, framerate=30/1 ! fpsdisplaysink name=camera0 text-overlay=false async=false video-sink="appsink max-buffers=2 drop=true" \
v4l2src io-mode=1 device=/dev/video2 ! video/x-bayer, format=rggb, width=1640, height=1232, framerate=30/1 ! fpsdisplaysink name=camera1 text-overlay=false async=false video-sink="appsink max-buffers=2 drop=true" \
v4l2src io-mode=1 device=/dev/video3 ! video/x-bayer, format=rggb, width=1640, height=1232, framerate=30/1 ! fpsdisplaysink name=camera2 text-overlay=false async=false video-sink="appsink max-buffers=2 drop=true" \
v4l2src io-mode=1 device=/dev/video4 ! video/x-bayer, format=rggb, width=1640, height=1232, framerate=30/1 ! fpsdisplaysink name=camera3 text-overlay=false async=false video-sink="appsink max-buffers=2 drop=true" \
v4l2src io-mode=1 device=/dev/video5 ! video/x-bayer, format=rggb, width=1640, height=1232, framerate=30/1 ! fpsdisplaysink name=camera4 text-overlay=false async=false video-sink="appsink max-buffers=2 drop=true"

@Edison
Could you try don’t power off the sensor and skip the sensor initial after first initial to save i2c transfer time to verify the i2c transfer time cause this problem.

Hi ShaneCCC,

I modified the driver so the cameras are configured just once and they are never powered off.

If the cameras are already initialized when I run the pipeline it works fine but if they are not initialized, some times all the cameras start properly but sometimes don’t.

Regards,
Edison

@Edsion
Do you mean launch the multiple camera still got problem sometimes, but single camera don’t?
If that i2c speed should not the root cause. Could you try if v4l2-ctl get the same problem?

v4l2-ctl -d /dev/video0 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=50

@ShaneCCC

If I start one single camera, I have no problem, but if I start all the cameras at once is when I have problems. If when I try to start all the cameras with one single pipeline the cameras are already initialized so there is no need for an I2C transfer they start properly but if when I start the 5 cameras the sensors are not initialized we have to do an I2C transfer and is in that case when it fails randomly (sometimes all the cameras start and sometimes only some of them).

Also decreasing the I2C bus frequency improved the performance making the issue less frequent but it’s still present.

Regards,
Edison

@Edison
Have you verify launch multiple camera by v4l2-ctl to verify if the user space APP cause the issue.

Hi ShaneCCC,

Sorry I didn’t replied your comment before, I didn’t have the hardware to run the test.

I run the same test but this time using v4l2-ctl as you suggested and the result was the same. Sometimes the 5 cameras start properly and some times only some of them start.

Regards,
Edison

@Edison
You may need to confirm the failed only happened on multiple use cause? You may need run some combination to find it out. Or run the single one by one to figure it out.

Hi ShaneCCC,

I tried to download Firmware from driver package, and now I got same I2C problem when I run i2cdetect command.

Even thought I already modified I2C speed to 100KHz as josejich’s recommendation but the problem still be here

-- [   67.086498] tegra-i2c 3160000.i2c: no acknowledge from address 0x11
-- [   67.095235] tegra-i2c 3160000.i2c: no acknowledge from address 0x12
-- [   67.104727] tegra-i2c 3160000.i2c: no acknowledge from address 0x13
-- [   67.113593] tegra-i2c 3160000.i2c: no acknowledge from address 0x14
............

Fixed: it work well when I use console instead of uart debug cable.