Camera program breakdown!

Hi,
Our carrier board have 2 USB2.0 ports off USB1_D_N/P and USB2_D_N/P.We connect 2 cameras to run at the same time for object recognition.After some time,one of cameras will break down but the other one is OK.

Then,we inspect 2 USB2.0 ports that they are all amounted on the same USB-BUS of BUS-2.When one port break down,the port device number for USB2.0 change from video0、video1 to video0、video2.

So,because of the changing of port device number lead to the port’s breakdown.And then,we looking the syslog find that the port is re-enabled by hub.there are the syslog information:

–>15:52:15

what is the reason for this?

Can we get the rate of BUS-2 on the command?

hello Wance,

dual usb-camera preview works normally on our site.

could you share more information to us.
for example,

  1. did dual camera preview works normally with your setup?
  2. any suspicious logs caused one of camera crash?
  3. please execute tegrastats to monitor the system usage for your use-case.

Hi JerryChang,

Dual camera preview works normally at the start.After about several hours,one camera will crash.

But we don’t think it is problem for camera.We run just one camera 48 hours in High temperature box 70°C that is OK.
Under normal atmospheric temperature,we run two camera and one camera crash that it is probability 100% last several hours.

The picture is the kern.log at /var/log.According to the log,some resaon causes the USB2.0 re-enable ?

According to our carrier board,we add a USB2.0 port on the demo carrier board,as the follow picture.
We run two camera on the board and the camera still crash after some time with the built-in USB port.

Hi JerryChang,
I try again with tegrastats.When one of camera crash,I stop the monitoring.
The following is the data at the last 60 seconds:

RAM 3724/3853MB (lfb 17x4MB) cpu [53%,54%,44%,74%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 14%@998 EDP limit 1734
RAM 3724/3853MB (lfb 17x4MB) cpu [54%,60%,67%,45%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3719/3853MB (lfb 17x4MB) cpu [72%,50%,47%,50%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 21%@998 EDP limit 1734
RAM 3723/3853MB (lfb 17x4MB) cpu [57%,47%,72%,65%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3719/3853MB (lfb 17x4MB) cpu [54%,66%,73%,42%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 8%@998 EDP limit 1734
RAM 3721/3853MB (lfb 17x4MB) cpu [62%,47%,62%,58%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3722/3853MB (lfb 17x4MB) cpu [52%,78%,55%,55%]@1734 EMC 29%@1600 AVP 0%@80 GR3D 0%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [52%,51%,69%,54%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3724/3853MB (lfb 17x4MB) cpu [69%,58%,54%,45%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3719/3853MB (lfb 17x4MB) cpu [60%,62%,60%,49%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 5%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [58%,62%,59%,50%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 76%@998 EDP limit 1734
RAM 3720/3853MB (lfb 17x4MB) cpu [50%,55%,56%,65%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3721/3853MB (lfb 17x4MB) cpu [55%,54%,52%,75%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 55%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [56%,68%,50%,55%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3721/3853MB (lfb 17x4MB) cpu [54%,63%,59%,54%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 0%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [69%,45%,52%,56%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 2%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [69%,53%,61%,55%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3722/3853MB (lfb 17x4MB) cpu [48%,60%,59%,57%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 0%@998 EDP limit 1734
RAM 3729/3853MB (lfb 17x4MB) cpu [51%,53%,73%,58%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 40%@998 EDP limit 1734
RAM 3722/3853MB (lfb 17x4MB) cpu [59%,50%,51%,59%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3727/3853MB (lfb 17x4MB) cpu [60%,53%,58%,65%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 0%@998 EDP limit 1734
RAM 3728/3853MB (lfb 17x4MB) cpu [61%,63%,52%,55%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3721/3853MB (lfb 17x4MB) cpu [72%,57%,46%,57%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 77%@998 EDP limit 1734
RAM 3720/3853MB (lfb 17x4MB) cpu [64%,51%,52%,58%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 9%@998 EDP limit 1734
RAM 3726/3853MB (lfb 17x4MB) cpu [55%,41%,61%,73%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3729/3853MB (lfb 17x4MB) cpu [62%,69%,53%,51%]@1734 EMC 29%@1600 AVP 0%@80 GR3D 7%@998 EDP limit 1734
RAM 3720/3853MB (lfb 17x4MB) cpu [86%,54%,47%,63%]@1734 EMC 22%@1600 AVP 0%@115 GR3D 0%@998 EDP limit 1734
RAM 3719/3853MB (lfb 17x4MB) cpu [82%,75%,62%,42%]@1734 EMC 21%@1600 AVP 0%@80 GR3D 0%@998 EDP limit 1734
RAM 3726/3853MB (lfb 17x4MB) cpu [59%,42%,48%,71%]@1734 EMC 29%@1600 AVP 0%@80 GR3D 20%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [64%,46%,50%,69%]@1734 EMC 29%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3730/3853MB (lfb 17x4MB) cpu [65%,46%,46%,62%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3727/3853MB (lfb 17x4MB) cpu [59%,61%,56%,70%]@1734 EMC 29%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [63%,50%,63%,32%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 3%@998 EDP limit 1734
RAM 3728/3853MB (lfb 17x4MB) cpu [64%,47%,62%,46%]@1734 EMC 32%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3727/3853MB (lfb 17x4MB) cpu [69%,64%,47%,45%]@1734 EMC 30%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3725/3853MB (lfb 17x4MB) cpu [54%,51%,42%,80%]@1734 EMC 31%@1600 AVP 0%@80 GR3D 94%@998 EDP limit 1734
RAM 3726/3853MB (lfb 17x4MB) cpu [60%,61%,50%,53%]@1734 EMC 33%@1600 AVP 0%@80 GR3D 99%@998 EDP limit 1734
RAM 3741/3853MB (lfb 17x4MB) cpu [59%,49%,49%,72%]@1734 EMC 28%@1600 AVP 0%@80 GR3D 0%@998 EDP limit 1734
RAM 3723/3853MB (lfb 17x4MB) cpu [70%,50%,64%,61%]@1734 EMC 27%@1600 AVP 0%@115 GR3D 99%@998 EDP limit 1734
RAM 3722/3853MB (lfb 17x4MB) cpu [80%,68%,50%,80%]@1734 EMC 15%@1600 AVP 0%@115 GR3D 8%@998 EDP limit 1734
RAM 3729/3853MB (lfb 17x4MB) cpu [81%,76%,64%,45%]@1734 EMC 14%@1600 AVP 0%@115 GR3D 0%@998 EDP limit 1734
RAM 3721/3853MB (lfb 17x4MB) cpu [100%,99%,27%,1%]@1734 EMC 6%@1600 AVP 12%@115 GR3D 0%@998 EDP limit 1734
RAM 3723/3853MB (lfb 17x4MB) cpu [89%,74%,67%,42%]@1734 EMC 14%@1600 AVP 3%@80 GR3D 99%@998 EDP limit 1734
RAM 3727/3853MB (lfb 17x4MB) cpu [62%,53%,63%,62%]@1734 EMC 25%@1600 AVP 1%@80 GR3D 7%@998 EDP limit 1734
RAM 3728/3853MB (lfb 17x4MB) cpu [63%,60%,70%,48%]@1734 EMC 29%@1600 AVP 0%@115 GR3D 99%@998 EDP limit 1734
RAM 3730/3853MB (lfb 17x4MB) cpu [69%,57%,65%,46%]@1734 EMC 29%@1600 AVP 0%@115 GR3D 7%@998 EDP limit 1734
RAM 3730/3853MB (lfb 17x4MB) cpu [75%,79%,72%,51%]@1734 EMC 20%@1600 AVP 0%@115 GR3D 0%@998 EDP limit 1734
RAM 3723/3853MB (lfb 17x4MB) cpu [90%,83%,72%,28%]@1734 EMC 10%@1600 AVP 1%@115 GR3D 4%@998 EDP limit 1734
RAM 3723/3853MB (lfb 17x4MB) cpu [77%,74%,52%,68%]@1734 EMC 17%@1600 AVP 0%@115 GR3D 5%@998 EDP limit 1734
RAM 3299/3853MB (lfb 31x4MB) cpu [89%,61%,75%,75%]@1734 EMC 13%@1600 AVP 3%@115 GR3D 0%@998 EDP limit 1734
RAM 1743/3853MB (lfb 203x4MB) cpu [64%,30%,12%,23%]@1734 EMC 20%@1600 AVP 12%@80 GR3D 99%@998 EDP limit 1734
RAM 1743/3853MB (lfb 204x4MB) cpu [43%,1%,1%,4%]@1734 EMC 26%@1600 AVP 15%@80 GR3D 99%@998 EDP limit 1734
RAM 1743/3853MB (lfb 204x4MB) cpu [20%,19%,0%,5%]@1734 EMC 28%@1600 AVP 11%@80 GR3D 99%@998 EDP limit 1734
RAM 1743/3853MB (lfb 204x4MB) cpu [41%,0%,1%,1%]@1734 EMC 28%@1600 AVP 12%@80 GR3D 99%@998 EDP limit 1734
RAM 1742/3853MB (lfb 205x4MB) cpu [38%,3%,2%,1%]@1734 EMC 28%@1600 AVP 7%@80 GR3D 99%@998 EDP limit 1734
RAM 1743/3853MB (lfb 207x4MB) cpu [20%,15%,3%,0%]@1734 EMC 29%@1600 AVP 7%@80 GR3D 99%@998 EDP limit 1734
RAM 1742/3853MB (lfb 207x4MB) cpu [27%,13%,6%,0%]@1734 EMC 29%@1600 AVP 2%@80 GR3D 99%@998 EDP limit 1734
RAM 1742/3853MB (lfb 207x4MB) cpu [32%,9%,2%,0%]@1734 EMC 28%@1600 AVP 1%@80 GR3D 99%@998 EDP limit 1734
RAM 1742/3853MB (lfb 207x4MB) cpu [13%,17%,9%,2%]@1734 EMC 28%@1600 AVP 2%@80 GR3D 99%@998 EDP limit 1734
RAM 1742/3853MB (lfb 207x4MB) cpu [29%,12%,4%,1%]@1734 EMC 28%@1600 AVP 8%@80 GR3D 99%@998 EDP limit 1734
RAM 1743/3853MB (lfb 207x4MB) cpu [30%,13%,5%,3%]@1734 EMC 28%@1600 AVP 8%@80 GR3D 99%@998 EDP limit 1734
RAM 1743/3853MB (lfb 207x4MB) cpu [22%,6%,13%,1%]@1734 EMC 28%@1600 AVP 6%@80 GR3D 99%@998 EDP limit 1734

Have you disabled USB_AUTOSUSPEND ?

echo -1 > /sys/module/usbcore/parameters/autosuspend

Can be permanently suspended at boot time with:

sudo sed -i ‘$s/$/ usbcore.autosuspend=-1/’ /boot/extlinux/extlinux.conf

Hi Patouceul,
Before I disable,I cat the data as follows.

root@tegra-ubuntu:~# cat /sys/module/usbcore/parameters/autosuspend
-1

So,do I need to input the second command ?

If

cat /sys/module/usbcore/parameters/autosuspend

shows -1 before you issue the first command, then probably it is is already disabled and you dont have to try the permananent disable method. Maybe your extlinux.conf already has this line or some startup script disabled this at linux boot.

Hi Patouceul,

With “-1”,it is still behave that one of camera crash.
The GR3D is 99% keeping some time when one of camera crash.But before this,the GR3D is changing.

How do I analyse it?

THANKS

Could you attach the output of command

sudo dmesg

? This might help to find out what has happened.

Using :

sudo dmesg --follow

after starting your application, you should be able to see error messages as problem happens.

Hi Patouceul,

Please look the first descriptors.
The SUDO DMESG is same with the /var/syslog or /var/kernel.log.We found the problem at 15:52:15 in the picture.
But how ?
How can we avoid it ?

THANKS.

Just a thought on this…USB re-enumerates if a device is disconnected/reconnected…or if power blinks off/on. Two cameras no doubt consume more power than one…is your HUB powered or unpowered? If the HUB is not powered, try one that is…an external power source might help with marginal power delivery.

linuxdev,

Our carrier board have 2 USB2.0 ports off USB1_D_N/P and USB2_D_N/P.USB0_D_N/P as the OTG with USB0_VBUS_DET powered.There doesn’t have an external HUB.And every USB port has a independent power of 5V.

Except the USB0_VBUS_DET,the kernel module of TX1 has not other power singal for USB.


The message at the time you mentioned is a HUB reset. I suppose that because the issue only occurs after a long period of time it would be difficult to record a momentary power drop to the USB, but perhaps if you have a storage scope with the ability to trigger recording based on a slight deviation of the 5V to USB it might show something at the moment of HUB reset (better yet is if the scope is MSO and you can see the USB traffic surrounding any trigger on 5V dipping).

An alternate experiment would be if there is any way you can remove the 5V power from the camera side and supply that 5V power independently from an outside source to see what happens…perhaps the HUB reset would happen anyway, but reset would never occur if it is power related.

Hi linuxdev,

At the first,we have test the power with the ability to trigger recording based on a slight deviation of the 5V to USB.

You look the second picture Posted 02/16/2017 08:56 AM. We add a LDO module(19.4V to 5V) and a USB2.0 port of USB2_D_N/P on the carrier board P2597.When we run two cameras,the built-in USB port of USB1 still appears the same phenomenon randomly with the two USB ports.

I will try agin with external 5V power.But it is small probability for the 5V power.

THANKS

Hi linuxdev,

I test it with external 5V power to USB.One port is still disabled by hub,re-enabling…

linuxdev,
If is it beacause the two ports are on the same BUS or the rate speed ?

You have custom hardware, but typically what a HUB does is put a transaction translator (TT) in place to work between two different speeds of bus. The best is the multi-TT (there may be a case where a single TT helps, but this is not ideal).

A multi-TT on a HUB means each port of the HUB has hardware which will run at the maximum speed of the device plugged into that port, and if the speed is lower, the rest of the HUB will continue to run at its maximum speed.

In a case where a slower device is plugged into a USB bus and no TT is present, then the whole bus will slow down instead of just the one device. Only if all devices on that non-TT USB bus are the same speed would full performance be reached.

I do not know if your D_P/D_M have TT’s, but if they both run the same speed and that speed is the maximum of the bus, then you would not need a TT.

My thoughts on power delivery were because the amount of power available for delivery depends on whether the port is in USB3 mode or not. Earlier @Honey_Patouceul mentioned auto-suspend, which is the software half of host-provided power delivery. With auto-suspend disabled, and with known constant power able to continuously power your devices without question, you can probably rule out the USB layer being directly responsible (there could still be signal integrity issues from an intermittent noise source, but the basic function of USB is mostly ruled out as a cause). Power blinking would cause re-enumeration like what you see. Auto-suspend would also cause this if one of the devices does not correctly deal with low power mode and suspend.

On the software side enumeration takes place for all devices whenever a device is removed or added to the tree of devices using that root HUB. It isn’t guaranteed, but it seems like a driver has stopped and started…whether the driver stopped because of a physical disconnect (not likely after removing auto-suspend and guaranteeing power deliver) or a software issue would not matter to the USB hotplug (software seems more likely)…re-enumeration would occur.

There weren’t any messages to indicate which specific device caused the re-enumeration, so it is difficult to track further. If looking at hardware delivery as good, and if we consider that this part of USB has lots of use (obvious bugs were already fixed long ago), and if we consider that the cameras work for awhile before failing, odds increase that the issue is noise on the USB data lines. A USB analyzer could tell you if there was some sort of data error, but you’d still need to know the source of the error…I’m thinking it isn’t your camera, nor is it the Jetson…it’s reasonable to check the data lines for their physical layout as to how susceptible they are to noise. Note that if you did find a USB error via logic analyzer you might at least narrow it down to inspecting that particular device’s trace layout.

My custom hardware consult from P2597_B04_Concept_schematics as follow picture.
The D_P/D_M signal of port1 and port2 are all from TX1 module.

Viaing lsusb -t,I can look the root hub be loaded on bus1/2/3 when I just add a keyboard.So,I think should the transaction translator (TT) be on the TX1 module?

linuxdev,

I think the TT you said is USB-HUB Chip.
OK?

THANKS