Problem
FTDI connected devices (pump/valves) stop responding sometimes.
Platform
Jetson AGX ORIN dev kit
Linux version 5.10.65-tegra
Setup
I mostly use Pyserial to handle serial communication.
I connect by path, i.e. /dev/serial/by-path/platform-3610000.xhci-usb-0:4.1:1.0-port0
Last time it happened, I ran screen and tried getting the device to respond, without any luck though. My conclusion is that pyserial is innocent, and usbserial or ftdi_sio is the problem.
There are three valves and one pump connected at 9600 baudrate, and it can happen to any of them.
I’ve tried using a logic analyser to see what happens. There I could see the response on the line, but it never reaches the tty. So the device is still available, I can connect to the ttyUSB, but no responses are read back. Sometimes it helps closing/reopening the port.
I cannot see anything suspicious in dmesg when it occurrs.
They are hooked up to the various usb ports on the devkit:
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.3:1.0-port0”
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.1:1.0-port0”
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.4:1.0-port0”
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.2:1.0-port0”
This is just a minor possibility, but does the Jetson share the same power source as the valve? If so, then there is a small chance that the failure is related to power fluctuations. The Jetson is very picky about power quality.
Serial console logs which are showing from just before the failure and up into the failure could be very important. An example is if it showed some sort of power rail log.
Also, RS-232 is really a physical interface to serial data. Is this really a DB-9 connector somewhere with a balanced pair being converted to USB? Or is it just USB serial conversion without the DB-9 and balanced pair?
They all have their own power supplies.
I was hooked up on the debug UART and triggered the error. Nothing came up in the log, dmesg or journalctl.
Regarding RS-232 the pump has a “hub” which converts USB to RS-232 (DA-15 connector), whereas the valves have an on board ftdi chip, so they probably only convert USB to UART.
I don’t know what else to test, but the dmesg log did show a kernel OOPS which I think is unrelated to the UART. It is conceivable though that the other driver causes a problem during that OOPS. Don’t know.
If I understand the problem correctly, to Jetson, your device is connected over USB. So, if you are suspecting some issue with USB communication on Jetson side, then need to connect Logic analyzer. Dmesg you have attached has no clue and the stack dumps are harmless and you can ignore.
I now have 6 usb/serial devices connected to the Jetson now. I use a USB hub to achieve this.
All devices show the same problem. The thing i did with logic analyzer was to ensure that the devices actually do send a response. They do, it is Jetson that fails to read them.
dmesg for the newly added devices:
[ 3179.621559] usb 1-4.2: new high-speed USB device number 7 using tegra-xusb
[ 3179.734647] usb 1-4.2: New USB device found, idVendor=2109, idProduct=2817, bcdDevice=90.34
[ 3179.734668] usb 1-4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3179.734679] usb 1-4.2: Product: USB2.0 Hub
[ 3179.734688] usb 1-4.2: Manufacturer: VIA Labs, Inc.
[ 3181.109774] usb 1-4.2.1: new full-speed USB device number 8 using tegra-xusb
[ 3181.276349] usb 1-4.2.1: New USB device found, idVendor=0483, idProduct=a080, bcdDevice= 1.00
[ 3181.276400] usb 1-4.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3181.276429] usb 1-4.2.1: Product: USB Virtual Serial Port
[ 3181.276451] usb 1-4.2.1: Manufacturer: CoolLED
[ 3181.276473] usb 1-4.2.1: SerialNumber: 207539624633
[ 3181.417617] usb 1-4.2.2: new full-speed USB device number 9 using tegra-xusb
[ 3181.580844] usb 1-4.2.2: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[ 3181.580894] usb 1-4.2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3181.580919] usb 1-4.2.2: Product: CP2102 USB to UART Bridge Controller
[ 3181.580938] usb 1-4.2.2: Manufacturer: Silicon Labs
[ 3181.580957] usb 1-4.2.2: SerialNumber: 0001
[ 3181.718176] usb 1-4.2.4: new full-speed USB device number 10 using tegra-xusb
[ 3181.886327] usb 1-4.2.4: New USB device found, idVendor=0403, idProduct=6015, bcdDevice=10.00
[ 3181.886383] usb 1-4.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3181.886409] usb 1-4.2.4: Product: Fast USB to RS-232 Adapter
[ 3181.886430] usb 1-4.2.4: Manufacturer: FTDI
[ 3181.886449] usb 1-4.2.4: SerialNumber: DM6VZ56J
[ 3181.896541] usb 1-4.2.4: Detected FT-X
[ 3181.905113] usb 1-4.2.4: FTDI USB Serial Device converter now attached to ttyUSB3
[ 3181.959819] usbcore: registered new interface driver cp210x
[ 3181.959858] usbserial: USB Serial support registered for cp210x
[ 3181.967060] usb 1-4.2.2: cp210x converter now attached to ttyUSB4
[ 3181.967125] usbcore: registered new interface driver cdc_acm
So these use different chips, not all FTDI as before, so the problem is not tied to ftdi_sio.
So I have come up with a “workaround” that might be a hint to the root cause.
If I run the command usbreset for a device, then it will work again. Closing and reopening the port will still make it stop work, but it seems like usbreset always makes it work again.
What happens when I run that command? I haven’t dug into it yet.
Where did you get “usbreset”? If it is what I think it is, then it uses an ioctl call (everything in UNIX/Linux is a “file”, and can be read/written like a file, but device special files and pseudo files are really an extension of a driver, and they offer custom calls for custom purposes, an I/O Control command) for “USBDEVFS_RESET” to tell the controller to initialize the device (same as fresh plugin). Example: https://askubuntu.com/questions/645/how-do-you-reset-a-usb-device-from-the-command-line
Thank you for that!
That exact thread mentioned usbreset command, which I tried out and managed to get a port working again.
It looks like it actually sends a reset signal to the usb device, interesting.
We have another unit incoming, and it will be very interesting to see if it behaves any better than this one.
It also loses contact with our GigE vision camera periodically, but that is probably not related.