ttyUSB ftdi serial ports stop responding sporadically on Orin Devkit

Hi,

Problem
FTDI connected devices (pump/valves) stop responding sometimes.

Platform
Jetson AGX ORIN dev kit
Linux version 5.10.65-tegra

Setup
I mostly use Pyserial to handle serial communication.
I connect by path, i.e. /dev/serial/by-path/platform-3610000.xhci-usb-0:4.1:1.0-port0
Last time it happened, I ran screen and tried getting the device to respond, without any luck though. My conclusion is that pyserial is innocent, and usbserial or ftdi_sio is the problem.
There are three valves and one pump connected at 9600 baudrate, and it can happen to any of them.

I’ve tried using a logic analyser to see what happens. There I could see the response on the line, but it never reaches the tty. So the device is still available, I can connect to the ttyUSB, but no responses are read back. Sometimes it helps closing/reopening the port.

I cannot see anything suspicious in dmesg when it occurrs.

Should I try upgrading the operating system?

Best regards,
Fredrik Wigert

hello fredrik.wigert,

please see-also [How-To] session of Jetson AGX Orin Developer Kit User Guide - How-to | NVIDIA Developer.
are you able to see debug uart with USB micro-B port? its default baudrate is 115200/8n1,
thanks

Yes, I can login over debug UART. Will there be more logging to see there than in dmesg?

hello fredrik.wigert,

UEFI and OS would have their debug logs as well as the console on SBSA UART, i.e. /dev/ttyACM1.

I don’t understand how this relates to my original problem.
Care to elaborate?

hello fredrik.wigert,

please share the hardware connections, how you setup your FTDI devices?

Well, I use the following hardware:
3x RVM Microfluidic Electric Rotary Valve - Advanced Microfluidics
1x Tecan Cavro XLP 6000 Pump
The cavro pump comes with a hub which converts USB to RS232 via a ftdi chip. The valves have ftdi chips onboard, and have usb connections on them.

They are hooked up to the various usb ports on the devkit:
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.3:1.0-port0”
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.1:1.0-port0”
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.4:1.0-port0”
“/dev/serial/by-path/platform-3610000.xhci-usb-0:4.2:1.0-port0”

Does this help?

This is just a minor possibility, but does the Jetson share the same power source as the valve? If so, then there is a small chance that the failure is related to power fluctuations. The Jetson is very picky about power quality.

Serial console logs which are showing from just before the failure and up into the failure could be very important. An example is if it showed some sort of power rail log.

Also, RS-232 is really a physical interface to serial data. Is this really a DB-9 connector somewhere with a balanced pair being converted to USB? Or is it just USB serial conversion without the DB-9 and balanced pair?

Hi,

They all have their own power supplies.
I was hooked up on the debug UART and triggered the error. Nothing came up in the log, dmesg or journalctl.

Regarding RS-232 the pump has a “hub” which converts USB to RS-232 (DA-15 connector), whereas the valves have an on board ftdi chip, so they probably only convert USB to UART.

I attach dmesg.

Any further tests I can do?
dmesg.txt (74.0 KB)

I don’t know what else to test, but the dmesg log did show a kernel OOPS which I think is unrelated to the UART. It is conceivable though that the other driver causes a problem during that OOPS. Don’t know.

If I understand the problem correctly, to Jetson, your device is connected over USB. So, if you are suspecting some issue with USB communication on Jetson side, then need to connect Logic analyzer. Dmesg you have attached has no clue and the stack dumps are harmless and you can ignore.

I now have 6 usb/serial devices connected to the Jetson now. I use a USB hub to achieve this.
All devices show the same problem. The thing i did with logic analyzer was to ensure that the devices actually do send a response. They do, it is Jetson that fails to read them.

dmesg for the newly added devices:

[ 3179.621559] usb 1-4.2: new high-speed USB device number 7 using tegra-xusb
[ 3179.734647] usb 1-4.2: New USB device found, idVendor=2109, idProduct=2817, bcdDevice=90.34
[ 3179.734668] usb 1-4.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3179.734679] usb 1-4.2: Product: USB2.0 Hub
[ 3179.734688] usb 1-4.2: Manufacturer: VIA Labs, Inc.
[ 3181.109774] usb 1-4.2.1: new full-speed USB device number 8 using tegra-xusb
[ 3181.276349] usb 1-4.2.1: New USB device found, idVendor=0483, idProduct=a080, bcdDevice= 1.00
[ 3181.276400] usb 1-4.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3181.276429] usb 1-4.2.1: Product: USB Virtual Serial Port
[ 3181.276451] usb 1-4.2.1: Manufacturer: CoolLED
[ 3181.276473] usb 1-4.2.1: SerialNumber: 207539624633
[ 3181.417617] usb 1-4.2.2: new full-speed USB device number 9 using tegra-xusb
[ 3181.580844] usb 1-4.2.2: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[ 3181.580894] usb 1-4.2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3181.580919] usb 1-4.2.2: Product: CP2102 USB to UART Bridge Controller
[ 3181.580938] usb 1-4.2.2: Manufacturer: Silicon Labs
[ 3181.580957] usb 1-4.2.2: SerialNumber: 0001
[ 3181.718176] usb 1-4.2.4: new full-speed USB device number 10 using tegra-xusb
[ 3181.886327] usb 1-4.2.4: New USB device found, idVendor=0403, idProduct=6015, bcdDevice=10.00
[ 3181.886383] usb 1-4.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 3181.886409] usb 1-4.2.4: Product: Fast USB to RS-232 Adapter
[ 3181.886430] usb 1-4.2.4: Manufacturer: FTDI
[ 3181.886449] usb 1-4.2.4: SerialNumber: DM6VZ56J
[ 3181.896541] usb 1-4.2.4: Detected FT-X
[ 3181.905113] usb 1-4.2.4: FTDI USB Serial Device converter now attached to ttyUSB3
[ 3181.959819] usbcore: registered new interface driver cp210x
[ 3181.959858] usbserial: USB Serial support registered for cp210x
[ 3181.967060] usb 1-4.2.2: cp210x converter now attached to ttyUSB4
[ 3181.967125] usbcore: registered new interface driver cdc_acm

So these use different chips, not all FTDI as before, so the problem is not tied to ftdi_sio.

lsmod | grep ftdi
ftdi_sio 53248 3
usbserial 40960 8 ftdi_sio,cp210x

It feels to me like usbserial has issues, becaues it’s the common component that all these devices rely on.

modinfo usbserial
filename: /lib/modules/5.10.65-tegra/kernel/drivers/usb/serial/usbserial.ko
license: GPL v2
description: USB Serial Driver core
author: Greg Kroah-Hartman gregkh@linuxfoundation.org
depends:
intree: Y
name: usbserial
vermagic: 5.10.65-tegra SMP preempt mod_unload modversions aarch64

Is there a more recent version of usbserial I can test?

So I have come up with a “workaround” that might be a hint to the root cause.
If I run the command usbreset for a device, then it will work again. Closing and reopening the port will still make it stop work, but it seems like usbreset always makes it work again.

What happens when I run that command? I haven’t dug into it yet.

Correction: Sometimes Silicon Labs CP210x UART Bridge cannot be reset, it just hangs when I try.

Where did you get “usbreset”? If it is what I think it is, then it uses an ioctl call (everything in UNIX/Linux is a “file”, and can be read/written like a file, but device special files and pseudo files are really an extension of a driver, and they offer custom calls for custom purposes, an I/O Control command) for “USBDEVFS_RESET” to tell the controller to initialize the device (same as fresh plugin). Example:
https://askubuntu.com/questions/645/how-do-you-reset-a-usb-device-from-the-command-line

Thank you for that!
That exact thread mentioned usbreset command, which I tried out and managed to get a port working again.

It looks like it actually sends a reset signal to the usb device, interesting.

We have another unit incoming, and it will be very interesting to see if it behaves any better than this one.
It also loses contact with our GigE vision camera periodically, but that is probably not related.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.