Jetson AGX Orin Exar driver issue

Hello we are running into an issue running a software controlled motor which uses the Exar USB driver (Software Drivers - MaxLinear) on the AGX Orin. The motor in particular is a Teknic Clearpath-SC motor. The motor has its own cpp library and uses the Exar USB driver mentioned above. The details can be found here (Download Teknic software, manuals, drawings, etc.) under ClearPath > Software > Linux_Software.tar.gz.

We are able to compile and install the cpp library as well as the Exar USB driver, we are then able to run their HelloWorld example, which starts a connection with the motor and prints some benign information. The issue occurs if we try to run the same binary again, the program fails to connect to the motor claiming that it is already busy.

It may sound like this is a problem with the motor and not the jetson, however the issue seems to be isolated to the AGX Orin. We were able to follow the above procedure with no issue when using an AGX Xavier running Jetpack 5.0.2. A Teknic representative also mentioned that a different customer was having similar issues with the AGX Orin running 5.x, but had no issue with a Xavier NX running 4.x. This strongly suggests an issue with the AGX Orin. In our tests we also reflashed the Orin with versions 5.0.2, 5.1, and 5.1.1 to no avail.

We would be grateful for any help you can provide.

To add some info which may be a symptom of the issue. On the Xavier, running lsusb prints the following
Xavier

whereas on the Orin it prints only
Orin

Hi,
For information. So the application works in in the first run, and then always fails?

correct

Do you run the program as root/sudo? Or as a regular user?

All the previous tests were run as a regular user, but I just tested sudo and it makes no difference.

This is kind of tangential, but why does lsusb not print the description for the device? Shouldn’t it fetch it from here: USB Devices

Hi,
We are checking this. Will update if there is further finding.

If you monitor “dmesg --follow”, then there should be a message upon USB plugin. If and only if the device has no error will lsusb show the device, but plugin will mention plugin even if there are errors (assuming the signal quality is not so bad that no signal is seen). Even if no driver is available the device plugin’s dmesg and lsusb would list the device when signal quality is even minimal.

This is the output of dmesg when plugging in:


It looks clean to me, and even identifies the manufacturer and product, so it’s strange that the line on lsusb is empty.

Finally, here is the output from dmesg, on the first run of the helloworld program:

And here is the second run where it fails:

I can’t say much about that, but it does appear it is related to the CDC UART driver. What I saw on the web which is quite old and only very loosely associated with this looks like there might have been a UART driver bug many years ago. I’m wondering if you’ve installed the driver from something which might be rather old? If this is the case, then perhaps the CDC ACM driver could be updated while leaving the rest as is.

Before and after running this (or attempting to), what do you see from “lsmod” (I’m curious if the driver shows up both before and after an error)?

Also, what is the output of "zcat /proc/config.gz | egrep 'USB.ACM'?

We have tried different versions of the cdc_xr_usb_serial driver, but we are currently using the latest version (release date: September 2021) with the comment: The driver has been tested on various kernel versions from 3.6.x to 5.11.x.

The strange part is that it works flawlessly on Jetson Xavier JP 5.0.2. Which to me indicates that it is not a problem with the driver, but with the Orin.

Pre failure:
lsmod

ls /sys/bus/usb/drivers/cdc_xr_usb_serial/ showing the device binded to the correct driver
Screenshot 2023-04-18 130506

zcat /proc/config.gz | egrep 'USB.ACM'
Screenshot 2023-04-18 130646

Post failure:
lsmod

ls /sys/bus/usb/drivers/cdc_xr_usb_serial/ still bound
Screenshot 2023-04-18 132426

zcat /proc/config.gz | egrep 'USB.ACM'
Screenshot 2023-04-18 132510

Hi Dane,

Just for our own information, can you confirm whether there is an active plan to get to the bottom of this bug, or if it has been put on the back burner? This is just so we can adapt our development strategy accordingly.

Thank you

I was actually hoping to see the lsmod as a text file (e.g., “lsmod 2>&1 | tee log_lsmod.txt”), that way I can use “diff” on it. Screenshots with that much output are hard to read reliably, but it does look like cdc_acm is correctly loaded (although I did not diff the list as a whole).

I am curious about one other detail though. Do you see anything from:
egrep -i -R '(cdc|acm)' /etc/udev/*

If a file shows up there, then you might consider attaching it to the forum (if the forum doesn’t like the file you could rename it with a “.txt” file name extension).

egrep -i -R '(cdc|acm)' /etc/udev/* has no output

Ok, that simplies it. udev can modify “standard” behavior, unless that particular driver needs udev rules to identify the hardware before it can “do the right thing”. It is quite common in custom hardware, which uses standard drivers (and CDC ACM is standard), that some modification of file naming is made in “/dev” so that the custom software knows which particular device (UART in this case) is used. For those cases udev basically creates an alternate name for software which depends on the alternate name (and would fail to find some or all of the hardware in that case even if the driver loads). I don’t see any indication that this is done with this hardware, but it is worth keeping an eye out for.

Maybe you are onto something. This is the output of egrep -i -R '(cdc|acm)' /etc/udev/* on our Jetson Xavier which works flawlessly (please note that this machine is in active use so this could be completely unrelated)

70-snap.snapd.rules (61.8 KB)

This content might not be owned by a package, and so this might not show anything, but what is the result of these commands (the file might be a symbolic link, and if it is, then the file actually pointed at by the link is of interest, and either/or of the original/sym link might not be owned by a package, but if it is the package name is of interest):

ls -l /etc/udev/rules.d/70-snap.snapd.rules
# If this is a symbolic link, substitute the full path for the link, otherwise:
dpkg -S /etc/udev/rules.d/70-snap.snapd.rules

Looks like that file is generated (rather than being provided directly), and I’d like to find out where the configuration comes from (on the unit which works). Also, on the unit which works, did you install any kind of driver or other file on the unit? Was that same software not installed on the non-working unit? Was there any kind of configuration command or setup involved in the working unit which might not have been applied yet to the failing unit?

In the case of a configuration or other command I’m thinking that what is generated might differ if the installation and/or other software was installed while the unit was connected via USB, versus if not. I am trying to find any clue as to what was supposed to be installed, and whether or not the USB might have been connected (or not connected) at the time of the software install on the working unit. Anything which might suggest a difference between how that file was generated on one Jetson versus the other is of interest.

Incidentally, there might be USB events which regenerate this file. That means that we might be able to test by temporarily copying the 70-snap.snapd.rules from the working unit as a test (first back up the original), but if something causes generation of a replacement file, then it might go away.

I don’t know all of what snapd does, but it is generally started by systemd during boot (which in turn is PID 1, the only process the kernel starts, and known as “init”). There are a lot of systemd configurations possible, sometimes based on only enabling or disabling something; at other times you have to install software to make enable/disable possible. Yours is obviously related to ModemManager, and so it gets worse: I don’t know what makes the particular device visible to snapd, and once snapd is running, I don’t know which snapd service or file works with that particular USB device. It’s going to be a bit of a detective story to figure that out.

On the working unit this becomes more useful (I’m just poking around until seeing something which might say where the command comes from):

  • find /etc/systemd -iname '*snap*'
  • `egrep -R -i ‘ModemManager’ *
  • systemctl list-units | egrep -i ‘(modemmanager|uart|cdc|acm)’
  • Is there any file on the working Jetson in “/etc” which is named after the product?
  • snap list

Note that udev content will work just by plugging in a USB device (or unplug/replug), but other content might require a systemd startup (which might also rewrite a configuration file).

Incidentally, for the “/etc/udev/rules.d/70-snap.snapd.rules” file, if it is a regular file (on the failing unit) and not a symbolic link, then you temporarily disable it via “sudo gzip /etc/udev/rules.d/70-snap.snapd.rules”, copying that file from the working unit. Then to get rid of that copied version and put the original back, just “sudo gunzip /etc/udev/rules.d/70-snap.snapd.rules” (compress, replace original, then put back by uncompressing original). If the file is a symbolic link, then you have to use link commands. You can try replacing that file on a running Jetson and see what happens. Probably you’d plug the USB item in after those rules are in place; if it works, then reboot and see if the file is still present and if things still work (but beware something might trigger removing your edited file at any moment).

PS: I might not be around much for a week, so I might be slow to reply, but explore how any “/etc” configuration might differ between the working and non-working unit.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.