Weird behavior with flashing arduino via Jetson

We are trying to flash an Arduino Leonardo board with avrdude 6.2 from a Nvidia Tegra TX2 4GB with ubuntu 18.04 (kernel=4.9.140-tegra, Jetpack 4.3).

The way the arduino leonardo flashing works is that it is restarted to bootloader mode when you open its serial port with baud rate 1200 and set dtr to low (the board identifies it self as a cdc_adm /dev/ttyACM0).

It stays in bootloader mode for 8 seconds and if no flashing command is issued it boots to the user program already flashed.

When trying to flash it using this setup it starts to flash but the device reboots mid way to user program mode.

The flashing works from a pc (x86 architecture) with ubuntu 16.04, from another pc with ubuntu 18.04, and from a Nvidia Tegra TX2 8GB with ubuntu 16.04 (Jetpack 3.3, linux kernel 4.4)

Do you know if there is a gap between the Jetpack versions that can have an influence with flashing?

We suspect the cdc_adm driver might have a difference causing this issue

If the arduino gets that command within 8 seconds, does it still need the command every 8 seconds? Or does it continue flashing until a command or power cycle forces reboot? If starting the command within 8 seconds does not require continued commands within every 8 second period, then I would think it is an issue with the arduino, although possibly complicated by anything causing the arduino to think the serial port disconnected.

Regarding disconnects, if you monitor “dmesg --follow” during the time you are flashing the arduino, do you see any log output, especially related to the serial port? Does the file “/dev/ttyACM0” go away when flash reboot hits?

Hi,
I am urielom8ug teammate, I upload here the flashing output on the left side of the pictures and the dmesg log on the right side of the pictures, 3 pictures are from working environment (x86) and 3 of them are flashing from Jetson TX2 [4GB] with Jetpack 4.3 that don’t work.
You can see on the “working pics” that after the reset to flashing mode the flashing is started and after the flash success there is another reboot where the arduino come back with the same ttyACM number.
On the “not working pics” you can see that after the first reset to flashing mode the flash running get stuck and then the board make a reboot to data mode in different ttyACM number, and the flashing stays stuck in the same place.
I hope the issue is more clear now.
Thanks!
Not working 1


Not working 2

Not working 3

Working 1

Working 2

Working 3

I have not used Arduino with this before, but found an interesting URL which might be related to the issue:
https://arduino.stackexchange.com/questions/38012/major-difference-between-dtr-signal-and-rts-signal

Normally, on a TX2, a serial UART would use a TX and RX and ground for the data, and this would be universally the same across all systems. Options may include hardware flow control via CTS/RTS, and many people use serial UARTs with no hardware flow control, and then leave CTS/RTS unconnected. So long as the UART is told to use these (and is wired for these), then “normally” this would work. Having CTS/RTS succeed in one direction only tends to imply one UART has this enabled, but the other UART does not. In your case, with DTR, the implication is that you probably have to use CTS/RTS on each UART pair, plus some other setup with DTR. Perhaps this is just missing DTR in one half of the setup.

I am just guessing, but it seems likely that either the Jetson side software is not aware of a CTS/RTS and DTR requirement, or else one of the wires for a function is not connected/implemented. Other than a logic analyzer I’m not sure what to tell you about testing this. If you do have a logic analyzer, then you can probably go straight to the testing of CTS/RTS/DTR and find the Jetson side as not reacting to one of those.

The Arduino we are using is based on atmega 32u4 (arduino leonardo) which provides a USB connection, this usb connection is detected by the jetson correctly and it is using cdc_adm to provide a serial device.

Now in order to flash the Arduino, we open the serial device at baudrate 1200, this makes the arduino to reboot to bootloader where it will stay for 8 seconds.
See https://github.com/sudar/Arduino-Makefile/issues/30

All this seems to work on the Jetson, the thing that seems to be broken is that when the device re-enumerates in usb as bootloader mode (same vid, different pid in usb) it will receive a different ttyACM (i.e. before reboot it was ttyACM0 and then it will become ttyACM1) and it will fail to enter flashing mode, sometimes it will stay the same ACM and midway flashing it will reboot.

All this happens with JP4.3, in JP 3.3 it works 100% of the times.

To us it seems there is a change in udev/cdc_adm/etc that is breaking this

Likely you are running into something related to udev differences between Ubuntu 16.04 and Ubuntu 18.04. JetPack3.x is Ubuntu 16.04, while JetPack4.x is Ubuntu 18.04. It probably is not a great choice, but one workaround might be to redetect and reopen the device special file name as USB is re-enumerated, allowing a new file name to substitute for the original (probably not a good way to deal with it).

For a bit of background when describing a better way to deal with this, it is standard that when a USB device reconnects enumeration is performed, that there is nothing requiring a disconnect/reconnect of a specific device to be identified by the same original enumeration, although it seems reasonable to not increment numbers if there was no actual hardware change. Indeed, USB does not know the reconnect is the same device…for all USB knows, this could be a new device which happens to be the same model of hardware (physical reconnect and electrical reset are the same so far as USB is concerned).

More important information for you is probably about understanding “udev”. Every time a hot plug device event is generated udev has an opportunity to run based on the device ID or other criteria. “udev” can act as a custom constructor to device special file naming and creation. This applies to you because a device re-enumerating, regardless of whether it is from the device resetting or the cable being physically unplugged/replugged, generates such an event, and you need to intercept the default naming scheme and update it. One of the actions a udev rule can take is to create an auto incrementing device special file naming convention, or to hold the file name to some specific name without increment.

As an example, if udev finds a serial number or other specific device ID information, then the “/dev” name picked can be based on details of the device or just its basic class. Symbolic links could be added in “/dev” via udev, thus providing multiple names for a device. An example is that in “/dev/input” (input devices like mice) various mouse/keyboard related “event” files are generated depending on enumeration order. There will also be symbolic links in “/dev/input/by-id/” and “/dev/input/by-path/” which point at the actual device special file in “/dev/input/”, but allow finding that same device by different names (depending on whether you are interested in the driver path or the common description of the device). You could presumably edit udev rules for your specific file/device which takes precedence over defaults and forces the “/dev” file to be named what you want it to be, or you could create symbolic links with a constant name while the actual file does its own thing with incrementing numbers at each re-enumeration.

I am going to suggest that this is likely an issue related to differences between Ubuntu 16.04 and Ubuntu 18.04, and that a udev rule might work around the issue. I doubt this is an issue specific to Jetsons. Having the file name stay constant is not necessarily going to be the only step, and I’ll explain why.

If you open a file via symbolic link, and the symbolic link is broken and reconnected to a new hard link, then it is possible your software will still have to close and open that file again…the name would be the same, but if your flash software opens once and does not reopen after re-enumeration, then it might still fail. If the hard linked device special file is removed and added back in with the same name, then your end software may also need to take steps to re-open. Don’t know, it’d require testing to know for sure.

There is a lot of information on udev over the internet. Debian and Ubuntu documents will probably be directly applicable, and even documents for other Linux distributions will likely not require any customization. Here are some sample URLs:
https://wiki.debian.org/udev
https://opensource.com/article/18/11/udev
https://www.thegeekdiary.com/beginners-guide-to-udev-in-linux/

Just so you know, before you start, use “lsusb” to find details of your device’s IDs. The “ID” can be used to limit lsusb to just your device, e.g., a recovery mode TX2 has ID “0955:7c18”, and so you can do a fully verbose lsusb for just that device, and log it, via this (substitute for your ID):
sudo lsusb -d 0955:7c18 -vvv | tee log_0955-7c18.txt

Some information in a verbose lsusb may be used as an identifier when working with udev.

Also, know that there are system udev files in “/lib/udev/rules.d/”. You should not change those directly. However, there is a local configuration/customization directory, and if you have a file of that name there, then it uses that local configuration version instead. If you copy something from “/lib/udev/rules.d/” into “/etc/udev/rules.d/”, then your new copy takes precedence. If you don’t like that copy, delete it and you are back to the system version. If there never was a system version, then you have a brand new rule file. The numbering/naming convention will be obvious, and you’d pick a number/name which occurs early enough to apply your rule before something else names the device.

Thank you for the very detailed explanation about udev, we use udev rules so we are aware of some of the details but not all of them and one of our trials was to disable our rules, didnt work unfortunately.

Regarding differences between ubuntu 18.04,we thought this too so we ran this on a x64 laptop with 18.04 and it worked correctly so it isn’t a problem with ububtu difference but I agree it seems to be something with udev/kernel module.

Regarding the re-enumeration, we applied some code that detects the new name and sometimes it starts the flashing process but it fails midway.

Can it be that the cdc_acm module deasserts dtr or some other “virtual” control signal?

The catch is this: The order of enumeration, and the items which are enumerated, will change results. To really know if this difference is for 16.04 versus 18.04 you would need to put the same USB HUBs/bridges/devices in. I agree your test was a good test, but it is not definitive as to differences of 16.04 versus 18.04 being an issue, since a guarantee would require replicating all USB.

To reemphasize, I do not know the particular Arduino hardware and have not used it, and do not have any to test on. However, if this is like a lot of USB flash mechanisms, then it is possible that the USB drops out and reconnects during a flash even when working correctly. Don’t know for sure in your case.

I do not know if the cdc_acm even examines dtr. If dtr is examined, I do not know how it is handled. This is quite possibly related to (or causing) the problem, but I could not tell you how. This is one reason why looking at it with a logic analyzer would help. The cdc_acm driver is itself part of the regular kernel and not customized for use in Jetsons, but the cdc_acm for arm64 may still have differences versus in x86_64/amd64.

Understood regarding enumeration, still it acts very differently than other devices where it will keep the old device hanging for a bit enough to get it enumerated as ttyACM1.

Regarding using a logic analyzer to see the DTR, it is a virtual signal so you won’t see it as the arduino is providing a USB device not using a USB to serial bridge.

The thing that baffles us is that it is working with JP3.3 with the exact same device

There is an ability to monitor USB traffic as if it were physically available, and although this may not expose enough to see serial UART, this would probably indicate disconnect and reconnect events:
https://www.kernel.org/doc/Documentation/usb/usbmon.txt

Normally I would use this from Fedora (though not recently), which literally has package “usbmon”, but I am not finding it for Ubuntu. The actual kernel feature for using this is present by default, e.g., see:
zcat /proc/config.gz | grep 'USB_MON'
ls -l /lib/modules/$(uname -r)/kernel/drivers/usb/mon/usbmon.ko

You can probably use this on Ubuntu with wireshark instead. See:
https://wiki.wireshark.org/CaptureSetup/USB#Linux

You will want to answer “yes” to allow superusers to capture (see the above URL):
sudo apt-get install wireshark
…or at least set a user to allow by adding to group “wireshark”.

The “setfacl” in the wireshark URL does not matter on Jetsons which do not have enforced SElinux. However, your desktop PC may require setfacl if you use this from a PC. Also, the debugfs is already mounted on the Jetson, and so you have no need to mount debugfs.

I think the “SerialUSB” will work for your purposes since any keyboard/mouse is just a serial UART, and other than being connected to buttons, this should be no different than your UART. The part which is important is not even the UART traffic, it is the USB disconnect/reconnect/re-enumeration cause, and so I wouldn’t worry if you can’t see UART traffic so long as you find out why it is re-enumerating. Especially important to know is if your other systems which work correctly re-enumerate in the same way, but handle differently, versus whether re-enumeration exists only from this failing case TX2’s release.

My guess is that both working and failing will have the same USB behavior, but udev rules will differ. Watching USB events can verify (or disprove) this.