Jetson TX2 NX UART Errors

[Andre_Antunes]

Hello,

We are trying to use a Jetson TX2 NX to communicate with a ProfiCNC Cube Orange Flight Controller through the MAVLink 2 protocol on a UART port (UART1, pins 203 and 205, /dev/ttyTHS2), using 500000 baud rate (other rates were tested, from 9600 all the way to 921600).
Our software and hardware has been validated to work on a Jetson Nano, however, due to the need of a more robust and performant platform, it was chosen to adopt the TX2 NX.

We have changed the .dts file on the TX2 NX to enable the UART1 port and disabled every other UART interface, as we don’t need them.

.dts File Used: a.dts (238.2 KB)

The TX2 NX is running on a custom carrier which houses both the Jetson and the Flight Controller, and the signal on the Tx and Rx lines has been observed in an oscilloscope and found to be a perfectly square wave on both lines (we previously had issues where the signal coming out of the 1.8V to 3.3V level shifter was being distorted, however that is not the case this time).

The problem we are facing is that we have constant timeouts and data transfer errors, which occur with the TX2 NX but don’t occur with the Nano, while running the same software, on the same carrier board.

[linuxdev]

I don’t know if I can say anything particularly useful here, but if you monitor “dmesg --follow”, do you see any logged output during the problem? Is this a random failure, or is it something you can repeat? If you put the NX’s TX/RX (I am assuming you are not using CTS/DTS) in loopback mode with TX and RX directly wired together, do you still have data issues?

[mike_nv]

@Andre_Antunes please share the dmesg log for eventual kernel errors or warnigns.

[Andre_Antunes]

Hi @linuxdev this was actually quite useful! Unfortunately we still did not make it work, but gave us some additional insights.

Using loopback on the TX and RX pins, it’s possible that the TX2 NX is injecting noise into the UART TX line.

Let me try to explain with the most detail I can provide.

TEST 1:

I tested a simple command on both the Jetson Nano and Jetson TX2 NX, echo "hi" > /dev/ttyTHSx. On the Jetson Nano that is ttyTHS1, on the TX2 NX it’s the ttyTHS2.

Initially we plug a cable from the TX to the RX pins, and sent commands to the /dev/ttyTHS . This was generating an infinite spam on the cat terminal. Looking at the answer from Kristina at Unexpected results testing serial loopback using echo and cat - Unix & Linux Stack Exchange, we found the issue:

Since this serial port is looped back to itself, here is what happened after running echo "hi" > /dev/ttyS1 :

1. The echo command appends a newline to the end of the message by default, so “hi” + LF is sent out to /dev/ttyS1
2. Because “onlcr” was set, the serial device converted the LF to CRLF so the physical message sent out the Tx line was “hi” + CRLF
3. Because “icrnl” was set, the physical messaged received on the Rx line converted the CR to LF. So the message outputted by ‘cat’ was “hi” + LFLF.
4. Because “echo” was set, the message received on the Rx (“hi” + LFLF), was then sent back out on the Tx line.
5. Because of onlcr, “hi” + LFLF became “hi” + CRLFCRLF.
6. Because of icrnl, “hi” + CRLFCRLF became “hi” + LFLFLFLF
7. Because of echo, “hi” + LFLFLFLF was then sent out the Tx

And so on…

In order to fix this problem, I ran the following command:

stty -F /dev/ttyS1 -echo -onlcr

Disabling “echo” prevents an infinite loop of messages and disabling “onlcr” prevents the serial device from converting LF to CRLF on output. Now cat receives one “hi” (with a single newline!) for each time I run echo .

Afterwards, we were able to send single messages thought the UART. On the pictures bellow you may find prints of the tests on the executed :

Jetson Nano (works fine…):

Jetson TX2 NX (it does not generate infinite spam, but it prints a crazy amount of ‘command not found’):

@mike_nv , looking at kern.log, as far as we know, there don’t seem to be any issues.
kern.txt (92.9 KB)

Note:

During the 1st test we realised that if the hardware loopback was connected while the Jetsons were turned off, the Jetson Nano would boot normally, but TX2 NX would not boot. We would always need to unplug the connection, boot the TX2 NX, and plug it back again for the test.

TEST 2:

Using a logic analyser to probe the lines, the difference is clearly visible.

Jetson Nano:

Jetson TX2 NX:

With the Jetson Nano we can see the expected behaviour, the Jetson writes the “hi” text on the UART line and goes silent, on the TX2 NX it writes “hi” has a small wait and then introduces more data on the line which it was not commanded to do.

[mike_nv]

Hi @Andre_Antunes, would you experience the same issue when using minicom or screen?

[Andre_Antunes]

I can test this with minicom, but i would assume the result to be the same as the data is actually being sent and can be seen in a logic analyzer!

Which “side” would you like me to test in minicom, the TX or the RX?

[mike_nv]

Replace the cat command by a serial terminal utility (e.g minicom or screen). I want to understand where the “appended message” is coming from.

[Andre_Antunes]

Hi, i have tested with minicom and the result appears to be the same, screenshot below:

[mike_nv]

Thank you. I will review the screenshots and discuss internally the matter and come back to you soon.

[Andre_Antunes]

To add a bit more information, we soldered some cable leads from UART0 to the Flight Controller we have been testing and the communication works perfectly!
However, we need the UART1 port, as the UART0 port is used elsewhere.

[linuxdev]

I have not finished reading all reply, but wanted to start with the error of “command not found”. If you are using a serial device which is tied to serial console, then you can expect this (or if you accidentally use a valid command, e.g., “ls”, then it might spit back some unexpected information).

Devices in “/dev” are not real files, they are in fact a kernel driver pretending to be a file (normal file read/write ops, in combination with IOCTL calls, make it possible to work with the kernel driver as if it is a file). You can expect that any “/dev/ttyS#” is the exact same UART hardware as the corresponding “/dev/ttyTHS#” if and only if the “#” is the same. You shouldn’t use both drivers simultaneously, and if you talk to a ttyTHS which has the ttyS bound to serial console, then this is exactly what would happen.

The reason for using a ttyS for serial console is that the earlier boot code (prior to the Linux kernel running) has only the legacy driver, and does not have the ability to use the DMA-capable ttyTHS driver. During boot there would be a loss of continuity of serial console logs if it were to switch from a ttyS to a ttyTHS, and so serial consoles all use the ttyS.

If you run this command, then expect anything with “group” of “dialout” to be just a serial UART, but anything with a “group” of “tty” to be used for serial console:
ls -l /dev/ttyS* /dev/ttyTHS*

Also, note that a different tty can be used for serial console on different Jetson devices. Is there any accidental use of a UART which is also a console? [EDIT: I see you got this working, so the information might be useful to someone, so I’ve left the answer in place.]

FYI, control of UART setup is initially via the device tree. The PINMUX spreadsheet for the particular device is how you might set this up to a different default. However, the software which uses this as a serial console is a separate answer…one change might be needed in U-Boot, and another for Linux itself. You might want to post another question specific for that exact hardware and L4T release (see “head -n 1 /etc/nv_tegra_release”) on disabling or changing serial console to make a given port available (if the port is not used with serial console, then the answer is simpler).