Nvidia Orin Malformed UART Messages

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
, ,
1

Hello,

I am having trouble with my Orin (and Xavier) devkit machines when I am trying to use the UART interface to read NMEA messages from another board containing an IMU. This issue is only present on the Orin and Xavier devkits I have that are running Ubuntu 20 from JetPack 5.0.2 Runtime as I have Xavier machines which run Ubuntu 18 from JetPack 4.6.3 which do not experience the problem using the same board, wiring and UART pins.

The problem I am facing is the NMEA messages are sometimes cut off or malformed when read by the units running Ubuntu 20. This same problem is not experienced with the Ubuntu 18 units. These units have been flashed using SDK manager from a machine running Ubuntu 18 following the user guide for flashing a machine with JetPack. I have tried to disable nvgetty as a potential solution (according to this Jetson Nano how to use UART on /ttyTHS1) but this has not been effective. This data is coming in at approx. 100Hz from the other IMU board and we are reading this data in via /dev/ttyTHS0. I have included a sample of the good and bad data we have been experiencing below. This error is happening at random and does not appear to line up with any specific load being applied to the system or changes in its surroundings. Any help would be appreciated.

Example of properly formed data. Noting that each entry is on its own line with a ‘$’ in front of it

$PINVMDOP,1346.205,497051.000,1.02,0.91,0.47,0.76,0.49,0.37,0.32,U*54
$PINVMWEK,1346.205,2250*09
$PINVMIAM,1346.210,0.03589,-0.03110,-10.07874,1.94550,-0.85449,0.23651,24.88*23
$PINVMIAM,1346.220,0.00000,-0.06101,-10.08472,1.88446,-0.78583,0.41199,24.63*2B
$PINVMIAM,1346.230,0.03469,-0.06699,-10.09668,2.00653,-0.67139,0.37384,25.00*29
$PINVMIAM,1346.240,0.01675,-0.06938,-10.07874,1.86157,-0.79346,0.48065,24.82*2E
$PINVMIAM,1346.250,-0.00479,-0.07537,-10.08950,2.01416,-0.76294,0.26703,24.88*0F
$PINVMIAM,1346.260,0.02273,-0.05981,-10.08831,1.88446,-0.76294,0.28229,24.88*2E
$PINVMIAM,1346.270,0.02153,-0.05742,-10.06438,1.99127,-0.80872,0.37384,25.00*24
$PINVMIAM,1346.280,0.01196,-0.05024,-10.06079,1.97601,-0.81635,0.35858,24.88*2F
$PINVMIAM,1346.290,0.04905,-0.05144,-10.03687,1.91498,-0.77057,0.33569,25.06*2D
$PINVMIAM,1346.300,0.03708,-0.05264,-10.07634,1.99127,-0.83923,0.28229,24.69*25
$PINVMBMP,1346.305,885.37,24.25,1123.47*17

Example of malformed data. Please note the second line where the newline was not present and the data that was supposed to be on the next line is missing its ‘$’.

$PINVMDOP,4740.320,419527.000,2.17,1.85,1.13,1.60,0.93,0.74,0.56,U*50
**$PINVMWEK,4740.320,22PINVMIAM,4740.330,0.16270,-0.08374,-9.35378,1.11389,-2.28119,-0.11444,33.97*3F**
$PINVMIAM,4740.340,0.15313,-0.05981,-9.38848,1.22070,-2.36511,0.03815,33.85*14
$PINVMIAM,4740.350,0.14355,-0.06938,-9.37292,1.08337,-2.24304,0.00000,33.85*13
$PINVMIAM,4740.360,0.15193,-0.06101,-9.41479,1.07574,-2.29645,-0.03052,33.73*3A
$PINVMIAM,4740.370,0.15432,-0.06938,-9.35737,1.12915,-2.31171,-0.00763,33.97*37
$PINVMIAM,4740.380,0.15313,-0.07537,-9.34661,1.14441,-2.35748,-0.12207,33.97*3B
$PINVMIAM,4740.390,0.15671,-0.06699,-9.38608,1.19781,-2.42615,0.02289,34.03*15
$PINVMIAM,4740.400,0.17346,-0.05623,-9.37173,1.15204,-2.28119,-0.09918,33.97*32
$PINVMBMP,4740.405,902.05,32.02,969.77*2E
2

To follow up to this the baud rate for the UART we are reading is 460800, and I have confirmed that the system is using this baud rate. From what i have read in other forum posts it seems that there is some instability in using UART when the baud exceeds 115200, but we do not have the option of lowering our data rate for our application.

4

Do you have the option of using 2 stop bits? This has sometimes helped.

5

I have tried to implement a second stop bit as per Serial port less reliable after upgrade to 35.1 - Jetson & Embedded Systems / Jetson Xavier NX - NVIDIA Developer Forums but it was not successful with multiple characters still missing from the input at random intervals. Specifically we are losing 8 characters and there seems to be no pattern regarding what messages are losing these characters (typically at the start or end of the message). I find it odd that this behavior is only being experienced after upgrading to 5.0+ and was never experienced in 4.6.3 despite having the same hardware configurations. Are there some settings in the Orin and Xavier systems which may be causing this issue? Is it possible that this is a kernel setting? Or maybe something in the UEFI? My systems were flashed using sdk manager so the kernel configuration should be default.

6

It is possible there is a device tree or clock setting which is an issue, but I don’t know specifically where to look for that. One thing you might examine, if you have a good oscilloscope, is if the timing and frequency of the TX/RX are close to the rate expected at a given setting. I doubt UEFI is involved unless the issue is only in UEFI boot stage, although it can share device tree settings.

Incidentally, 4.6.3 was never run on any Orin. Much of the serial port hardware is duplicated from older systems, e.g., Xavier, but Orin was never tested on the earlier releases. Examine the device tree on the working 4.6.3 and see if the “compatible” part is the same, and then if they are, check device tree settings for the UART. Note that 4.x kernels and 5.x kernels might share the same exact device tree, but since the drivers themselves are different, and because the device tree is essentially designed to pass as an argument for a driver, there could be differences which are correct.

1 Like
7

Hi noah.g,

Have you tired with current Jetpack 35.2.1?

Please also provide the full serial console log for further check.

8

After our testing it seems that Jetpack 35.2.1 has provided a significant improvement, especially when paired with the 2 stop bit solution previously mentioned by linuxdev. While helps to solve the issue of the messages arriving malformed or broken the issue is now of messages not being read in at all. Some messages are being missed by the UART at seemingly random intervals. (>2 per hour) This means that the messages are never visible to the system even though we have confirmed through many continuous hours of testing that these messages are being sent to the Orin/Xavier properly.

To clarify the ask about the full serial console log, are you asking for the log of the UART messages we are reading or some other serial log from the system?

Is there anything else you could suggest that would help to alleviate this issue?

Thanks,
Noah

9

I mean the serial console log for the system status.
For the Orin devkit, you could refer to the following instruction to capture them.
NVIDIA Jetson Orin - Serial Console

Do you mean that you still could not receive data from UART with R35.2.1?

10

I have gathered the log and attached it here, please let me know if there is any issue with it:
orin-serial-console.log (71.3 KB)

I should have been clearer with my previous message; we are able to receive messages from the UART using R35.2.1 but there are brief instances (at seemingly random intervals) where a message or two will not be read by the UART. (ie. if we expected to see messages A,B,C,D come in through UART there are some instances where only A,B,D are recieved even through all (A,B,C,D) are sent)

11

Thanks for your clarification.

We’ve a known issue about serial packet missing randomly on Orin. It could be fixed with updating usb firmware.
I’ve checked yours as follow:

[    9.481200] tegra-xusb 3610000.xhci: Firmware timestamp: 2021-12-01 05:27:10 UTC, Version: 80.03 release

Please wait for the next release, it should fix your problem.

13

Hi,

nvidia jetson agx orin developer kit has two fixed baud rates 16450 and 16550. But in my pc that baud rates are not available. So, what should I do now?

14

You should file a separate topic. However, those are not baud rates. Historically, there was originally a UART chip, the 16450. This had flaws, and the 16550 was created. This later got modified to 16550A. Those are chip compatibilities (the UART can be told to behave as any of those models of chip), not speeds. Normally one would only use the 16550A driver. Speeds are a separate topic. Most of the Jetson UARTs default to speed 115200, 8 bits, no parity, and one stop bit (CTS/RTS flow control, by default, is not normally enabled).

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.