CAN Interface not behaving right on AGX

Hardware Platform: DriveWorks AGX Xavier
Software Version: Drive Software 2.2
Host Machine Version: Ubuntu 18.04.4 LTS (Bionic Beaver)
SDK Manager Version: 1.1.0.6343

Hello,

I have implemented IMU Plugin on Linux host PC and seems to be working fine with the Sample_IMU_Logger on Linux host PC. However, We see some issues when running the IMU on Drive AGX platform, mainly those are related to CAN Interface. We experimented with two IMUs, IMU1 with built-in 120 Ohms resistor and IMU2 without built-in 120 Ohms resistor (we are using external cable for 120 Ohms resistor with IMU2).

On IMU1 (built-in 120 Ohms), We can see data on candump but data is not changing (Static). This setup works only once and have to restart the IMU to run it again.

[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )

As you can see, the data is not changing. And have to power-cycle the IMU to run this sample again.
Same setup on Linux Host works without any issues and IMU data is changing too.

On IMU2 (External wire for 120 Ohms), We don’t see data on CAN1 when candump. From ip statistics we learned that there are lots of Rx errors (termination issues?!?, but we already have termination cable).
To verify that termination cable has no issues we ran the same setup on Linux Host and the IMU Plugin and IMU works fine. And the IMU doesn’t work without termination cable (expected) on Linux host.

Has anyone had similar issue before? Can someone point to What we could be looking at? We know that IMU, cables and connectors are working.

Any help is appreciated. Thanks in advance.

Regards,
Rishit

Check the kernel log using dmesg. Usually CAN issues result in errors being spammed there.

Thanks for the reply Raul.

This is what the output looks like for IMU2.

dmesg | grep can1

[  660.889345] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.890923] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.892488] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.894060] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.895624] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.900923] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.902521] mttcan c320000.mttcan can1: Format Error Detected
[  660.904119] mttcan c320000.mttcan can1: Format Error Detected
[  660.905715] mttcan c320000.mttcan can1: Format Error Detected
[  660.907325] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.908892] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.036590] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.038161] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.039729] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.044810] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.046407] mttcan c320000.mttcan can1: Format Error Detected

From primary google search we found this thread. All of received CAN messages are errors. They had similar issue. Bad connection.

So we ran an experiment and connected our Linux host PC on same CAN networks as DriveWorks and IMU. Initially Drive AGX was turned off and we could see data using candump on linux host pc. Then we turned on Drive AGX and IMU stopped sending data due to Bus errors. Looks like DriveAGX has something wrong connection wise.

As for the IMU1, we learned that it sends out static data on the Linux Host as well. In that case, our CAN Interface and Plugin both are working on Drive AGX. Except we have to restart the IMU when we run the plugin again.

Hi @rborad,

Per https://en.wikipedia.org/wiki/CAN_bus, CAN bus may need two 120 ohms resistors. Could you try with two 120 ohms terminators for the connection? Also, are IMU1 and IMU2 the same IMU (using the same IMU plugin)? Thanks!

Can bus need 120Ohm termination, make sure add on both far end, it’s required in ISO 11898.

1 Like

Thanks for the reply @JimWang and @VickNV.

We did one more experiment. Below is the setup:

In this experiment, We see data from both the IMUs on the Linux Host when Drive AGX is not connected to the bus. As soon as we connected Drive AGX we stop seeing data from the IMUs to the Linux Host. When we remove the Drive AGX, the traffic was back up.

Note: For termination we are using one of this at each end: https://www.gridconnect.com/products/can-terminator?gclid=EAIaIQobChMIkLvGobKY6gIVAr7ACh2g5g8sEAQYASABEgL3tvD_BwE. And none of the IMUs have in-built termination.

Can you help us how else we can debug this, what else we can try to narrow-down the problem?

Thanks in advance.

Hi @rborad,

What does “Drive AGX (IMU1 DB9)” mean? the IMU connector on the cable harness? Why don’t you use the CAN connector on the cable harness?

Thanks for the reply @VickNV .

It was a mistake in tat diagram, this is the updated diagram. I am connecting to CAN1 DB9 connector on harness. Which has two DB9 labels Can1 and Can 2. From those labels I am using Can1 pin 7/ pin 2. Sorry for misunderstanding. I am aware of the fact that IMU connector uses UART.


Without IMU1 and IMU2 CAN devices on the CAN bus, is the CAN communication between Drive AGX and the Linux host (via the CANUSB transceiver) working?

When I connect Drive AGX and Linux host, and send data from Linux using cansend, I see the message on AGX candump.

When I connect my IMU to above setup (IMU + Linux + AGX), the IMU sends some data and stops sending data after few messages. But again, a message using cansend from Linux go though to the AGX. Then I restarted my IMU and same thing again, sent some data and then stopped but Linux - AGX CAN communication was working.

When I connect the IMU to the Linux, i see continuous data sent from the IMU to the linux candump without any problem.

does dmesg tells you something when this kind of error occurs? I forgot to send dmesg after this experiment. I will do that next time and share the screenshot. That might help. Thanks.

Edit: Here is the dmesg output form DriveAGX, this is just the tail of output. The output is full of these errors.

[1718433.055336] mttcan c320000.mttcan can1: Format Error Detected
[1718433.057517] mttcan c320000.mttcan can1: Format Error Detected
[1718433.059065] mttcan c320000.mttcan can1: Format Error Detected
[1718433.060638] mttcan c320000.mttcan can1: Format Error Detected
[1718433.062203] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.063772] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.065318] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.066872] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.068426] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.074106] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.075687] mttcan c320000.mttcan can1: Format Error Detected
[1718433.077274] mttcan c320000.mttcan can1: Format Error Detected
[1718433.078838] mttcan c320000.mttcan can1: Format Error Detected
[1718433.080407] mttcan c320000.mttcan can1: Format Error Detected
[1718433.081967] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.083524] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.085082] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.086674] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.088218] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.214826] mttcan c320000.mttcan can1: Stuff Error Detected

@VickNV, @JimWang , @shayNV, @LukeNV

In above reply I tried Drive AGX with Linux PC and they were able to communicate over the CAN network. With the same setup, removing Linux PC and Adding IMU to the network at the same DB9 connector gives Stuff errors and Format errors. Heres the picture of my setup if any of you can help me verify the setup.

Two ends are terminators I mentioned above.

From the middle two DB9s, one is connected to my IMU and other one to the Drive AGX.

What can i do to debug this issue? I do not have access to DriveAGX directly and I am getting help of a friend who is running these experiments for me. Its very inconvenient but I dont think there is any other way. I just want to test the plugin i wrote and see if it works with my IMU. Can I rent DriveAGX from NVIDIA?

Thanks in advance.
Rishit

CC: @rkomarabathuni

Please refer to “Setting up SocketCAN Interface” topic in the document (especially the notes) and check if all Xavier/Aurix/IMU use same bit timing parameters.

Also could you help to check your aurix firmware version?

Sorry for delayed response. Below is the Firmware version.

SW Version: DRIVE-V5.1.6-E3550-EB-Aurix-With3LSS-ForHyperion-StepA-3.05.04

I will checkout the document you’ve mentioned here.

Thanks
Rishit

1 Like