CAN Interface not behaving right on AGX

[rborad]

Hardware Platform: DriveWorks AGX Xavier
Software Version: Drive Software 2.2
Host Machine Version: Ubuntu 18.04.4 LTS (Bionic Beaver)
SDK Manager Version: 1.1.0.6343

Hello,

I have implemented IMU Plugin on Linux host PC and seems to be working fine with the Sample_IMU_Logger on Linux host PC. However, We see some issues when running the IMU on Drive AGX platform, mainly those are related to CAN Interface. We experimented with two IMUs, IMU1 with built-in 120 Ohms resistor and IMU2 without built-in 120 Ohms resistor (we are using external cable for 120 Ohms resistor with IMU2).

On IMU1 (built-in 120 Ohms), We can see data on candump but data is not changing (Static). This setup works only once and have to restart the IMU to run it again.

[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )
[1569028114105615] Gyro(X:-0.09375 Y:-0.046875 Z:-0.0234375 )
[1569028114103955] Orientation(R:-250 P:-250 Y:0 )
[1569028114105036] Acceleration(X:-0.0500183 Y:-0.0800171 Z:-9.79001 )

As you can see, the data is not changing. And have to power-cycle the IMU to run this sample again.
Same setup on Linux Host works without any issues and IMU data is changing too.

On IMU2 (External wire for 120 Ohms), We don’t see data on CAN1 when candump. From ip statistics we learned that there are lots of Rx errors (termination issues?!?, but we already have termination cable).
To verify that termination cable has no issues we ran the same setup on Linux Host and the IMU Plugin and IMU works fine. And the IMU doesn’t work without termination cable (expected) on Linux host.

Has anyone had similar issue before? Can someone point to What we could be looking at? We know that IMU, cables and connectors are working.

Any help is appreciated. Thanks in advance.

Regards,
Rishit

[raul.tambre.clevon]

Check the kernel log using dmesg. Usually CAN issues result in errors being spammed there.

[rborad]

Thanks for the reply Raul.

This is what the output looks like for IMU2.

dmesg | grep can1

[  660.889345] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.890923] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.892488] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.894060] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.895624] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.900923] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.902521] mttcan c320000.mttcan can1: Format Error Detected
[  660.904119] mttcan c320000.mttcan can1: Format Error Detected
[  660.905715] mttcan c320000.mttcan can1: Format Error Detected
[  660.907325] mttcan c320000.mttcan can1: Stuff Error Detected
[  660.908892] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.036590] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.038161] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.039729] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.044810] mttcan c320000.mttcan can1: Stuff Error Detected
[  661.046407] mttcan c320000.mttcan can1: Format Error Detected

From primary google search we found this thread. All of received CAN messages are errors. They had similar issue. Bad connection.

So we ran an experiment and connected our Linux host PC on same CAN networks as DriveWorks and IMU. Initially Drive AGX was turned off and we could see data using candump on linux host pc. Then we turned on Drive AGX and IMU stopped sending data due to Bus errors. Looks like DriveAGX has something wrong connection wise.

As for the IMU1, we learned that it sends out static data on the Linux Host as well. In that case, our CAN Interface and Plugin both are working on Drive AGX. Except we have to restart the IMU when we run the plugin again.

[VickNV]

Hi @rborad,

Per CAN bus - Wikipedia, CAN bus may need two 120 ohms resistors. Could you try with two 120 ohms terminators for the connection? Also, are IMU1 and IMU2 the same IMU (using the same IMU plugin)? Thanks!

[JimWang]

Can bus need 120Ohm termination, make sure add on both far end, it’s required in ISO 11898.

[rborad]

Thanks for the reply @JimWang and @VickNV.

We did one more experiment. Below is the setup:

In this experiment, We see data from both the IMUs on the Linux Host when Drive AGX is not connected to the bus. As soon as we connected Drive AGX we stop seeing data from the IMUs to the Linux Host. When we remove the Drive AGX, the traffic was back up.

Note: For termination we are using one of this at each end: CAN Terminator with 120 Ohm Resistor – Grid Connect. And none of the IMUs have in-built termination.

Can you help us how else we can debug this, what else we can try to narrow-down the problem?

Thanks in advance.

[VickNV]

Hi @rborad,

What does “Drive AGX (IMU1 DB9)” mean? the IMU connector on the cable harness? Why don’t you use the CAN connector on the cable harness?

[rborad]

Thanks for the reply @VickNV .

It was a mistake in tat diagram, this is the updated diagram. I am connecting to CAN1 DB9 connector on harness. Which has two DB9 labels Can1 and Can 2. From those labels I am using Can1 pin 7/ pin 2. Sorry for misunderstanding. I am aware of the fact that IMU connector uses UART.

[VickNV]

Without IMU1 and IMU2 CAN devices on the CAN bus, is the CAN communication between Drive AGX and the Linux host (via the CANUSB transceiver) working?

[rborad]

When I connect Drive AGX and Linux host, and send data from Linux using cansend, I see the message on AGX candump.

When I connect my IMU to above setup (IMU + Linux + AGX), the IMU sends some data and stops sending data after few messages. But again, a message using cansend from Linux go though to the AGX. Then I restarted my IMU and same thing again, sent some data and then stopped but Linux - AGX CAN communication was working.

When I connect the IMU to the Linux, i see continuous data sent from the IMU to the linux candump without any problem.

does dmesg tells you something when this kind of error occurs? I forgot to send dmesg after this experiment. I will do that next time and share the screenshot. That might help. Thanks.

Edit: Here is the dmesg output form DriveAGX, this is just the tail of output. The output is full of these errors.

[1718433.055336] mttcan c320000.mttcan can1: Format Error Detected
[1718433.057517] mttcan c320000.mttcan can1: Format Error Detected
[1718433.059065] mttcan c320000.mttcan can1: Format Error Detected
[1718433.060638] mttcan c320000.mttcan can1: Format Error Detected
[1718433.062203] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.063772] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.065318] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.066872] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.068426] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.074106] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.075687] mttcan c320000.mttcan can1: Format Error Detected
[1718433.077274] mttcan c320000.mttcan can1: Format Error Detected
[1718433.078838] mttcan c320000.mttcan can1: Format Error Detected
[1718433.080407] mttcan c320000.mttcan can1: Format Error Detected
[1718433.081967] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.083524] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.085082] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.086674] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.088218] mttcan c320000.mttcan can1: Stuff Error Detected
[1718433.214826] mttcan c320000.mttcan can1: Stuff Error Detected

[rborad]

@VickNV, @JimWang , @shayNV, @LukeNV

In above reply I tried Drive AGX with Linux PC and they were able to communicate over the CAN network. With the same setup, removing Linux PC and Adding IMU to the network at the same DB9 connector gives Stuff errors and Format errors. Heres the picture of my setup if any of you can help me verify the setup.

Two ends are terminators I mentioned above.

@rborad

https://www.gridconnect.com/products/can-terminator?gclid=EAIaIQobChMIkLvGobKY6gIVAr7ACh2g5g8sEAQYASABEgL3tvD_BwE

From the middle two DB9s, one is connected to my IMU and other one to the Drive AGX.

What can i do to debug this issue? I do not have access to DriveAGX directly and I am getting help of a friend who is running these experiments for me. Its very inconvenient but I dont think there is any other way. I just want to test the plugin i wrote and see if it works with my IMU. Can I rent DriveAGX from NVIDIA?

Thanks in advance.
Rishit

CC: @rkomarabathuni

[VickNV]

Please refer to “Setting up SocketCAN Interface” topic in the document (especially the notes) and check if all Xavier/Aurix/IMU use same bit timing parameters.

Also could you help to check your aurix firmware version?

[rborad]

Sorry for delayed response. Below is the Firmware version.

SW Version: DRIVE-V5.1.6-E3550-EB-Aurix-With3LSS-ForHyperion-StepA-3.05.04

I will checkout the document you’ve mentioned here.

Thanks
Rishit

[VickNV]

Please also check if this topic helps on your problem. Thanks!

Unable to get Radar data using CAN Interface

I tried to connect the Continental ARS 308 radar with the NVidia DRIVE Xavier using CAN bus. But I do not get any data out of the CAN bus. I have attached the screenshot showing how I connected the radar to the DRIVE Box. I used a 120 ohm resistor to terminate the CAN communication and in-turn connect the radar to the CAN_1&2 Port in the DRIVE Box. [20200924_163231] When I run the “candump” command, I find no response showing up. When I checked the SDK logs using the “dmesg” command, I get t…

[VickNV]

Hi @rborad,

May I know if this issue is fixed on your side?

[rborad]

Hi @VickNV,

We are getting DriveAGX soon, I will try your suggestions once we have the box. Thanks for following up. I will update the status here.

Thanks for helping out.

Regards,
Rishit

[rborad]

@VickNV,

I was able to get mttcan can1 (can1) through my Host PC. I am able to send and receive messages to/from the Host to can1. I know that both Host PC and can1 are configured correctly. I moved both the devices to network where I can connect more CAN devices. (just simple daisy chained 4 DB9 connectors, image attached)

Next I connect my silent IMU to the CAN network. It runs into the problem when I send message to my IMU device to start sending message at 200Hz. (more details below) Here is the dmesg log:
errorlog.txt (71.2 KB)

At the startup, I am able to send a CAN message to my IMU though cansend can1 18FF5500#8001 command and enable the IMU to send periodic messages. The IMU receives this command and starts sending data packets at 200Hz rate on the CAN bus. But because mttcan can1 detects stuff errors and format errors, mttcan can1 enters error passive state and the bus shows some packets are received from the IMU around every four seconds. During all this, Host PC and mttcan can1 is able to send and receive messages broadcasted on the bus. As soon as mttcan can1 is removed from the bus and just IMU and Host PC are connected, HostPC starts receiving messages at 200Hz from the IMU.

I am assuming external CAN Transceiver is not required connecting to the can1 interface (Mechanical Guid says CAN_H & CAN_L not CAN_tx & CAN_rx). Our IMU has 5V transceiver, how do I know the transceiver specification for the DriveAGX CAN, can you please point me out to the HW spec?

EDIT: In above experiment the IMU has 5V transceiver, when IMU with 3.3V transceiver is connected, the IMU goes to error mode and stops sending packets after 8-10 packets sent. Our Goal is to make both IMUs work, is it possible?

EDIT2: IMU with 3.3V sends address claim packet every one second. CAN1 receives each packet all right. With this setup, I send can message through CAN1 and i don’t see anything on the bus, after 10 TX tries I get

write: no buffer space available

(qlen is 10). How ever CAN1 still receives address claim packets from IMU every second and receives random messages i send from Linux Hosts. Basically CAN1 TX is stopped. ErrorLog: TXErrorRXGoodStats.txt (893 Bytes) TXErrorRXGoooddmesg.txt (83.0 KB)

Then I configured IMU3.3V to send out data packet at minimum rate possible to check if CAN1 receives those but then both TX and RX gets stuck.
TXErrorRXErrordmesg.txt (83.0 KB) TXErrorRXErrorStats.txt (893 Bytes)

Hope this helps.

Regards,
Rishit

[rborad]

Thanks for pointing out to this thread. I think here the problem was pin connections on DB9 (1&7 instead of 2&8). I have created a network with CAN1&2 connected through pin 1&7 and CAN6&4 is connected through DB9 pins 2&8. And I am able to send commands from can0(CAN1&2) to can1(CAN6&4) and vice-versa.

Update,
The IMU i am trying to connect uses 3.3V transceiver internally to convert CAN_tx_rx to CAN_H&_L. I tried another IMU with 5V transceiver and and I see that CAN1 receives some message at variable rate. Looks like there are so many errors. CANErrorStat.txt (902 Bytes)

It would be really helpful if you could direct me to transceiver spec on the DriveAGX.

Thanks in advance.
Regards,
Rishit

[VickNV]

We are checking with the internal team and will get back to you here.

BTW, you may not notice that we created below poll. Please help to vote there. Thanks!
https://forums.developer.nvidia.com/t/what-about-your-experience-in-using-the-bundled-sekonix-ar0231-gmsl/158695/3

[VickNV]

In terms of the voltage level, on the DRIVE AGX cable harness, it should be 5V for the CAN interfaces. Please note that you need to have 120 ohm termination on both ends. On DRIVE AGX side, we only lightly terminate the bus. The 5V transceiver you are using might not have the 120 ohm termination. You can refer to “Figure C-4. Example of E3556 System Connections” in DRIVE AGX Developer Kit Product Brief (PDF) as an example of RTC module that connects to CAN 6&4.