VPN connection cause Orin system hang

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
1

I am trying to set up the ORIN DEV KIT as a server for a particular pilot workload.

I have tried booting to multi-user.target but it blanks the screen and I can’t control the unit. Both in multi-user and graphical targets the system will stop working after I disconnect the peripherals. I’ve tried disabling all power options that may cause this issue.

The big issue here is that I need to be connecting to this “server” remotely via SSH over VPN. It all works when the user is logged in on the device and all peripherals are there. Unfortunately after disconnecting the peripherals the SSH connectivity breaks (some minutes after). When I reattach the peripherals, it all magically returns to work and both WiFi and VPN are on.

I wish to use ORIN as a headless unit. Please help.

2

I can’t answer because I don’t know enough about Wi-Fi configuration, but the default Ubuntu behavior is typically that Wi-Fi only connects when the user is logged in to the GUI. I have not done this, but I know it is possible to configure the Wi-Fi to always be up in multi-user.target. I think if you do that, then the GUI won’t matter. Basically, I think this is an Ubuntu behavior and not dependent upon the hardware.

3

OK I have managed to transition from WiFi to cable. The client finally allowed for this. Problem is resolved I guess.

4

The problem is not resolved. On cable after disconnecting mouse and keyboard the Orin disconnects from VPN after a day or so.

5

please follow the tutorial here to use micro usb port and see if your serial console log is still alive after all these system stuck issue you hit.

Please be aware that I am talking about micro usb port but not type C usb port.

6

Basically, what @WayneWWW says, but I’ll add that perhaps you could leave a serial console log running, and this would indicate when the network goes down; however, so long as it fails and you don’t mess with the network or attach mouse/keyboard (still using serial console), you could simply run “dmesg” after that event and we could see what is needed at the end of that log. Additionally, once this has happened, you might add the output (via serial console) for:

  • ifconfig (or alternatively ip -s addr)
  • route (or alternatively ip route)

I understand it might be difficult since this requires physical access and a second computer. Sounds like it is not local to your location.

8

Thanks for the responses. I will run dmesg and syslog as soon as I get access…

1 Like
9

PS. Before sending it to the client the orin was flashed using USB and the installation was made on the 512GB NVMe drive.

10

Managed to catch this in the SSH on a second unit that’s been experiencing similar issues.

Message from syslogd@surveily-orin-01 at Feb  3 09:47:14 ...
 kernel:[  446.328294] Internal error: Oops: 8600000d [#1] PREEMPT SMP
11

Hi,

I don’t really care about how you get the dmesg. Ssh or uart serial console are both fine to me.

Just need to make sure you can dump the full dmesg and attach it here.

BTW, if this is “system stuck” issue, it could be kernel panic, on that case, ssh may not work. That is why I keep asking you to dump the log from micro usb port. As that port won’t go dead even in kernel panic case.

12

Wayne as I said previously, that device is 1500km away from me and I don’t have any resource to access it physically. I am working on getting the dmesg and syslog.

13

that device is 1500km away from me a

Sorry. I didn’t notice that. But I still need to tell the truth that if this is really kernel panic, then my previous comment is still valid.

14

pwd on priv

15

Sorry to say this. But could you just share dmesg? I am not quite sure what is the difficulty to share dmesg here.

If there is any problem there, please let me know.

It is not a good idea to directly check a 830MB syslog… Many info are not needed…

16

dmesg, same pwd:

17

dmesg doesn’t have any entries when the hang/freeze/crash happens

18

We believe the situation is like this: LAN connections to Orin work fine. L2TP connections cause it to freeze.

The L2TP is installed via sudo apt install network-manager-l2tp network-manager-l2tp-gnome

We can reproduce the crash by rsyncing a large file from Orin to local via L2TP. It works fine via LAN.

We don’t have LAN access on a daily basis, it needs to be a VPN.

19

both dmesg and syslog shared in above links @WayneWWW

20

Hi @Turowicz ,

I checked your log.

But firstly I want to double confirm and clarify the situation here.

  1. I checked the comments again. But from no where I can know that this device is far away from you…

  2. If the device is far away from you, then how did you “disconnect the peripherals”?

  3. What is the scenario when you dumped the dmesg?

  4. So you can remove the peripherals but you cannot connect a micro usb cable to the devices? Is flashing jetson device a feasible action you can take on your side? I am not asking for reflashing the board. Just want to understand the situation.

21

@WayneWWW

  1. I clearly stated the device is managed remotely in the third paragraph of my original message.
  2. There is a person on site (client) that I can ask to help. I don’t want to bother them though.
  3. I’ve started dmesg with dmesg -wH and then did the steps that crash Orin: file download over VPN (l2tp).
  4. The client won’t be able to do any of the flashing since it’s too complex and requires more tools than one should ask a client of.

PS. We have now managed to set up a secondary Orin In-house and have reproduced a similar issue. The only difference is that in-house Orin boots back up after crash and appears on VPN, contrary to the client one. I can connect USB to it though.