I am trying to move some of our workload from the Tegra A SoC to the Tegra B SoC. I am running into an issue where if the Tegra A is rebooted (via sudo reboot), the Tegra B can become unresponsive.
To reproduce this issue:
Flash Tegra A and Tegra B with drive 4.1.8.0L software using DriveInstall_4.1.8.0L_SDK_b43.run
Assign a static IP to both tegras. by editing /etc/network/interfaces - we use 192.168.3.50 for A and 192.168.3.49 for B.
Run ping 192.168.3.49 to see when B is active
Issue the command aurixreset to the AURIX Console. After running this you should see Tegra B stop responding to pings, then come back online when it finishes rebooting.
ssh to Tegra A and run sudo reboot. At this point Tegra B should stop responding to pings and will not recover until power is lots or until aurixresetis run.
I am looking for either (1) a way to recover Tegra B after it enters the non-responsive state, or (2) a way to reset the system that does not cause Tegra B to freeze. We want to avoid using AURIX Console over USB for this purpose because it would require retrofitting our systems. If there is another way to access the console that would work well.
We currently use PX2 Software Version 4.1.8.0L, and it would be difficult for us to switch away from this version. If this is the only way to resolve our issue we will probably instead add the USB cable to each of our systems.
are you able to open the debug console for tegra B when you execute the reboot command on tegra A?
(please refer to the instructions for connecting to tegra B through ttyUSB6 )
if you could post the output of tegra B debug console and check if it is responsive and the output of command ifconfig after rebooting tegra A it might help up understand better the problem.
Are you able to share why it would be difficult for you to upgrade to the latest version released for Drive PX2?
The Tegra B usb console (on ttyUSB6) stops responding at the same moment that Tegra B stops responding to pings. Around 30 seconds after it freezes, the message “[00203361] wdt: expired vmid 0” appeared in the log. Full output at time of crash follows:
[ OK ] Started Getty on tty1.
[ **] (1 of 2) A start job is running for...lient on reboot (34s / 5min 14s[ ***] (1 of 2) A start job is running for...lient on reboot (34s / 5min 14s[ *** ] (1 of 2) A start job is running for...lient on reboot (35s / 5min 14s[* ] (1 of 2) A start job is running for...lient on reboot (37s / 5min 14s[** ] (1 of 2) A start job is running for...lient on reboot (37s / 5min 14s[*** ] (1 of 2) A start job is running for...lient on reboot (38s / 5min 14s[ **] (1 of 2) A start job is running for...lient on reboot (40s / 5min 14s[ *] (1 of 2) A start job is running for...lient on reboot (40s / 5min 14s[ **] (1 of 2) A start job is running for...lient on reboot (41s / 5min 14s[*** ] (1 of 2) A start job is running for...lient on reboot (43s / 5min 14s[** ] (1 of 2) A start job is running for...lient on reboot (43s / 5min 14s[* ] (1 of 2) A start job is running for...lient on reboot (44s / 5min 14s[ OK ] Started LSB: start ptpd client on reboot.
[ OK ] Created slice User Slice of nvidia.
Starting User Manager for UID 1001...
[ OK ] Started Session 1 of user nvidia.
[ OK ] Started User Manager for UID 1001.
Stopping User Manager for UID 1001...
[ OK ] Stopped User Manager for UID 1001.
[ OK ] Removed slice User Slice of nvidia.
[ **] A start job is running for Setup Wizard (2min 18s / no limit)�[00203361] wdt: expired vmid 0
[00263361] wdt: expired vmid 0
[00323363] w
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
# Tegra A network configuration
auto eth0
iface eth0 inet static
address 192.168.3.50
netmask 255.255.255.0
Tegra B:
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
# Tegra B network configuration
auto eth0
iface eth0 inet static
address 192.168.3.49
netmask 255.255.255.0
Dear @jchadwell,
I tested the same steps on latest release and I don’t see any issue. I could connect to both Tegras via minicom and notice eth0 IP were set as expected.
I don’t believe that the static IP is part of the issue. I only included it in my post to help your team replicate exactly what I am seeing. However, I don’t have a great way to test it without configuring an external IP for the board. We typically interact with our boards over the external IP interface (no monitor / keyboard / mouse plugged in).
I will check if the issue replicates without setting the IP configuration on just Tegra B, which I should be able to accomplish by ssh’ing through Tegra A’s external IP then modifying Tegra B over the internal LAN. I’ll post the results of this test here in an hour or so.
I was able to confirm that modifying the networking configuration on Tegra B is not necessary to cause this issue to replicate. It doesn’t seem like the network setup is the root cause of this issue.
I also tested on the latest drive release, and I was unable to reproduce the issue. It seems like some change between version 4.1.8 and version 5.0.10 resolves the bug. We will try to update to the newer version at some point, but it seems like it will take a good deal of rework on our current build system and camera pipeline implementation to bring it up to date.
Can you confirm that there is no way to trigger the same behavior as the “aurixreset” aurix console command without a USB debugging cable attached? This command consistently recovers the board from the frozen state. If there is another way to trigger it that would help us bridge the gap between now and when we are able to update to a newer version.
Dear @jchadwell,
You can connect to aurix shell only via usb debugging cable(like using minicom) to run aurixreset. As you said, it could be an issue on older SW version and got fixed in subsequent releases.