Could not connect to the device via SSH. Check the IP address, and make sure that SSH service is running on the device

Hi everyone,

We have encountered an issue with our Nivida Jetson Orin AGX (64G dev kit version). We used to be able to flash using SDK manager (jetpack 6.2.1, host machine: linux ubuntu 24.04). But the last flash failed with error like this

‘Could not connect to the device via SSH. Check the IP address, and make sure that SSH service is running on the device.’

  • We tried both manual recovery and automatic and both have the same errors

  • We tried to flash it to either SSD or disk but both are the same

  • We tried different USB cables and the same

  • We attempted to reboot the Orin, but it failed to complete the boot process fully after reaching the system configuration stage

    sdkm-2025-10-04-21-31-54.log (1.1 MB)

  • We attached the sdk log for your reference

Any recommendation is highly appreciated.

*** Please note that this reply is generated by LLM automatically ***

Based on the provided context, I’ll try to help you troubleshoot the issue with flashing your NVIDIA Jetson Orin AGX 64G dev kit.

Error Analysis

The error message “Could not connect to the device via SSH. Check the IP address and make sure that SSH service is running on the device” suggests that the SDK Manager is unable to establish a connection with the device via SSH. This could be due to various reasons such as:

  1. Incorrect IP address or network configuration.
  2. SSH service not running on the device.
  3. Device not in recovery mode.

Troubleshooting Steps

  1. Verify IP address and network configuration: Ensure that the IP address of the device is correct and the network configuration is proper. You can try pinging the device to verify connectivity.
  2. Check SSH service: Verify that the SSH service is running on the device. You can do this by checking the device’s system logs or using a tool like nmap to scan for open ports.
  3. Put the device in recovery mode: Try putting the device in recovery mode by running the command sudo ./flash.sh -r (assuming you’re using the flash.sh script). This will reboot the device in recovery mode, and the SDK Manager should be able to connect to it.
  4. Try manual recovery: If automatic recovery fails, try manual recovery by following the steps outlined in the SDK Manager documentation.
  5. Check USB cables and connections: Verify that the USB cables and connections are secure and not loose.
  6. Reboot the device: Try rebooting the device and see if it completes the boot process fully.

Comparison with Similar Issues

Based on the provided context, I noticed that there are similar issues reported by other users, such as:

  • “Orinnx poweroff and power on,it not work。” ( Jetson Orin NX boot issue)
  • “Timeout when flashing Jetson Orin Nano 8G module” (Jetson Orin Nano boot issue)

These issues suggest that there might be a problem with the device’s boot process or power management. However, without more information, it’s difficult to say for certain.

Recommendations

Based on the analysis and troubleshooting steps, I recommend the following:

  1. Verify the IP address and network configuration.
  2. Check the SSH service on the device.
  3. Put the device in recovery mode and try flashing again.
  4. Try manual recovery if automatic recovery fails.
  5. Check USB cables and connections.
  6. Reboot the device and see if it completes the boot process fully.

If none of these steps resolve the issue, please provide more information about the device’s configuration, system logs, and any other relevant details.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

Hi,

Just want to clarify. Do you hit an issue with flash failed or the flash is done but there is ssh problem after the flash? Or you could not tell what I mean here?

Hi Wayne, the Linux OS has been flashed, and the Orin can boot. However, it hangs after completing the system configuration, so we’re unable to proceed further—for example, we can’t even check whether the SSH server is running (but we know it’s not running)

Hi,

If this is NV devkit, please use the micro usb port to check the serial log after you power on the board.

This is the only precise way to check what is going on with the board.

note: please be aware that not the type C port but the micro usb port.

Can you flash the Jetson and tell it to not install any optional components? For example, don’t install anything to host PC, don’t install CUDA, so on. You can always install those later (you can uncheck those items during flash even though it isn’t always obvious you can uncheck everything except flash). Then, if it fails, get a serial console boot log.

Incidentally, all of that other content installs to a fully booted system over ssh. During a flash the flash will complete and the Jetson automatically reboots, and this is when those other components are installed. If you uncheck everything except for installing those optional components, then you have to start fully booted with ssh available (which implies you have a user account with password available after first boot account setup).

If you can boot with no optional content, then you would want to log in and see if you can ssh from that admin account to itself at the 127.0.0.1 (or localhost) address. This would tell you if things are working normally. If this works, then a failure implies the host PC or network is getting in the way of ssh (e.g., sometimes a host PC has security designed to not allow ssh, or else the USB version of networking is refused because it won’t allow a USB device to auto configure; you don’t have to use the USB networking for optional component install).

hi Wayne and everyone,

Attached is the full log (for two tries). The last errors are

[ 23.187023] Please complete system configuration setup on desktop to proceed…
[ 30.325582] nvidia-modeset: ERROR: GPU:0: nvRmApiAlloc(memory) failed for vrr 0x22
[ 30.325989] nvidia-modeset: ERROR: GPU:0: Failed to setup Rgline active session for vrr

The system configuration step also has an error
tail: can’t open ‘/var/log/installer/debug’: No such file or directory
tail: can’t open ‘/var/log/syslog’: No such file or directory
tail: no files

Your feedback is highly appreciated!

minicom-2nd.log (217.0 KB)

hi linuxdev,

I have given it a try but still the same. The Orin AGX can’t boot up fully and the ssh server is not running. I feel that It’s not a network or usb issue as we were able to connect to it before.

It gets through MB1 and MB2 boot stages, then fails, so early boot works. The video shows it did indeed boot, it just had issues with the user space environment. I will ask some questions which might be relevant:

  • What is the exact byte size of your host side “Linux_for_Tegra/system.img.raw” file after the flash?
  • Is there anything at all unusual about the host PC itself?
  • Was first boot setup done entirely on the Jetson, and without using the “l4t_create_default_user.sh” script?
  • Does this succeed from the host side:
sudo mount -o loop bootloader/system.img.raw /mnt
# ...edit "/mnt/boot/extlinux/extlinux.conf", remove any
# occurrence of the word "quiet".
# Use "cd" to exit "/mnt" and go back to "Linux_for_Tegra/"
sudo umount /mnt

There is something more unusual we could try. It is possible to flash on command line and reuse an existing image of the rootfs. The edited system.img.raw can be converted to system.img using this, assuming you are in “Linux_for_Tegra/bootloader/” of the host PC:
./mksparse --fillpattern=0 system.img.raw system.img
(you could just rename system.img.raw to system.img, but then flash would take a lot longer)

The reason for asking about the exact byte size of system.img.raw is that command line flash of any size other than default requires specifically naming the size.

It is possible to complete first boot setup in other ways as well, and normally one does this to the “Linux_for_Tegra/rootfs/” directory, and this is then generated into a system.img, but you loopback mount the edited system.img.raw on “Linux_for_Tegra/rootfs/”, complete the l4t_create_default_user.sh, and umount it, followed by conversion of the system.img.raw to system.img. What this will do is to give you exact control of the image to be flashed.

Normally the Linux_for_Tegra/rootfs/ is a “nearly” exact match to the final system.img, but minor edits occur based on the flash target. Making sure there is no “quiet” in the system.img, along with making sure first boot setup is complete in the actual system.img, then reusing that image, means you are pretty much guaranteed there is nothing being altered. If the fault is in the rootfs image, then you have a copy on the host which should have been correct.

This can also be command line flashed to reuse the image with an initrd flash.

I am convinced that at least the Linux kernel is running, and you would not have reached a desktop (working or not) if boot was not going significantly into user space. If the failure is one of the first boot setup, then editing like this will guarantee first boot setup if the image has the “l4t_create_default_user.sh” complete (and logging is guaranteed ok by removing “quiet” from extlinux.conf; you couldn’t guarantee this in the “Linux_for_Tegra/rootfs/boot/extlinux/extlinux.conf” since it is one of the boot files which might be changed during a normal flash).

Even if you don’t do all of this, then you could flash fresh, but use the “bootloader/l4t_create_default_user.sh” to complete the first boot setup and remove that from the boot before the flash even begins. You don’t have to do the loopback image edit, but you probably already have the image, and other than any edits you might make, it is known this is the exact match to the flashed content (you can control exactly what to change prior to flash reusing the image, but you won’t know that exactly nothing changed if flashing normally).

Hi everyone, we have re-installed the host machine and retried the flashing again and it was successful. Even though we don’t know exactly what happened, we believe it’s related to some settings in our host machine. Thanks very much to everyone here!

[ 8.735400] systemd[1]: Configuration file /lib/systemd/system/systemd-logind.service is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 8.735403] systemd[1]: Configuration file /lib/systemd/system/systemd-logind.service is marked world-writable. Please remove world writability permission bits. Proceeding anyway.
[ 8.736153] systemd[1]: Configuration file /lib/systemd/system/modprobe@.service is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 8.736157] systemd[1]: Configuration file /lib/systemd/system/modprobe@.service is marked world-writable. Please remove world writability permission bits. Proceeding anyway.
[ 8.736311] systemd[1]: Configuration file /lib/systemd/system/user.slice is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 8.736314] systemd[1]: Configuration file /lib/systemd/system/user.slice is marked world-writable. Please remove world writability permission bits. Proceeding anyway.
[ 8.736490] systemd[1]: Configuration file /lib/systemd/system/systemd-ask-password-wall.path is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 8.736493] systemd[1]: Configuration file /lib/systemd/system/systemd-ask-password-wall.path is marked world-writable. Please remove world writability permission bits. Proceeding anyway.
[ 8.736867] systemd[1]: Configuration file /lib/systemd/system/systemd-ask-password-wall.service is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 8.736870] systemd[1]: Configuration file /lib/systemd/system/systemd-ask-password-wall.service is marked world-writable. Please remove world writability permission bits. Proceeding anyway.
[ 8.737218] systemd[1]: Configuration file /lib/systemd/system/sssd.service is marked executable. Please remove executable permission bits. Proceeding anyway.
[ 8.737221] systemd[1]: Configuration file /lib/systemd/system/sssd.service is marked world-writable. Please remove world writability permission bits. Proceeding anyway.

Just to clarify. The problem I saw here is your previous log showed such thing.

Thus, it looks like your previous installation has some permission issue. That was why reinstall might help.

Now that it works you might want to consider cloning of at least the rootfs as a backup. If you clone just the rootfs, and save the raw clone (there is also a sparse clone), then you can loopback mount it on a host PC and then later use rsync over ssh to update it from the Jetson as you get to save points.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.