Fresh Flash on Xavier out of box fail: Error: CUDA cannot be installed on device

Hi, I just installed Jetpack 4.1.1 on my host Ubuntu 18.04 machine and pulled the Xavier out of the box and began the process of flashing it for the first time. After it flashed the operating system and started installing the packages, I got this error:

Error: CUDA cannot be installed on device. This may be caused by other apt-get command running on device when installing CUDA. Please use apt-get command in a terminal to make sure the following packages are installed correctly on device before continuing:
cuda-toolkit-10-0 libgomp1 libfreeimage-dev libopenmpi-dev openmpi-bin
After these packages are installed on the device, press Enter key to continue.

So… I assumed that I needed to issue an apt-get command on a terminal on the Xavier, not the host with Jetpack, but that failed:

sudo apt-get install cuda-toolkit-10-0 libgomp1 libfreeimage-dev libopenmpi-dev openmpi-bin

resulted in a bunch of:
E: Unable to locate package cuda-toolkit-10-0 etc…

Please help! Does anyone know what I am doing wrong or how to make the stock flash work without failing?

Thanks!

Scott

Hi,

Could you reboot your device and install the library with JetPack again?
This issue occurs when JetPack wants to install the package before the system getting ready.

Thanks.

Hi, I did try re-installing everything from scratch again a couple of times (this includes the operating system as well) and ended up with the same result.

If I go into JetPack and just try installing the library only to Xavier, it asks for the ip address of the Xavier. How do I figure out what the ip address is? I have it connected to the host by the supplied USB C cable. What command would I issue in the terminal to see the Xavier’s ip address? I will then try to just install the libraries from JetPack only and see how that works.

Thanks!

Scott

OK, I just figured out that the ip address of the connected Xavier is 192.168.55.1

Using Jetpack to only install the packages on the Xavier also fails in exactly the same way. Just before the error it reports a series of:

dpkg-query:package ‘cuda-toolkit-10-0’ is not installed and no information is available

I have followed the JetPack installation instructions without any deviation. I have now started from scratch and elected to have JetPack install everything on the host and on the Xavier. All items are selected as install. Still no change in the results. Failure to install these critical packages. I can’t understand why the process doesn’t work as described out of the box. I am not trying to do anything fancy, just want a stock operational Xavier to test running some Python code on.

Any ideas of anything I can try?

The host is connected to the Xavier through the USB cable.

Thanks!

Scott

To support the last comments, here are the actual results in the terminal window:

Copying /home/scott/Downloads/jetpack_download/cuda-repo-l4t-10-0-local-10.0.117_1.0-1_arm64.deb file to target…
nvidia:
(Reading database … 112599 files and directories currently installed.)
Preparing to unpack …/cuda-repo-l4t-10-0-local-10.0.117_1.0-1_arm64.deb …
Unpacking cuda-repo-l4t-10-0-local-10.0.117 (1.0-1) over (1.0-1) …
Setting up cuda-repo-l4t-10-0-local-10.0.117 (1.0-1) …
OK
Get:1 file:/var/cuda-repo-10-0-local-10.0.117 InRelease
Ign:1 file:/var/cuda-repo-10-0-local-10.0.117 InRelease
Get:2 file:/var/cuda-repo-10-0-local-10.0.117 Release [574 B]
Err:3 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Could not resolve ‘ports.ubuntu.com
Err:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Could not resolve ‘ports.ubuntu.com
Err:5 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Could not resolve ‘ports.ubuntu.com
Err:6 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Could not resolve ‘ports.ubuntu.com
Get:2 file:/var/cuda-repo-10-0-local-10.0.117 Release [574 B]
Get:7 file:/var/cuda-repo-10-0-local-10.0.117 Release.gpg [819 B]
Get:7 file:/var/cuda-repo-10-0-local-10.0.117 Release.gpg [819 B]
Reading package lists… Done
Reading package lists… Done
Building dependency tree
Reading state information… Done
gnupg is already the newest version (2.2.4-1ubuntu1.1).
The following packages were automatically installed and are no longer required:
libdbusmenu-gtk4 libdbusmenu-qt5-2 libgsettings-qt1 liblockfile-bin
liblockfile1 libqt5sql5 libqt5sql5-sqlite lockfile-progs
Use ‘sudo apt autoremove’ to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists… Done
Building dependency tree
Reading state information… Done
Connection to 192.168.55.1 closed.
dpkg-query: package ‘cuda-toolkit-10-0’ is not installed and no information is available
dpkg-query: package ‘libfreeimage-dev’ is not installed and no information is available
dpkg-query: package ‘libopenmpi-dev’ is not installed and no information is available
dpkg-query: package ‘openmpi-bin’ is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
1
Error: CUDA cannot be installed on device. This may be caused by other apt-get command running on device when installing CUDA. Please use apt-get command in a terminal to make sure following packages are installed correctly on device before continuing:
cuda-toolkit-10-0 libgomp1 libfreeimage-dev libopenmpi-dev openmpi-bin
After these packages are installed on device, press Enter key to continue

Hi,

USB is only used for flash OS image.
After that, all the package/libraries installation go through the internet.

We will try this internally first. Update to you soon.

Hi sidener2002,

Please try below steps to install CUDA:

mkdir cuda-l4t
Copy cuda-repo-l4t-10-0-local-10.0.117_1.0-1_arm64.deb to cuda-l4t folder
Copy cuda-l4t.sh to cuda-l4t folder
cd cuda-l4t
sh cuda-l4t.sh cuda-repo-l4t-10-0-local-10.0.117_1.0-1_arm64.deb 10.0 10-0

OK, I will give this a try this afternoon, Thanks! I thought that for Xavier (per the diagram in the Jetpack installation instructions) that both the OS and the packages were sent through the USB? Only the host is directly connected to the internet. It is the first of two diagrams on how to connect everything. Am I wrong about that?

Maybe the other packages won’t install either.

Scott

USB is used only for flash. After a flash the system reboots and packages are installed over wired ethernet. “ssh” is used (sometimes via “scp”), and so if you can’t manually “ssh nvidia@”, then JetPack will also fail. Manually checking if you can ssh to account “nvidia” is a great way to see if this is an issue (and sometimes it simply wants the first ssh to be confirmed that you trust the connection).

I have also seen and used this option in JetPack-4.1.1 and faced similar problems with Ubuntu-16.04 host.
I think the problem was the connection through USB which was renewed about each 30s.
Then I suceeded with the other option by reflashing with usb and post installing with wired ethernet.

Thanks everyone for the suggestions! I still need to try the recommendations NVIDIA gave above.

With respect to just having the USB cable connection and internet to host only, The current JetPack 4.1.1 installation guide indicates that both the OS and the packages would be installed with that configuration. No mention of an additional LAN connection to install the packages.

So, is the 4.1.1 installation guide incorrect? I’m confused as to why things wouldn’t work by following the guide exactly. I don’t have access at work to plug the host or the Xavier into LAN for security reasons.

Thanks!

Scott

Hi,

Sorry for the unclear statement.

For Jetson Xavier, you can use USB Type-C as Ethernet.
Please select the ‘l4tbr0’ for installation.

Thanks.

OK, I followed the instructions above.

Steps 1, 2, and 3 were already complete. The directory was there and the files were in the directory. However, when I ran the script (step 5) I still get the same results. Here is what happens in the terminal:

nvidia@jetson-0423018054460:~/cuda-l4t$ sh cuda-l4t.sh cuda-repo-14t-10-0-local-10.0.117_1.0-1_arm64.deb 10.0 10-0
[sudo] password for nvidia:
dpkg: error: cannot access archive ‘/home/nvidia/cuda-l4t/cuda-repo-14t-10-0-local-10.0.117_1.0-1_arm64.deb’: No such file or directory
OK
Get:1 file:/var/cuda-repo-10-0-local-10.0.117 InRelease
Ign:1 file:/var/cuda-repo-10-0-local-10.0.117 InRelease
Get:2 file:/var/cuda-repo-10-0-local-10.0.117 Release [574 B]
Err:3 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Could not resolve ‘ports.ubuntu.com
Err:4 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease
Could not resolve ‘ports.ubuntu.com
Err:5 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease
Could not resolve ‘ports.ubuntu.com
Err:6 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease
Could not resolve ‘ports.ubuntu.com
Get:2 file:/var/cuda-repo-10-0-local-10.0.117 Release [574 B]
Get:7 file:/var/cuda-repo-10-0-local-10.0.117 Release.gpg [819 B]
Get:7 file:/var/cuda-repo-10-0-local-10.0.117 Release.gpg [819 B]
Reading package lists… Done
E: Release file for file:///var/cuda-repo-10-0-local-10.0.117/Release is not valid yet (invalid for another 197d 1h 33min 44s). Updates for this repository will not be applied.
Reading package lists… Done
Building dependency tree
Reading state information… Done
gnupg is already the newest version (2.2.4-1ubuntu1.1).
You might want to run ‘apt --fix-broken install’ to correct these.
The following packages have unmet dependencies:
libopencv : Depends: libtbb2 but it is not installable
libopencv-dev : Depends: libtbb2 but it is not installable
Depends: libtbb-dev but it is not installable
E: Unmet dependencies. Try ‘apt --fix-broken install’ with no packages (or specify a solution).
Reading package lists… Done
Building dependency tree
Reading state information… Done
E: Unable to locate package cuda-toolkit-10-0
E: Unable to locate package libfreeimage-dev
E: Unable to locate package libopenmpi-dev
E: Unable to locate package openmpi-bin

Hi,

Thanks for the update.

We are checking this with our internal team.
Will update information with you later.

Sorry for any inconvenience.

Hi sidener2002

It seems like we are having a network problem when the device trying to download cuda through the host internet.
There is a page like this in attachment when you choose the usb mode. Please make sure you selected the right port that has access to the internet.
Also, it would be great you can do “ip addr” and “ip route” on a separated terminal, at the flashing step, and share us the result. So we can understand more of you situation.

Thanks,
Dongbo

Hi, I removed JetPack and re-installed everything from scratch. Selecting to install everything available. It finally worked! Your screenshot showing internet eno1 was the key I think. I had been leaving that default, but my host computer is only connected to the internet by wifi. So, I used wld3s0 as the internet connection and then everything worked. It would have been nice for JetPack to throw out an error if I tried to use a non-existent internet connection.

Thanks so much for your help! I think I am on track now. Looking forward to running some models on the Xavier in Python.

Scott

Hi sindener2002,
Glad this worked for you.
Your suggestion is actually really good. We will make sure add something to avoid people choose non-existent internet network interface.

Thanks.
Dongbo

I have the same problem, already spent 5 hours (I have TX2 experience) and I am still struggling on this. Just Curious that Why Nvidia does not choose a two-step process to install Jetpack? Step one just flushes the L4T OS via host machine, and step 2 installs other pkgs only from Xaiver. In that way, the problem can be simplified and save us lots of time to repeatedly flush the OS.

@shunguang you may use flash.sh instead of jetpack for flashing as a first step and use jetpack post install as a second step, for example

@Andrey1984:

I eventually solved the problem by following approach:

  1. Flashing the target os first from the host
  2. logging into Xaviver, manually insatll CUDA by
mkdir cuda-l4t
Copy cuda-repo-l4t-10-0-local-10.0.117_1.0-1_arm64.deb to cuda-l4t folder
Copy cuda-l4t.sh to cuda-l4t folder
cd cuda-l4t
sh cuda-l4t.sh cuda-repo-l4t-10-0-local-10.0.117_1.0-1_arm64.deb 10.0 10-0
  1. Going back to “host-target” architecture, but pull off the usb cabe, and let both machines connect to a router, magically the rest of the pkgs are successfully installed.

Here we should blame Nvidia that the OS is not pre-installed. It just so wired.