ERROR: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

After I install nvidia_367 and libcuda1-367. Jeston TK1 can’t run any cuda program.

root@n2:/usr/local/cuda/samples/bin/armv7l/linux/release/gnueabihf# ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL

root@n2:~# nvidia-smi
NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

root@n2:~# modprobe nvidia-367
modprobe: FATAL: Module nvidia-367 not found.

how to solve it?

For the Jetson products the video driver from desktop PCs cannot be used (and “367” sounds like a desktop PC driver). Are you trying to add a new video driver version to a Jetson TK1? If so, then those other drivers won’t load, the GPU on the Jetson is directly attached to the memory controller (and the other drivers only work for PCI cards). If you are looking for a video or GPU access function which relies on PCI, then you you are out of luck on the Jetson. Also, desktop drivers are for x86_64 architecture, the JTK1 is armhf.

No.nvidia-367 is for armhf. I install it from official repo of NV via apt-get.

az@n2:~$ dpkg -l |grep nvidia
ii nvidia-367 367.57-0ubuntu0.14.04.1 armhf NVIDIA binary driver - version 367.57

Why I update drivers to this version is for the problem: https://devtalk.nvidia.com/default/topic/988913/a-catastrophic-error-detected-in-the-compilation-of-cuckoohash-/

It’s seem the problem related to the GPU driver?

I’m using CUDA 6.5 which for L4T 21.2 ARMv7 32-bit,which is download from https://developer.nvidia.com/cuda-toolkit-65 , in this page, the Q&A said:

Q: Will the installer replace the driver currently installed on my system?
A: The installer will provide an option to install the included driver, and if selected, it will replace the driver currently on your system.

so I’m updated the driver provided in the repo. Is that OK?

Don’t forget that JetPack offers to install packages on both host and Jetson…so if you installed via JetPack be sure to differentiate between destinations of host and Jetson. I use Fedora so I can’t use JetPack…not sure if it lists a version like 367 in relation to the armhf version, but I suppose Jetson must have some version, so maybe that applies to both host and Jetson. I think it is package “mesa-utils” that provides program “glxinfo”, but I’d run this on both host and Jetson to see what it says (make sure it is a local login to the Jetson and not remote or you might not get correct results):

glxinfo | egrep -i nvidia

I install it just in TK1, NOT host.

You can verify if the hardware accelerated version of files are in place via:

sha1sum -c /etc/nv_tegra_release

Does graphical display work at all on the JTX1? If so, then the hardware accelerated files must be in place because the Jetson does not have graphical mode without it.

There are also cases where CUDA may not work even if files are in place if there is no rendering context to work with (meaning not in graphical mode…I don’t think a monitor has to exist, but the Xorg X11 server must run even if no HDMI is connected), but first see if the files exist.

nvidia-smi is not available for Jetson. This combined with the clue that user installed a GPU driver from apt-get, means the install is likely invalid. Please re-flash your Jetson with the latest JetPack-L4T.

Note that a GPU driver package from Ubuntu repo should not be used for Jetson. Linuxdev is correct that this sounds like the desktop driver, not the correct driver for ARM from NVIDIA. That package should probably not be in their repo. My guess is that while they were migrating to armhf and re-compiling all their deb’s, the 367 package was amongst those that built ok, and was added to the repo without them realizing it shouldn’t have been. This is likely the PCIe desktop driver, whereas Jetson uses a userspace driver since the GPU is integrated.

Sorry for the confusion - please use the latest JetPack-L4T to install the drivers and CUDA toolkit.

If nvidia-smi is not available for Jetson, then how can we check devices status when running cuda program? any alternative?

I means,checking devices memory available,etc.

Can you use cudaMemGetInfo() API function?