Cupy installation on Nvidia Drive AGX Xavier

Hi,
I am using cupy in a ROS package. It is working on desktop and am trying to implement that on a NVIDIA Drive AGX Xavier. I am currently trying to install Cupy from source using the command pip install cupy. I set the environment variable CUDA_PATH=/usr/local/cuda-10.2. I get the following error

Collecting cupy
  Downloading https://files.pythonhosted.org/packages/79/12/68cc4dabe0b7d7bf69716459acc72820f747fc2b3ab63b145e0fd35b8658/cupy-7.1.1.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 208kB/s
    Complete output from command python setup.py egg_info:
    Options: {'profile': False, 'annotate': False, 'linetrace': False, 'wheel_libs': [], 'package_name': 'cupy', 'no_rpath': False, 'no_cuda': False, 'wheel_includes': [], 'long_description': None, 'use_hip': False}
    **************************************************
    *** WARNING: nvcc path != CUDA_PATH
    *** WARNING: nvcc path: None
    *** WARNING: CUDA_PATH: /usr/local/cuda-10.2
    **************************************************

    -------- Configuring Module: cuda --------
    /tmp/tmpAWyvrr/a.cpp:5:10: fatal error: cufft.h: No such file or directory
     #include <cufft.h>
              ^~~~~~~~~
    compilation terminated.
    command 'aarch64-linux-gnu-gcc' failed with exit status 1

    ************************************************************
    * CuPy Configuration Summary                               *
    ************************************************************

    Build Environment:
      Include directories: ['/usr/local/cuda-10.2/include']
      Library directories: ['/usr/local/cuda-10.2/lib64']
      nvcc command       : ['/usr/local/cuda-10.2/bin/nvcc']

    Environment Variables:
      CFLAGS          : (none)
      LDFLAGS         : (none)
      LIBRARY_PATH    : (none)
      CUDA_PATH       : /usr/local/cuda-10.2
      NVTOOLSEXT_PATH : (none)
      NVCC            : (none)
      ROCM_HOME       : (none)

    Modules:
      cuda      : No
        -> Include files not found: ['cublas_v2.h', 'cuda.h', 'cuda_profiler_api.h', 'cuda_runtime.h', 'cufft.h', 'curand.h', 'cusparse.h', 'nvrtc.h']
        -> Check your CFLAGS environment variable.

    ERROR: CUDA could not be found on your system.
    Please refer to the Installation Guide for details:
    https://docs-cupy.chainer.org/en/stable/install.html

    ************************************************************

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-c0hpdC/cupy/setup.py", line 128, in <module>
        ext_modules = cupy_setup_build.get_ext_modules()
      File "cupy_setup_build.py", line 744, in get_ext_modules
        extensions = make_extensions(arg_options, compiler, use_cython)
      File "cupy_setup_build.py", line 492, in make_extensions
        raise Exception('Your CUDA environment is invalid. '
    Exception: Your CUDA environment is invalid. Please check above error log.


    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-c0hpdC/cupy/

I do not see cufft.h in /usr/local/cuda-10.2/include. Am I missing some installation? Have anyone used Cupy on Drive AGX?

You should be able to find it in the installed DRIVE OS directory (~/nvidia/nvidia_sdk/).

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

On the target, I do not see ~/nvidia/nvidia_sdk. On the host, I have ~/nvidia/nvidia_sdk, which has deb files for cuda etc… How do I compile cupy? Do I have to cross compile and transfer the binaries to target? Do you have any documentation on cross compiling using pip?

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

It is recommended to use cross-compilation as the development process for DRIVE OS. There is no documentation available on cross-compiling using pip.

Since I cannot cross compile using pip, I will have to build on the target. Currently, there are no cufft.h etc header files on the target. How can I install these on the target machine?

I will consult with our team and provide you with an update.

Upon checking, we are using DRIVE OS 5.1.6. Please provide an update for this OS version

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

The header file can be found on the target system at the specified location.

/usr/local/cuda-10.2/targets/aarch64-linux/include/cufft.h


I dont seem to have the cufft.h header file in that location.

Could you confirm if you are on DRIVE OS 5.2.6? To assist in our investigation, could you run the following commands and send us the results?

$ cat /etc/nvidia/version-ubuntu-rootfs.txt
$ ls -l /etc/nvidia/rootfilesystem-manifest/

My DRIVE OS is 5.1.6. I had corrected it in one of the replies. Sorry for the misunderstanding. The outputs are as given below

Also, I copied the headers and libs from the host machine (/usr/local/cuda-10.2/targets/aarch64-linux) and copied it to /home/nvidia/target and tried to compile the cupy using the command given below.

CUDA_PATH="/usr/local/cuda-10.2" CFLAGS="-I/usr/local/cuda/include -I/home/nvidia/target/aarch64-linux/include " LDFLAGS="-L/usr/local/cuda/lib64 -L/usr/lib -L/home/nvidia/target/aarch64-linux/lib/stubs" pip3 install cupy --no-cache-dir -vvvv

I have attached the verbose output for your reference
output.txt (108.0 KB)

I also tried to download the source code from github (GitHub - cupy/cupy: NumPy & SciPy for GPU) - v9, and I get the same error.

It appears you gave us incorrect information about being on DRIVE OS 5.2.6.
We do not have experience with installing CuPy on DRIVE OS, and are unsure if manually copying files would be sufficient. Perhaps you could try using DRIVE OS 5.2.6.
In the meantime, you may want to seek help regarding the error log on the Github repository.

Hi, I raised this issue at the cupy github repository ( Installation of CuPy on NVIDIA Drive AGX Xavier · Issue #7372 · cupy/cupy (github.com)). They suggested to use higher version of python for which aarch64 wheel is available. I installed python 3.8 and installed cupy using available binary package for cuda 10.2 (Installation — CuPy 11.5.0 documentation).
After installation, when I try to import cupy, I get the following error regarding cusolver. Can you please suggest a way to resolve this?

Python 3.8.0 (default, Dec 9 2021, 17:53:27)
[GCC 8.4.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
Traceback (most recent call last):
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/init.py”, line 18, in
from cupy import _core # NOQA
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/_core/init.py”, line 3, in
from cupy._core import core # NOQA
File “cupy/_core/core.pyx”, line 1, in init cupy._core.core
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/cuda/init.py”, line 8, in
from cupy.cuda import compiler # NOQA
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/cuda/compiler.py”, line 13, in
from cupy.cuda import device
File “cupy/cuda/device.pyx”, line 9, in init cupy.cuda.device
ImportError: /home/nvidia/.local/lib/python3.8/site-packages/cupy_backends/cuda/libs/cusolver.cpython-38-aarch64-linux-gnu.so: symbol cusolverDnZZgesv_bufferSize version libcusolver.so.10 not defined in file libcusolver.so.10 with link time reference
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “”, line 1, in
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/init.py”, line 20, in
raise ImportError(f’‘’
ImportError:

Failed to import CuPy.

If you installed CuPy via wheels (cupy-cudaXXX or cupy-rocm-X-X), make sure that the package matches with the version of CUDA or ROCm installed.

On Linux, you may need to set LD_LIBRARY_PATH environment variable depending on how you installed CUDA/ROCm.
On Windows, try setting CUDA_PATH environment variable.

Check the Installation Guide for details:
https://docs.cupy.dev/en/latest/install.html
Original error:
ImportError: /home/nvidia/.local/lib/python3.8/site-packages/cupy_backends/cuda/libs/cusolver.cpython-38-aarch64-linux-gnu.so: symbol cusolverDnZZgesv_bufferSize version libcusolver.so.10 not defined in file libcusolver.so.10 with link time reference

The libcusolver.so.10 is currently in /usr/local/cuda-10.2/targets/aarch64-linux/lib and I have added this the LD_LIBRARY_PATH variable.