Cupy installation on Nvidia Drive AGX Xavier

Hi,
I am using cupy in a ROS package. It is working on desktop and am trying to implement that on a NVIDIA Drive AGX Xavier. I am currently trying to install Cupy from source using the command pip install cupy. I set the environment variable CUDA_PATH=/usr/local/cuda-10.2. I get the following error

Collecting cupy
  Downloading https://files.pythonhosted.org/packages/79/12/68cc4dabe0b7d7bf69716459acc72820f747fc2b3ab63b145e0fd35b8658/cupy-7.1.1.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 208kB/s
    Complete output from command python setup.py egg_info:
    Options: {'profile': False, 'annotate': False, 'linetrace': False, 'wheel_libs': [], 'package_name': 'cupy', 'no_rpath': False, 'no_cuda': False, 'wheel_includes': [], 'long_description': None, 'use_hip': False}
    **************************************************
    *** WARNING: nvcc path != CUDA_PATH
    *** WARNING: nvcc path: None
    *** WARNING: CUDA_PATH: /usr/local/cuda-10.2
    **************************************************

    -------- Configuring Module: cuda --------
    /tmp/tmpAWyvrr/a.cpp:5:10: fatal error: cufft.h: No such file or directory
     #include <cufft.h>
              ^~~~~~~~~
    compilation terminated.
    command 'aarch64-linux-gnu-gcc' failed with exit status 1

    ************************************************************
    * CuPy Configuration Summary                               *
    ************************************************************

    Build Environment:
      Include directories: ['/usr/local/cuda-10.2/include']
      Library directories: ['/usr/local/cuda-10.2/lib64']
      nvcc command       : ['/usr/local/cuda-10.2/bin/nvcc']

    Environment Variables:
      CFLAGS          : (none)
      LDFLAGS         : (none)
      LIBRARY_PATH    : (none)
      CUDA_PATH       : /usr/local/cuda-10.2
      NVTOOLSEXT_PATH : (none)
      NVCC            : (none)
      ROCM_HOME       : (none)

    Modules:
      cuda      : No
        -> Include files not found: ['cublas_v2.h', 'cuda.h', 'cuda_profiler_api.h', 'cuda_runtime.h', 'cufft.h', 'curand.h', 'cusparse.h', 'nvrtc.h']
        -> Check your CFLAGS environment variable.

    ERROR: CUDA could not be found on your system.
    Please refer to the Installation Guide for details:
    https://docs-cupy.chainer.org/en/stable/install.html

    ************************************************************

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-c0hpdC/cupy/setup.py", line 128, in <module>
        ext_modules = cupy_setup_build.get_ext_modules()
      File "cupy_setup_build.py", line 744, in get_ext_modules
        extensions = make_extensions(arg_options, compiler, use_cython)
      File "cupy_setup_build.py", line 492, in make_extensions
        raise Exception('Your CUDA environment is invalid. '
    Exception: Your CUDA environment is invalid. Please check above error log.


    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-c0hpdC/cupy/

I do not see cufft.h in /usr/local/cuda-10.2/include. Am I missing some installation? Have anyone used Cupy on Drive AGX?

You should be able to find it in the installed DRIVE OS directory (~/nvidia/nvidia_sdk/).

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

On the target, I do not see ~/nvidia/nvidia_sdk. On the host, I have ~/nvidia/nvidia_sdk, which has deb files for cuda etc… How do I compile cupy? Do I have to cross compile and transfer the binaries to target? Do you have any documentation on cross compiling using pip?

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

It is recommended to use cross-compilation as the development process for DRIVE OS. There is no documentation available on cross-compiling using pip.

Since I cannot cross compile using pip, I will have to build on the target. Currently, there are no cufft.h etc header files on the target. How can I install these on the target machine?

I will consult with our team and provide you with an update.

Upon checking, we are using DRIVE OS 5.1.6. Please provide an update for this OS version

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.9.1.10844
other

Host Machine Version
native Ubuntu 18.04
other

The header file can be found on the target system at the specified location.

/usr/local/cuda-10.2/targets/aarch64-linux/include/cufft.h


I dont seem to have the cufft.h header file in that location.

Could you confirm if you are on DRIVE OS 5.2.6? To assist in our investigation, could you run the following commands and send us the results?

$ cat /etc/nvidia/version-ubuntu-rootfs.txt
$ ls -l /etc/nvidia/rootfilesystem-manifest/

My DRIVE OS is 5.1.6. I had corrected it in one of the replies. Sorry for the misunderstanding. The outputs are as given below

Also, I copied the headers and libs from the host machine (/usr/local/cuda-10.2/targets/aarch64-linux) and copied it to /home/nvidia/target and tried to compile the cupy using the command given below.

CUDA_PATH="/usr/local/cuda-10.2" CFLAGS="-I/usr/local/cuda/include -I/home/nvidia/target/aarch64-linux/include " LDFLAGS="-L/usr/local/cuda/lib64 -L/usr/lib -L/home/nvidia/target/aarch64-linux/lib/stubs" pip3 install cupy --no-cache-dir -vvvv

I have attached the verbose output for your reference
output.txt (108.0 KB)

I also tried to download the source code from github (GitHub - cupy/cupy: NumPy & SciPy for GPU) - v9, and I get the same error.

It appears you gave us incorrect information about being on DRIVE OS 5.2.6.
We do not have experience with installing CuPy on DRIVE OS, and are unsure if manually copying files would be sufficient. Perhaps you could try using DRIVE OS 5.2.6.
In the meantime, you may want to seek help regarding the error log on the Github repository.

Hi, I raised this issue at the cupy github repository ( Installation of CuPy on NVIDIA Drive AGX Xavier · Issue #7372 · cupy/cupy (github.com)). They suggested to use higher version of python for which aarch64 wheel is available. I installed python 3.8 and installed cupy using available binary package for cuda 10.2 (Installation — CuPy 11.5.0 documentation).
After installation, when I try to import cupy, I get the following error regarding cusolver. Can you please suggest a way to resolve this?

Python 3.8.0 (default, Dec 9 2021, 17:53:27)
[GCC 8.4.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
Traceback (most recent call last):
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/init.py”, line 18, in
from cupy import _core # NOQA
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/_core/init.py”, line 3, in
from cupy._core import core # NOQA
File “cupy/_core/core.pyx”, line 1, in init cupy._core.core
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/cuda/init.py”, line 8, in
from cupy.cuda import compiler # NOQA
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/cuda/compiler.py”, line 13, in
from cupy.cuda import device
File “cupy/cuda/device.pyx”, line 9, in init cupy.cuda.device
ImportError: /home/nvidia/.local/lib/python3.8/site-packages/cupy_backends/cuda/libs/cusolver.cpython-38-aarch64-linux-gnu.so: symbol cusolverDnZZgesv_bufferSize version libcusolver.so.10 not defined in file libcusolver.so.10 with link time reference
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “”, line 1, in
File “/home/nvidia/.local/lib/python3.8/site-packages/cupy/init.py”, line 20, in
raise ImportError(f’‘’
ImportError:

Failed to import CuPy.

If you installed CuPy via wheels (cupy-cudaXXX or cupy-rocm-X-X), make sure that the package matches with the version of CUDA or ROCm installed.

On Linux, you may need to set LD_LIBRARY_PATH environment variable depending on how you installed CUDA/ROCm.
On Windows, try setting CUDA_PATH environment variable.

Check the Installation Guide for details:
https://docs.cupy.dev/en/latest/install.html
Original error:
ImportError: /home/nvidia/.local/lib/python3.8/site-packages/cupy_backends/cuda/libs/cusolver.cpython-38-aarch64-linux-gnu.so: symbol cusolverDnZZgesv_bufferSize version libcusolver.so.10 not defined in file libcusolver.so.10 with link time reference

The libcusolver.so.10 is currently in /usr/local/cuda-10.2/targets/aarch64-linux/lib and I have added this the LD_LIBRARY_PATH variable.

Dear @gibin.zachariah,
cusolverDnZZgesv_bufferSize version libcusolver.so.10 not defined in file libcusolver.so.10 with link time reference

It seems this function is missing the installed cusolver library.

Are you using any Driveworks NV DNNs/Mapping modules in your application development which are not part of driveworks release? If not, I can give a try installing cupy from source in DRIVE OS 5.2.6

Dear @gibin.zachariah,
I could install cupy on DRIVE OS 5.2.6

nvidia@tegra-ubuntu:~$ pip install cupy
Collecting cupy
Downloading https://files.pythonhosted.org/packages/79/12/68cc4dabe0b7d7bf69716459acc72820f747fc2b3ab63b145e0fd35b8658/cupy-7.1.1.tar.gz (3.7MB)
100% |████████████████████████████████| 3.7MB 186kB/s
Collecting fastrlock>=0.3 (from cupy)
Downloading https://files.pythonhosted.org/packages/84/64/7d4f96734aba053194877f0b342a3a57402d2b2215aaae39b0cd619a326d/fastrlock-0.8.1.tar.gz (75kB)
100% |████████████████████████████████| 81kB 5.1MB/s
Collecting numpy>=1.9.0 (from cupy)
Downloading https://files.pythonhosted.org/packages/b7/6f/24647f014eef9b67a24adfcbcd4f4928349b4a0f8393b3d7fe648d4d2de3/numpy-1.16.6.zip (5.1MB)
100% |████████████████████████████████| 5.2MB 164kB/s
Collecting six>=1.9.0 (from cupy)
Downloading https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl
Building wheels for collected packages: cupy, fastrlock, numpy
Running setup.py bdist_wheel for cupy … done
Stored in directory: /home/nvidia/.cache/pip/wheels/72/63/2a/7635c93abc95360cfa1d8a77b6e5e70e2701b593668350c81a
Running setup.py bdist_wheel for fastrlock … done
Stored in directory: /home/nvidia/.cache/pip/wheels/a8/1a/5d/93c57426816bddeec880bd7e408b1ec2e0ee5af61283037154
Running setup.py bdist_wheel for numpy … done
Stored in directory: /home/nvidia/.cache/pip/wheels/cb/c2/c1/d99e8bf789c8dd07623af6be95e6a89984c85a05e31b8513c3
Successfully built cupy fastrlock numpy
Installing collected packages: fastrlock, numpy, six, cupy
Successfully installed cupy-7.1.1 fastrlock-0.8.1 numpy-1.16.6 six-1.16.0

Thank you for the confirmation. Can you please check if you are able to import cupy in python? Previously when I installed a wheel, it was throwing an error when importing. Can you also please confirm the python version used?

>>python3
>> import cupy

Dear @gibin.zachariah,
I hit the same issue as CUPY 7.0 fails with Python 2.7 · Issue #2763 · cupy/cupy · GitHub

are you running with python3?

Dear @gibin.zachariah,
Please check below output

nvidia@tegra-ubuntu:~$ pip3 install cupy
Collecting cupy
Collecting numpy<1.24,>=1.17 (from cupy)
  Using cached https://files.pythonhosted.org/packages/51/60/3f0fe5b7675a461d96b9d6729beecd3532565743278a9c3fe6dd09697fa7/numpy-1.19.5.zip
Collecting fastrlock>=0.5 (from cupy)
Building wheels for collected packages: numpy
  Running setup.py bdist_wheel for numpy ... done
  Stored in directory: /home/nvidia/.cache/pip/wheels/ee/cd/78/686734467766f26e3d2b42605dce9fdb7d24c1c1d26f2fb8fc
Successfully built numpy
Installing collected packages: numpy, fastrlock, cupy
Successfully installed cupy-9.6.0 fastrlock-0.8.1 numpy-1.19.5

 nvidia@tegra-ubuntu:~$ python3
Python 3.6.9 (default, Nov 25 2022, 14:10:45)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cupy as p
>>>

thank you for the confirmation. We will upgrade to 5.2.6 and try .