Cross-compile samples for DPX2(4.1.8.0L_SDK) fails with "cuda_runtime.h: No such file or directory"

Dear Moderator,
I have followed the steps from both /usr/local/driveworks-0.3/doc/nvdwx_html/dwx_samples_building.html and https://devtalk.nvidia.com/default/topic/997207/driveworks/driveworks-0-2-1-build-sample-but-cannot-run-on-px2. My cmake runs without any issues. However, when I ran make -j command, it fails with “cuda_runtime.h: No such file or directory”. I started debugging this issue and found cuda_runtime.h do exist in the system. I am stuck with this issue for the past two days. Kindly can you help me to solve this problem.?

I will really appreciate your effort and time.

user@qpu-272:/usr/local/driveworks-0.3$ sudo mkdir build
user@qpu-272:/usr/local/driveworks-0.3$ cd build

user@qpu-272:/usr/local/driveworks-0.3/build$ pwd
/usr/local/driveworks-0.3/build
user@qpu-272:/usr/local/driveworks-0.3/build$ ls
user@qpu-272:/usr/local/driveworks-0.3/build$ sudo cmake -DCMAKE_BUILD_TYPE=Release -DVIBRANTE_PDK:STRING=/home/user/Downloads/NVIDIA/DRIVEINSTALL/VibranteSDK/vibrante-t186ref-linux -DCMAKE_TOOLCHAIN_FILE=/usr/local/driveworks-0.3//samples/cmake/SamplesInstallConfiguration.cmake /usr/local/driveworks-0.3/samples/src/hello_world
– Driveworks Samples install dir:
– Driveworks Samples install dir:
– The C compiler identification is GNU 4.8.4
– The CXX compiler identification is GNU 4.8.4
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Configuring done
– Generating done
CMake Warning:
Manually-specified variables were not used by the project:

VIBRANTE_PDK

– Build files have been written to: /usr/local/driveworks-0.3/build

user@qpu-272:/usr/local/driveworks-0.3/build$ ls
CMakeCache.txt CMakeFiles cmake_install.cmake Makefile
user@qpu-272:/usr/local/driveworks-0.3/build$

user@qpu-272:/usr/local/driveworks-0.3/build$ sudo make -j VERBOSE=1
/usr/local/bin/cmake -H/usr/local/driveworks-0.3/samples/src/hello_world -B/usr/local/driveworks-0.3/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/bin/cmake -E cmake_progress_start /usr/local/driveworks-0.3/build/CMakeFiles /usr/local/driveworks-0.3/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory /usr/local/driveworks-0.3/build' make -f CMakeFiles/sample_hello_world.dir/build.make CMakeFiles/sample_hello_world.dir/depend make[2]: Entering directory /usr/local/driveworks-0.3/build’
cd /usr/local/driveworks-0.3/build && /usr/local/bin/cmake -E cmake_depends “Unix Makefiles” /usr/local/driveworks-0.3/samples/src/hello_world /usr/local/driveworks-0.3/samples/src/hello_world /usr/local/driveworks-0.3/build /usr/local/driveworks-0.3/build /usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/DependInfo.cmake --color=
Dependee “/usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/DependInfo.cmake” is newer than depender “/usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/depend.internal”.
Dependee “/usr/local/driveworks-0.3/build/CMakeFiles/CMakeDirectoryInformation.cmake” is newer than depender “/usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/depend.internal”.
Scanning dependencies of target sample_hello_world
make[2]: Leaving directory /usr/local/driveworks-0.3/build' make -f CMakeFiles/sample_hello_world.dir/build.make CMakeFiles/sample_hello_world.dir/build make[2]: Entering directory /usr/local/driveworks-0.3/build’
/usr/local/bin/cmake -E cmake_progress_report /usr/local/driveworks-0.3/build/CMakeFiles 1
[100%] Building CXX object CMakeFiles/sample_hello_world.dir/main.cpp.o
/usr/bin/c++ -O3 -DNDEBUG -o CMakeFiles/sample_hello_world.dir/main.cpp.o -c /usr/local/driveworks-0.3/samples/src/hello_world/main.cpp
/usr/local/driveworks-0.3/samples/src/hello_world/main.cpp:39:26: fatal error: cuda_runtime.h: No such file or directory
#include <cuda_runtime.h>
^
compilation terminated.
make[2]: *** [CMakeFiles/sample_hello_world.dir/main.cpp.o] Error 1
make[2]: Leaving directory /usr/local/driveworks-0.3/build' make[1]: *** [CMakeFiles/sample_hello_world.dir/all] Error 2 make[1]: Leaving directory /usr/local/driveworks-0.3/build’
make: *** [all] Error 2
user@qpu-272:/usr/local/driveworks-0.3/build$

user@qpu-272:/usr/local/driveworks-0.3/build$ which nvcc
/usr/local/cuda-8.0/bin/nvcc
user@qpu-272:/usr/local/driveworks-0.3/build$ sudo find / -name nvcc > ~/t
user@qpu-272:/usr/local/driveworks-0.3/build$ cat ~/t
/home/user/Downloads/NVIDIA/SAMPLES/usr/local/cuda-8.0/bin/nvcc
/usr/local/cuda-8.0/bin/nvcc
user@qpu-272:/usr/local/driveworks-0.3/build$ echo PATH /usr/local/cuda-8.0/bin:/opt/ros/indigo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games user@qpu-272:/usr/local/driveworks-0.3/build
user@qpu-272:/usr/local/driveworks-0.3/build$ find /usr/local -name cuda_runtime.h
/usr/local/cuda-8.0/targets/aarch64-linux/include/cuda_runtime.h
/usr/local/cuda-8.0/targets/x86_64-linux/include/cuda_runtime.h
user@qpu-272:/usr/local/driveworks-0.3/build$
user@qpu-272:/usr/local/driveworks-0.3/build$ echo LD_LIBRARY_PATH /usr/local/cuda-8.0/lib64:/opt/ros/indigo/lib user@qpu-272:/usr/local/driveworks-0.3/build
user@qpu-272:/usr/local/driveworks-0.3/build$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
user@qpu-272:/usr/local/driveworks-0.3/build$

Dear SamJey,

Did you setup cuda environment completely? if not, please refer to below setup process and try it again. Thanks.

$gedit ~/.bashrc
export PATH=/usr/local/cuda-8.0/bin/:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib:$LD_LIBRARY_PATH

source ~/.bashrc nvcc --version

$ /usr/local/cuda-8.0/bin/cuda-install-samples-8.0.sh ~/

$cd ~/NVIDIA_CUDA8.0_Samples/1_Utilities/deviceQuery
make ./deviceQuery

Dear Steve,
Thank you very much for your kind reply.

I followed the steps provided in your reply. It still failed with the same error.

Are these CUDO 8.0 and Driver version 340.102 compatible with each other? This is because I am running driver version 340.102 with a GPU named “Graphics Device”. I have also provided the output the nvidia-smi command at the end of this reply. I googled but I am bit confused as people are saying diff answers.

Could you please provide your guidance on what went wrong?

user@qpu-286:/usr/local$ echo $LD_LIBRARY_PATH
/usr/local/cuda/targets/aarch64-linux/lib:/opt/ros/indigo/lib

user@qpu-286:/usr/local$ echo $PATH
/usr/local/cuda-8.0/bin:/opt/ros/indigo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

user@qpu-286:/usr/local$ nvcc --version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_16:30:19_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72

user@qpu-286:/usr/local$ sudo /usr/local/cuda-8.0/bin/cuda-install-samples-8.0.sh ~/
[sudo] password for user:
Copying samples to /home/user/NVIDIA_CUDA-8.0_Samples now…
Finished copying samples.

user@qpu-286:/usr/local$ find . -name deviceQuery
./cuda-8.0/samples/1_Utilities/deviceQuery

user@qpu-286:/usr/local$ cd ./cuda-8.0/samples/1_Utilities/deviceQuery

user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ ls
deviceQuery.cpp Makefile NsightEclipse.xml readme.txt

user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ sudo make
/usr/local/cuda-8.0/bin/nvcc -ccbin g++ -I…/…/common/inc -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_60,code=compute_60 -o deviceQuery.o -c deviceQuery.cpp
nvcc warning : The ‘compute_20’, ‘sm_20’, and ‘sm_21’ architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
/usr/local/cuda-8.0/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_60,code=compute_60 -o deviceQuery deviceQuery.o
nvcc warning : The ‘compute_20’, ‘sm_20’, and ‘sm_21’ architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
mkdir -p …/…/bin/x86_64/linux/release
cp deviceQuery …/…/bin/x86_64/linux/release
user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ sudo ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL
user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$

user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ nvidia-smi
Thu Sep 14 18:00:41 2017
±-----------------------------------------------------+
| NVIDIA-SMI 340.102 Driver Version: 340.102 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Graphics Device Off | 0000:01:00.0 N/A | N/A |
| 50% 49C P8 N/A / N/A | 357MiB / 2047MiB | N/A Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 Not Supported |
±----------------------------------------------------------------------------+
user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$

I will really appreciate your help.

Thanks
Sam

Dear Steve,

It seems CUDA 8.0 is not compatible with driver 340.102 version. Uninstalling the old one and installing the 371 fixed the problem and I am able to run deviceQuery without any issues.

I am back to the old problem now.

user@qpu-286:/usr/local/driveworks-0.3/build$ sudo make -j
[100%] Building CXX object CMakeFiles/sample_hello_world.dir/main.cpp.o
/usr/local/driveworks-0.3/samples/src/hello_world/main.cpp:39:26: fatal error: cuda_runtime.h: No such file or directory
#include <cuda_runtime.h>
^
compilation terminated.
make[2]: *** [CMakeFiles/sample_hello_world.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/sample_hello_world.dir/all] Error 2
make: *** [all] Error 2
user@qpu-286:/usr/local/driveworks-0.3/build$

user@qpu-286:/usr/local/driveworks-0.3/build$ nvcc --version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_16:30:19_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72
user@qpu-286:/usr/local/driveworks-0.3/build$

Any hint will be appreciated?

Thanks
Sam

Actually there is some include path is missing. This is because, when I ran another sample program, it gave me a diff error.

user@qpu-286:/usr/local/driveworks-0.3/build$ sudo make -j
– Driveworks Samples install dir: /usr/local
– Configuring done
– Generating done
– Build files have been written to: /usr/local/driveworks-0.3/build
Scanning dependencies of target sample_ipc_socketclientserver
[100%] Building CXX object CMakeFiles/sample_ipc_socketclientserver.dir/sample_ipc_socketclientserver.cpp.o
/usr/local/driveworks-0.3/samples/src/ipc/sample_ipc_socketclientserver.cpp:31:29: fatal error: dw/core/Context.h: No such file or directory
#include <dw/core/Context.h>
^
compilation terminated.

In case someone needs, besides PATH and LD_LIBRARY_PATH, you also need to set CPATH:

export CPATH=/usr/local/cuda-10.1/targets/x86_64-linux/include:$CPATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export PATH=/usr/local/cuda-10.1/bin:$PATH
1 Like