Hi,
In our project we decided to make use of Thrust not just in CUDA code, but also in general C++ (and even in other accelerator language) source files. To do this reliably, we download Thrust as part of our project. With code like:
# Declare where to get Thrust from.
set( TRACCC_THRUST_SOURCE
"GIT_REPOSITORY;https://github.com/NVIDIA/thrust.git;GIT_TAG;1.15.0"
CACHE STRING "Source for Thrust, when built as part of this project" )
mark_as_advanced( TRACCC_THRUST_SOURCE )
FetchContent_Declare( Thrust ${TRACCC_THRUST_SOURCE} )
# Options used in the build of Thrust.
set( THRUST_ENABLE_INSTALL_RULES TRUE CACHE BOOL
"Install Thrust together with this project" )
# Get it into the current directory.
FetchContent_MakeAvailable( Thrust )
This worked well so far, we’ve been able to use Thrust objects to communicate between host and device code.
Recently I switched to using a very new version of LLVM as the host compiler. And the latest LLVM version became vocal about some intrinsics that Thrust includes. Which highlighted that we need to use Thrust headers as “system headers” in our build to avoid getting warnings from them.
After I did that, the warnings in our build are gone for all C++ (and other kinds of non-CUDA) source files. But when building .cu files, we see things like this:
[ 78%] Building CUDA object device/cuda/CMakeFiles/traccc_cuda.dir/src/cca/component_connection.cu.o
cd /mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/device/cuda && \
/software/cuda/11.7.1/x86_64/bin/nvcc -forward-unknown-to-host-compiler \
-ccbin=/software/intel/clang-2022-09/x86_64-centos9-gcc11-opt/bin/clang++ \
-DALGEBRA_PLUGINS_INCLUDE_ARRAY -DDETRAY_CUSTOM_SCALARTYPE=float \
-DJSON_USE_IMPLICIT_CONVERSIONS=1 -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA \
-DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTRACCC_CUSTOM_SCALARTYPE=float \
-DVECMEM_DEBUG_MSG_LVL=0 -DVECMEM_HAVE_PMR_MEMORY_RESOURCE -DVECMEM_HAVE_SYCL_ATOMIC_REF \
-DVECMEM_SOURCE_DIR_LENGTH=57 \
-DVECMEM_SYCL_PRINTF_FUNCTION=cl::sycl::ext::oneapi::experimental::printf -Dtraccc_cuda_EXPORTS \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/device/cuda/include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/core/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/vecmem-build/core/CMakeFiles \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/vecmem-src/core/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/detray-src/core/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/thrust-src/dependencies/cub \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/acts-src/Core/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/acts-build/Core \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/acts-src/Plugins/Json/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/acts-src/thirdparty/nlohmann_json/single_include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/plugins/algebra/include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/plugins/algebra/array/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/frontend/array_cmath/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/common/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/storage/array/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/math/cmath/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/math/common/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/detray-src/plugins/algebra/array/include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/plugins/algebra/vecmem/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/frontend/vecmem_cmath/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/storage/vecmem/include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/plugins/algebra/eigen/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/frontend/eigen_eigen/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/storage/eigen/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/math/eigen/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/detray-src/plugins/algebra/eigen/include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/plugins/algebra/vc/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/frontend/vc_vc/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/storage/vc/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/math/vc/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/detray-src/plugins/algebra/vc/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/algebraplugins-src/frontend/vc_cmath/include \
-I/afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/device/common/include \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/vecmem-build/cuda/CMakeFiles \
-I/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/vecmem-src/cuda/include \
-isystem=/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/eigen3-src \
-isystem=/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/thrust-src \
-isystem=/mnt/hdd1/software/boost/1.80.0/x86_64-centos9-gcc11-opt/include \
-isystem=/mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/vc-src \
-allow-unsupported-compiler --expt-relaxed-constexpr -O2 -g -DNDEBUG \
--generate-code=arch=compute_52,code=[compute_52,sm_52] -Xcompiler=-fPIC \
--expt-relaxed-constexpr -Xcudafe --diag_suppress=20012 -std=c++17 -MD \
-MT device/cuda/CMakeFiles/traccc_cuda.dir/src/cca/component_connection.cu.o \
-MF CMakeFiles/traccc_cuda.dir/src/cca/component_connection.cu.o.d \
-x cu -c /afs/cern.ch/user/k/krasznaa/work/projects/traccc/traccc/device/cuda/src/cca/component_connection.cu \
-o CMakeFiles/traccc_cuda.dir/src/cca/component_connection.cu.o
/software/cuda/11.7.1/x86_64/bin/../targets/x86_64-linux/include/thrust/detail/type_traits.h:141:107: warning: builtin __has_trivial_constructor is deprecated; use __is_trivially_constructible instead [-Wdeprecated-builtins]
template< class T> struct has_trivial_constructor : public integral_constant< bool, is_pod< T> ::value || __has_trivial_constructor(T)> {
^
/software/cuda/11.7.1/x86_64/bin/../targets/x86_64-linux/include/thrust/detail/type_traits.h:157:112: warning: builtin __has_trivial_copy is deprecated; use __is_trivially_copyable instead [-Wdeprecated-builtins]
template< class T> struct has_trivial_copy_constructor : public integral_constant< bool, is_pod< T> ::value || __has_trivial_copy(T)> {
^
/software/cuda/11.7.1/x86_64/bin/../targets/x86_64-linux/include/thrust/detail/type_traits/has_trivial_assign.h:33:131: warning: builtin __has_trivial_assign is deprecated; use __is_trivially_assignable instead [-Wdeprecated-builtins]
template< class T> struct has_trivial_assign : public integral_constant< bool, (is_pod< T> ::value && (!is_const< T> ::value)) || __has_trivial_assign(T)> {
^
3 warnings generated.
Most of the build options are not relevant for this discussion, but as you can see, the flags try to tell nvcc to use Thrust out of the “cmake build directory”. But it doesn’t. It still picks them up from the CUDA installation itself.
This worries me quite a bit. Since apparently our build only worked so far because the Thrust version we build, and the one coming with CUDA, were binary compatible with each other…
So… did I forget about something? Do you know why nvcc didn’t find the headers under:
[bash][pcadp04]:build-x86_64_centos9_llvm_nvidia > ls -l /mnt/hdd1/krasznaa/projects/traccc/build-x86_64_centos9_llvm_nvidia/_deps/thrust-src
total 212
-rw-r--r--. 1 krasznaa zp 87693 Nov 8 16:07 CHANGELOG.md
drwxr-xr-x. 7 krasznaa zp 4096 Nov 8 16:07 ci
drwxr-xr-x. 2 krasznaa zp 4096 Nov 8 16:07 cmake
-rw-r--r--. 1 krasznaa zp 4321 Nov 8 16:07 CMakeLists.txt
-rw-r--r--. 1 krasznaa zp 3801 Nov 8 16:07 CODE_OF_CONDUCT.md
-rw-r--r--. 1 krasznaa zp 22255 Nov 8 16:07 CONTRIBUTING.md
lrwxrwxrwx. 1 krasznaa zp 20 Nov 8 16:07 cub -> dependencies/cub/cub
drwxr-xr-x. 3 krasznaa zp 4096 Nov 8 16:07 dependencies
drwxr-xr-x. 2 krasznaa zp 4096 Nov 8 16:07 doc
drwxr-xr-x. 6 krasznaa zp 4096 Nov 8 16:07 examples
-rwxr-xr-x. 1 krasznaa zp 4500 Nov 8 16:07 generate_mk.py
drwxr-xr-x. 6 krasznaa zp 4096 Nov 8 16:07 internal
-rw-r--r--. 1 krasznaa zp 13968 Nov 8 16:07 LICENSE
-rw-r--r--. 1 krasznaa zp 4570 Nov 8 16:07 Makefile
-rw-r--r--. 1 krasznaa zp 14793 Nov 8 16:07 README.md
drwxr-xr-x. 9 krasznaa zp 12288 Nov 8 16:07 testing
drwxr-xr-x. 10 krasznaa zp 4096 Nov 8 16:07 thrust
[bash][pcadp04]:build-x86_64_centos9_llvm_nvidia >
Any insight would be much appreciated.
Attila