Failed to compile SmileiPIC/Smilei/issues/744 please review the attached error log makelog2.log (157.6 KB)
Using the following configuration:
conda activate smilei_env
module load nvhpc-hpcx-cuda12/ phdf5/nvhpc/
export GPU_COMPILER=nvcc
export SMILEICXX=nvc++
export BUILD_DIR=/opt/apps/libsoft/smilei/nvhpc/5.1
export PYTHONEXE=/opt/apps/miniconda3/env/root/envs/smilei_gpu_env/bin/python
export HDF5_ROOT_DIR=/opt/apps/fileformat/phdf5/nvhpc/1.14.3
export CXXFLAGS="-O3 -march=cascadelake -mtune=cascadelake -noswitcherror -w -v -tp=cascadelake -gpu=cc50 -acc=gpu -std=c++14 -lcurand -cudalib=curand -Minfo=accel -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1"
export GPU_COMPILER_FLAGS="-O3 --std c++14 -arch=sm_50 -v --expt-relaxed-constexpr"
export LDFLAGS="-acc=gpu -gpu=cc50 -v -std=c++14 -cudalib=curand -lcudart -lcurand -lacccuda"
export SMILEICXX_DEPS=g++
make -j 12 config="verbose noopenmp gpu_nvidia" 2>&1 | tee makelog2.log
- Computing resources
- type of computer : Cluster NUMA node + 1 GPU Quadro M5000
- OS : Linux Ubuntu 22.04.4 LTS
- NVIDIA-SMI 555.42.06 Driver Version: 555.42.06 CUDA Version: 12.5
- C++ compiler g++ (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
- MPI version:
nvc++ 24.7-0 64-bit target on x86-64 Linux -tp cascadelake
NVIDIA Compilers and Tools
Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
nvc++ -I/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/include -I/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/include/openmpi -I/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/include/openmpi/opal/mca/hwloc/hwloc201/hwloc/include -I/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/include/openmpi/opal/mca/event/libevent2022/libevent -I/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/include/openmpi/opal/mca/event/libevent2022/libevent/include -pthread -L/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/lib -Wl,-rpath -Wl,/opt/apps/nvidia-hpc-sdk/24.7/Linux_x86_64/24.7/comm_libs/12.5/hpcx/hpcx-2.19/ompi/lib -Wl,--enable-new-dtags -lmpi
- HDF5 version (
h5cc --version):
gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- Python version (
python --version) : Python 3.12.4
- CXXFLAGS=
-noswitcherror
_Originally posted by @W-Wuxian in Failed to compile for GPU · Issue #742 · SmileiPIC/Smilei · GitHub
Hi lederervictor,
I grab the code and was able to recreate the error with 24.7, though it works with 24.5 and the upcoming 24.9.
Looks to be a problem with the CUDA 12.5 Thrust header files bring in unprotected CUDA code. The thrust header is being included in “src/Particles/nvidiaParticles.h”.
The two possible work arounds are to either add “-cuda” to CXXFLAGS so nvc++ knows to parse the CUDA code, or use “-gpu=cuda11.8” to revert to an older version of CUDA.
Again, I don’t see the issue in our upcoming 24.9 release since it’s using CUDA 12.6.
-Mat
Hi MatColgrove,
the “-cuda” option actually fix previous errors msg, but i got new one related to constexpr see the attached MAKELOG.log file.
MAKELOG.log (130.2 KB)
to reproduce the error, here is the step by step:
conda activate smilei_env
module load nvhpc-hpcx-cuda12/ phdf5/nvhpc/
source NVIDIA_SMILEI_ENV
make -j12 machine="machine_file" config="verbose gpu_nvidia" 2>&1 | tee MAKELOGlog
I created a NVIDIA_SMILEI_ENV.sh:
export BUILD_DIR=/opt/apps/libsoft/smilei/nvhpc/5.1
export GPU_COMPILER=nvcc
export SMILEICXX=nvc++
export PYTHONEXE=/opt/apps/miniconda3/env/root/envs/smilei_gpu_env/bin/python
export HDF5_ROOT_DIR=/opt/apps/fileformat/phdf5/nvhpc/1.14.3
and used this mahine_file:
SMILEICXX_DEPS = g++
# Options pour nvcxx (Compilateur C++ de NVHP)
CXXFLAGS += -w -v -cuda -noswitcherror
CXXFLAGS += -tp=cascadelake -gpu=cc50 -acc=gpu -std=c++14 -lcurand -cudalib=curand
CXXFLAGS += -Minfo=accel # what is offloaded/copied
CXXFLAGS += -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1
# Options pour NVCC (Compilateur CUDA)
GPU_COMPILER_FLAGS += -O3 --std c++14 -arch=sm_50 -v # --compiler-bindir="mpicc"
GPU_COMPILER_FLAGS += --expt-relaxed-constexpr
LDFLAGS += -acc=gpu -gpu=cc50 -v -std=c++14 -cudalib=curand -lcudart -lcurand -lacccuda
This error occurs if you compile the files with C++ 11. Although it’s a bit difficult to read your log given you have multiple makejobs (if you post again, please use a single job), I do see “-std=c++11” being added. Although “-std=c++14” is there as well, the C++11 flag is second, so take precedence.
I’m not sure where this is getting set, but possibly it’s in your environment’s CXXFLAGS setting.
For example:
nvc++ -w -v -cuda -noswitcherror -tp=cascadelake -gpu=cc50 -acc=gpu -std=c++14 -lcurand -cudalib=curand -Minfo=accel -D__GCC_ATOMIC_TEST_AND_SET_TRUEVAL=1 -D__VERSION="??-??" -DOMPI_SKIP_MPICXX -std=c++11 -I/opt/apps/fileformat/phdf5/nvhpc/1.14.3/include -Isrc -Isrc/MovWindow -Isrc/ElectroMagnSolver -Isrc/Interpolator -Isrc/ParticleBC -Isrc/DomainDecomposition -Isrc/Particles -Isrc/ElectroMagn -Isrc/Ionization -Isrc/Merging -Isrc/ElectroMagnBC -Isrc/ParticleInjector -Isrc/SmileiMPI -Isrc/Diagnostic -Isrc/Projector -Isrc/PartCompTime -Isrc/Pusher -Isrc/Tools -Isrc/Species -Isrc/Field -Isrc/Checkpoint -Isrc/Python -Isrc/Radiation -Isrc/Params -Isrc/Patch -Isrc/MultiphotonBreitWheeler -Isrc/picsar_interface -Isrc/Profiles -Isrc/Collisions -I/opt/apps/libsoft/smilei/nvhpc/5.1/src/Python -I/opt/apps/miniconda3/env/root/envs/smilei_gpu_env/include/python3.12 -I/opt/apps/miniconda3/env/root/envs/smilei_gpu_env/include/python3.12 -I/opt/apps/miniconda3/env/root/envs/smilei_gpu_env/lib/python3.12/site-packages/numpy/core/include -DSMILEI_USE_NUMPY -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -O3 -Kfast -g -DSMILEI_ACCELERATOR_GPU -DSMILEI_ACCELERATOR_GPU_OACC -c src/Collisions/BinaryProcesses.cpp -o /opt/apps/libsoft/smilei/nvhpc/5.1/src/Collisions/BinaryProcesses.o
Compiling src/Collisions/Collisions.cpp
Note that I’m seeing an issue compiling the file “MultiphotonBreitWheeler.cpp”, which I’ll look at in a bit.
I try my best, but I was unable to remove the -std=c++11, I am waiting for a response from the Smilei team on this subject. Thx for your help!
Sounds good, though I don’t see it in my build so suspect it’s something being added in your environment.
So it was export SMILEICXX=nvc++ who rewrote -std=c++14 with -std=c++11. Fix from the dev team was to rewrite every occurrence of -std=c++11 with -std=c++14 in the makefile file. Afterwards, the compilation did not failed but did not generate software executable.