Gcc incompatibility with nvcc during CUDA Compiler tests

[obagoren1]

Hello Everyone,

I am attempting to build colmap on my desktop, which runs Ubuntu 20.04. I have a GeForce GTX 1650 with the driver version 555.42.02. This driver and GPU can support up to Cuda 12.5, but as there are not a lot of libraries that yet have support for 12.5, I have installed a variety of different versions of CUDA.

Currently, I am trying to build COLMAP with CUDA 11.3. Here are the outputs from commands that I think are relevant to this problem:

$ which nvcc
/usr/local/cuda-11.3/bin/nvcc

$ ls /usr/bin/nvcc 
/usr/bin/nvcc -> /usr/local/cuda-11.3/bin/nvcc

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

$ /usr/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0

So, I know that whenever I call /usr/bin/nvcc it;s pointing to CUDA 11.3. I;ve additionally ensured that the /usr/local/cuda-11.3/bin is added to my PATH variable and /usr/local/cuda-11.3/lib64 is in my LD_LIBRARY_PATH variables./

The issue I am facing is when I run CUDA compiler tests. The full output from the error is below:

-- Found CUDA version 11.3 installed in /usr/local/cuda-11.3 via legacy CMake (<3.17) module. Using the legacy CMake module means that any installation of COLMAP will require that the CUDA libraries are available under LD_LIBRARY_PATH.
-- Found CUDA 
--   Includes : /usr/local/cuda-11.3/include
--   Libraries : /usr/local/cuda-11.3/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so
-- The CUDA compiler identification is unknown
-- Check for working CUDA compiler: /usr/bin/nvcc
-- Check for working CUDA compiler: /usr/bin/nvcc -- broken
CMake Error at /usr/share/cmake-3.16/Modules/CMakeTestCUDACompiler.cmake:46 (message):
  The CUDA compiler

    "/usr/bin/nvcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/obagoren/installs/colmap/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/ninja cmTC_85a15 && [1/2] Building CUDA object CMakeFiles/cmTC_85a15.dir/main.cu.o
    FAILED: CMakeFiles/cmTC_85a15.dir/main.cu.o 
    /usr/bin/nvcc     -x cu -c main.cu -o CMakeFiles/cmTC_85a15.dir/main.cu.o && /usr/bin/nvcc     -x cu -M main.cu -MT CMakeFiles/cmTC_85a15.dir/main.cu.o -o CMakeFiles/cmTC_85a15.dir/main.cu.o.d
    In file included from /usr/include/cuda_runtime.h:83,
                     from <command-line>:
    /usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
          |  ^~~~~
    ninja: build stopped: subcommand failed.
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  cmake/FindDependencies.cmake:103 (enable_language)
  CMakeLists.txt:104 (include)

I believe that this issue arises when a CUDA compiler that is not compatible with gcc >=8, which is an issue if I had CUDA 10.2, but I don’t see how CUDA 10.2 could be involved with this issue, as I don’t have it installed.

Any help or insight on how to debug this would be greatly appreciated!

Thank you :)

[Robert_Crovella]

You have an unacceptable version of CUDA installed at /usr
This is almost certainly due to the fact that at some point you installed via an Ubuntu provided package (which tends to install CUDA there) rather than a NVIDIA provided installer.

Just because you have /usr/bin/nvcc symlinked to /usr/local/cuda-11.3/bin/nvcc does not mean the rest of the setup is taken care of.

This is partly due to the fact that CMake is getting in your way. The bizarro behavior by CMake is that it is indicating that it found CUDA at /usr/local/… but then it is deciding to use /usr/bin/nvcc, which could not possibly be part of the CUDA install at /usr/local/…

Your machine has a corrupted mixed-up install. Unwinding it all would require lots of poking around on your machine to find out what is actually installed and how each item was installed. I probably won’t be able to help with that.

I would start by removing every trace of the CUDA toolkit from your computer, then install using a NVIDIA provided install method, set the appropriate variables for post-install steps, and try again. If you want to try a shortcut, rather than symlink /usr/bin/nvcc, I would just delete it entirely. That might steer CMake in the right direction. Or it might not.

Anyway, you have some junk installed at /usr/include, and CMake is finding that and deciding to try and use it (because a nvcc invoked from /usr/bin/nvcc would do that) , and it is clearly old and not compatible. That isn’t a NVIDIA problem or a nvcc issue. You need to clean up the machine install, and configure CMake correctly. CMake is not a NVIDIA product.

[obagoren1]

Thank you! You’re right, I installed CUDA through the installation process of another SDK. Once I purged and followed the proper process, this was resolved. Thank you!