The detected CUDA version mismatches the version that was used to compile Pytorch

Accelerated Computing CUDA CUDA Setup and Installation
,
1

Device information

  • RTX 3060 ti, 8GB VRAM
  • Local-installed CUDA version : 12.1

Reproduction of the error

I was trying to install a github project called GaussianObject, by creating virtual environment with conda :

conda create -n go python=3.10
conda activate go
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
conda install nvidia/label/cuda-11.7.0::cuda
pip install -r requirements.txt

And checked successful installation of CUDA and torch on my conda envrionment :

(go) C:\hayoung\work\GaussianObject>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

(go) C:\hayoung\work\GaussianObject>python
Python 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.cuda.is_available())
True
>>> print(torch.version.cuda)
11.7
>>>

However, the error occurs when i tried installing submodules with the following command :

pip install -e submodules/diff-gaussian-rasterization

The full error message is as follows :

(go) C:\hayoung\work\GaussianObject>pip install -e submodules/diff-gaussian-rasterization
Obtaining file:///C:/hayoung/work/GaussianObject/submodules/diff-gaussian-rasterization
  Preparing metadata (setup.py) ... done
Installing collected packages: diff-gaussian-rasterization
  Running setup.py develop for diff-gaussian-rasterization
    error: subprocess-exited-with-error

    Γ— python setup.py develop did not run successfully.
    β”‚ exit code: 1
    ╰─> [76 lines of output]
        running develop
        C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
        !!

                ********************************************************************************
                Please avoid running ``setup.py`` and ``easy_install``.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.

                See https://github.com/pypa/setuptools/issues/917 for details.
                ********************************************************************************

        !!
          easy_install.initialize_options(self)
        C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
        !!

                ********************************************************************************
                Please avoid running ``setup.py`` directly.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.

                See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
                ********************************************************************************

        !!
          self.initialize_options()
        running egg_info
        writing diff_gaussian_rasterization.egg-info\PKG-INFO
        writing dependency_links to diff_gaussian_rasterization.egg-info\dependency_links.txt
        writing top-level names to diff_gaussian_rasterization.egg-info\top_level.txt
        C:\Users\jerem\anaconda3\envs\go\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
          warnings.warn(msg.format('we could not find ninja.'))
        reading manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
        adding license file 'LICENSE.md'
        writing manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
        running build_ext
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "C:\hayoung\work\GaussianObject\submodules\diff-gaussian-rasterization\setup.py", line 17, in <module>
            setup(
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\__init__.py", line 103, in setup
            return distutils.core.setup(**attrs)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
            return run_commands(dist)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
            dist.run_commands()
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
            self.run_command(cmd)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\dist.py", line 989, in run_command
            super().run_command(command)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
            cmd_obj.run()
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\develop.py", line 34, in run
            self.install_for_development()
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\develop.py", line 109, in install_for_development
            self.run_command('build_ext')
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
            self.distribution.run_command(command)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\dist.py", line 989, in run_command
            super().run_command(command)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
            cmd_obj.run()
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\build_ext.py", line 88, in run
            _build_ext.run(self)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
            self.build_extensions()
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
            _check_cuda_version(compiler_name, compiler_version)
          File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version
            raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
        RuntimeError:
        The detected CUDA version (12.1) mismatches the version that was used to compile
        PyTorch (11.7). Please make sure to use the same CUDA versions.

        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Γ— python setup.py develop did not run successfully.
β”‚ exit code: 1
╰─> [76 lines of output]
    running develop
    C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``easy_install``.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://github.com/pypa/setuptools/issues/917 for details.
            ********************************************************************************

    !!
      easy_install.initialize_options(self)
    C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` directly.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
            ********************************************************************************

    !!
      self.initialize_options()
    running egg_info
    writing diff_gaussian_rasterization.egg-info\PKG-INFO
    writing dependency_links to diff_gaussian_rasterization.egg-info\dependency_links.txt
    writing top-level names to diff_gaussian_rasterization.egg-info\top_level.txt
    C:\Users\jerem\anaconda3\envs\go\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
      warnings.warn(msg.format('we could not find ninja.'))
    reading manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
    adding license file 'LICENSE.md'
    writing manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
    running build_ext
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "C:\hayoung\work\GaussianObject\submodules\diff-gaussian-rasterization\setup.py", line 17, in <module>
        setup(
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\__init__.py", line 103, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
        return run_commands(dist)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
        dist.run_commands()
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
        self.run_command(cmd)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\dist.py", line 989, in run_command
        super().run_command(command)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
        cmd_obj.run()
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\develop.py", line 34, in run
        self.install_for_development()
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\develop.py", line 109, in install_for_development
        self.run_command('build_ext')
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
        self.distribution.run_command(command)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\dist.py", line 989, in run_command
        super().run_command(command)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
        cmd_obj.run()
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\command\build_ext.py", line 88, in run
        _build_ext.run(self)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 345, in run
        self.build_extensions()
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
        _check_cuda_version(compiler_name, compiler_version)
      File "C:\Users\jerem\anaconda3\envs\go\lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version
        raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
    RuntimeError:
    The detected CUDA version (12.1) mismatches the version that was used to compile
    PyTorch (11.7). Please make sure to use the same CUDA versions.

    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

I don’t understand why the detected CUDA version refers to my local CUDA(12.1), since I’ve already have conda-installed CUDA(11.7). Can anyone help me out please?

2

What happens if you install pytorch for cuda 12.1 and then try running this command?

Also, I was wondering

could it be that, it accesses the file
Obtaining file:///C:/hayoung/work/GaussianObject/submodules/diff-gaussian-rasterization
which is not in your virtual env, so it accesses the cuda 12.1 instead of cuda 11.7? I could be wrong but, it seems that way.

3

I’ve created another conda environment with CUDA(12.1) installed :

(go12.1) C:\hayoung\work\GaussianObject>python
Python 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
2.2.1
>>> print(torch.cuda.is_available())
True
>>> print(torch.version.cuda)
12.1
>>> exit()

And retried installing submodules, the error changes. It says there is no crtdefs.h :

(go12.1) C:\hayoung\work\GaussianObject>pip install -e submodules/diff-gaussian-rasterization
Obtaining file:///C:/hayoung/work/GaussianObject/submodules/diff-gaussian-rasterization
  Preparing metadata (setup.py) ... done
Installing collected packages: diff-gaussian-rasterization
  Running setup.py develop for diff-gaussian-rasterization
    error: subprocess-exited-with-error

    Γ— python setup.py develop did not run successfully.
    β”‚ exit code: 1
    ╰─> [46 lines of output]
        running develop
        C:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\setuptools\command\develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
        !!

                ********************************************************************************
                Please avoid running ``setup.py`` and ``easy_install``.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.

                See https://github.com/pypa/setuptools/issues/917 for details.
                ********************************************************************************

        !!
          easy_install.initialize_options(self)
        C:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
        !!

                ********************************************************************************
                Please avoid running ``setup.py`` directly.
                Instead, use pypa/build, pypa/installer or other
                standards-based tools.

                See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
                ********************************************************************************

        !!
          self.initialize_options()
        running egg_info
        writing diff_gaussian_rasterization.egg-info\PKG-INFO
        writing dependency_links to diff_gaussian_rasterization.egg-info\dependency_links.txt
        writing top-level names to diff_gaussian_rasterization.egg-info\top_level.txt
        C:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\utils\cpp_extension.py:500: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
          warnings.warn(msg.format('we could not find ninja.'))
        reading manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
        adding license file 'LICENSE.md'
        writing manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
        running build_ext
        building 'diff_gaussian_rasterization._C' extension
        creating build
        creating build\temp.win-amd64-cpython-310
        creating build\temp.win-amd64-cpython-310\Release
        creating build\temp.win-amd64-cpython-310\Release\cuda_rasterizer
        "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc" -c cuda_rasterizer/backward.cu -o build\temp.win-amd64-cpython-310\Release\cuda_rasterizer/backward.obj -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include\TH -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -IC:\Users\jerem\anaconda3\envs\go12.1\include -IC:\Users\jerem\anaconda3\envs\go12.1\Include -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -IC:\hayoung\work\GaussianObject\submodules\diff-gaussian-rasterization\third_party/glm/ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 --use-local-env
        C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include\crt/host_config.h(231): fatal error C1083: 포함 νŒŒμΌμ„ μ—΄ 수 μ—†μŠ΅λ‹ˆλ‹€. 'crtdefs.h': No such file or directory
        backward.cu
        error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1\\bin\\nvcc.exe' failed with exit code 2
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Γ— python setup.py develop did not run successfully.
β”‚ exit code: 1
╰─> [46 lines of output]
    running develop
    C:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\setuptools\command\develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``easy_install``.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://github.com/pypa/setuptools/issues/917 for details.
            ********************************************************************************

    !!
      easy_install.initialize_options(self)
    C:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\setuptools\_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` directly.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
            ********************************************************************************

    !!
      self.initialize_options()
    running egg_info
    writing diff_gaussian_rasterization.egg-info\PKG-INFO
    writing dependency_links to diff_gaussian_rasterization.egg-info\dependency_links.txt
    writing top-level names to diff_gaussian_rasterization.egg-info\top_level.txt
    C:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\utils\cpp_extension.py:500: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
      warnings.warn(msg.format('we could not find ninja.'))
    reading manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
    adding license file 'LICENSE.md'
    writing manifest file 'diff_gaussian_rasterization.egg-info\SOURCES.txt'
    running build_ext
    building 'diff_gaussian_rasterization._C' extension
    creating build
    creating build\temp.win-amd64-cpython-310
    creating build\temp.win-amd64-cpython-310\Release
    creating build\temp.win-amd64-cpython-310\Release\cuda_rasterizer
    "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin\nvcc" -c cuda_rasterizer/backward.cu -o build\temp.win-amd64-cpython-310\Release\cuda_rasterizer/backward.obj -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include\TH -IC:\Users\jerem\anaconda3\envs\go12.1\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include" -IC:\Users\jerem\anaconda3\envs\go12.1\include -IC:\Users\jerem\anaconda3\envs\go12.1\Include -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -IC:\hayoung\work\GaussianObject\submodules\diff-gaussian-rasterization\third_party/glm/ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 --use-local-env
    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\include\crt/host_config.h(231): fatal error C1083: 포함 νŒŒμΌμ„ μ—΄ 수 μ—†μŠ΅λ‹ˆλ‹€. 'crtdefs.h': No such file or directory
    backward.cu
    error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.1\\bin\\nvcc.exe' failed with exit code 2
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
4

I have little to no experience when it comes to windows but can you check if that file actually exists in your system?

i think in windows. it would be dir /s filename or something i guess.

Also might I suggest that using docker rather than doing it on your windows host?
It would be easier for you to manage your dependencies.

Lastly, I would recommend you to also post your questions on different forums like stackoverflow since this is more of a git repo installation problem.

1 Like