First, let’s analyze what your current NVCC command line actually does. Please read all comments I added:
"E:\workspace\cuda\cuda\bin\nvcc.exe" // If you install CUDA toolkits into versioned folders, the path would indicate which CUDA version you're actually using.
--use-local-env // OK when translating from within MSVS IDE. Does not reconfigure the MSVC environment every time.
-ccbin "D:\visual studio\ide\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64" // OK, explicitly selects the host compiler 64 bit version.
-x cu // Not required. Let's all input files be handled as *.cu files.
-IE:\workspace\cuda\cuda\include // YES, required.
--keep-dir cmTC_62967\x64\Debug // Not required when not using --keep (see below).
-use_fast_math // YES, use the fast math library. Generates faster device code (for trigonometric functions, sqrt, reciprocals, etc)
-maxrregcount=0 // Not required. If this option is not specified, then no maximum is assumed. Value less than the minimum registers required by ABI will be bumped up by the compiler to ABI minimum limit.
--machine 64 // YES, must be 64 bit
--compile // BUG. Remove! Only --ptx or --optix-ir must be set. This means "Compile each .c/.cc/.cpp/.cxx/.cu input file into an object file." but you do not want to compile *.cu code to object files here, instead this should only translate it to PTX or OptiX-IR intermediate sources.
-cudart static // Not required, you shouldn't do CUDA runtime calls in OptiX device code.-
--machine 64 // Yes, duplicate
--keep // Not required. This will keep all intermediate files (*.i;*.ii;*.iii, etc.) from the NVCC compilation phase and it will place them in the above --keep-dir. That is only needed for debugging CUDA compiler issues.
--optix-ir // Yes, generate OptiX-IR output.
-std=c++17 // OK,
--generate-code=arch=compute_52,code=[compute_52,sm_52] // Generate code for Maxwell second generation GPU. OK when wanting to support all OptiX supported GPUs.
-Xcompiler="-Ob0 -Zi" // Host compiler settings.
-g // Not required, this means NVCC generates host debug code, but you're not generating any host code when translating *.cu files to OptiX module input PTX or OptiX IR. (Note that capital letter -G would mean debug device code and that is abysmally slow! Never set that unless you want to debug device code.)
-D"CMAKE_INTDIR=\"Debug\"" // Set CMake intermediate directory. Shouldn't be required
-D_MBCS // Not required, support for multi-byte chars on the host.
-D"CMAKE_INTDIR=\"Debug\"" // Duplicate from above.
-Xcompiler "/EHsc /W1 /nologo /Od /FS /Zi /RTC1 /MDd " // Host compiler options.
-Xcompiler "/FdcmTC_62967.dir\Debug\vc143.pdb" // Not required. Host compiler program database. NVCC compiles nly device code here.
-o cmTC_62967.dir\Debug\main.obj // BUG: This is for the --compile setting when generating CUDA object code. The should be -o cmTC_62967.dir\Debug\main.optixir when using --optix-ir and -o cmTC_62967.dir\Debug\main.ptx for when using --ptx.
"E:\<very_long_path_path>\main.cu" // YES, required input file.
// I would recommend to make path names under Windows rather short because there is a default limit of 260 characters (MAX_PATH) which can be increased with some registry setting (LongPathsEnabled).
Now, there are two ways to generate a specific NVCC command line with CMake.
One method is to build a custom build rule for each OptiX device code *.cu file. That can be done with a CMake macro which gets a list or *.cu files and a list of dependencies. I’m using this inside my OptiX examples.
This is the macro which can generate build rules for *.ptx and *.optixir output:
https://github.com/NVIDIA/OptiX_Apps/blob/master/3rdparty/CMake/nvcuda_compile_module.cmake
And this is how it’s called:
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/rtigo12/CMakeLists.txt#L202
It requires the find_package(CUDAToolkit 10.0 REQUIRED) and at least one of the find_package(OptiX* ) in the root CMakeLists.txt to be able to set the OPTIX_INCLUDE_DIR variable.
Note that this macro contains some commented-out message instructions which will print out the exact NVCC command line used for each of the input *.cu files during the CMake configuration which is helpful when testing different settings.
The other method is for applications which are using native CUDA kernels with the CUDA Runtime API and also translate OptiX device code to *.ptx or *.optixir by using the CMake LANGUAGE CUDA feature.
I that case the native CUDA files need to be compiled to object code (host and device parts) and the OptiX *.cu files which are only device code should not be in that list and must be handled differently.
CMake solves that by the Object Library feature.
I’ve posted a CMakeLists.txt which demonstrates that partitioning of the *.cu files in a project into native and OptiX module input files here:
https://forums.developer.nvidia.com/t/why-am-i-getting-optix-dir-notfound/279085/4
Read this as well:
https://forums.developer.nvidia.com/t/question-about-add-a-cu-in-my-project/284761/4