Nvidia-cuda-nvcc pip wheel installation

Hi,

I’m working on distributing a PyTorch package which depends on a number of custom CUDA extensions. To simplify the installation process, I’m trying to use the CUDA pip wheels as described here. The nvidia-cuda-runtime, nvidia-cudnn, and nvidia-tensorrt packages all seem to work as I expect and are recognized by PyTorch after installation. However, the dependencies which compile custom CUDA extensions are unable to find nvcc (error: [Errno 2] No such file or directory: '/usr/local/cuda/bin/nvcc' ).

The installation guide does indeed mention that when installing the pip wheels "additional care must be taken to set up your host environment to use CUDA outside the pip environment". However, as far as I can tell there is no documentation regarding what needs to be done to actually use the pip installed nvcc. I’m also not sure what counts as being “outside the pip environment,” in this case my python installation is able to find the rest of the installed packages correctly.

I’ve gone through my environment to see if I could find nvcc hidden away somewhere, but it seems like the actual nvcc binary is not present. The RECORD file of nvidia-cuda-nvcc is as follows, no nvcc to be found:

nvidia/cuda_nvcc/bin/ptxas,sha256=7Ro05XA6Y7yWRe-t_t6tq66MyWmEnxgm8pqFS37q85E,10299856
nvidia/cuda_nvcc/include/crt/common_functions.h,sha256=-U44f4yUGmwDPwd7Q_3Cz5if05xHGPSlAzz5zMylLSQ,13559
nvidia/cuda_nvcc/include/crt/device_double_functions.h,sha256=pznu65Pp1kT08mNkXB6J1epefIogxqMapQ0fnabNjr4,39789
nvidia/cuda_nvcc/include/crt/device_double_functions.hpp,sha256=YYIbqYhb5Qmf8c4YfcC_jytg4FRwcXPjv3TFTwhb24E,8568
nvidia/cuda_nvcc/include/crt/device_functions.h,sha256=0XCVr0LVOaJrBORxq123nCoZOSox7kOSqwNvuzKFqO8,122288
nvidia/cuda_nvcc/include/crt/device_functions.hpp,sha256=7HJY-yyG-rAfGjfbbDdbmm29weWfIbSw5LbCN7VeNrg,8360
nvidia/cuda_nvcc/include/crt/func_macro.h,sha256=EOpDlaM917bh9cwBiFBPF689DCMBw5hFarxLxFt-i74,1755
nvidia/cuda_nvcc/include/crt/host_config.h,sha256=ESTtFPjPZ_d3hXHGcwIKABz9rnYvBeQUulez-mgSRCE,10606
nvidia/cuda_nvcc/include/crt/host_defines.h,sha256=B6p4K3j0FKSzqjwULNsDJo-6h7GTRrLHeqqmwoeFIZM,8617
nvidia/cuda_nvcc/include/crt/host_runtime.h,sha256=QverDqBf0MLLZxl3kwvkNPyUojIgCB-WbjgDvvRnEaU,9346
nvidia/cuda_nvcc/include/crt/math_functions.h,sha256=tGGCtRGa-EVIHU902Z3reBCNQf_ePlrJ731KyZY0PQA,355800
nvidia/cuda_nvcc/include/crt/math_functions.hpp,sha256=PUVbdgFn8cu79xncLEi87Bmh20lN-waMnIKHYD0jGoE,100054
nvidia/cuda_nvcc/include/crt/mma.h,sha256=OmIQLmGzZl59b3ZfaRyDk_HuKvFGZAOChFFI8iVviQY,62502
nvidia/cuda_nvcc/include/crt/mma.hpp,sha256=spo0LX71tUCipxK517Bssj0nc-ZHf8oMWzvHoYYB_6I,66599
nvidia/cuda_nvcc/include/crt/sm_70_rt.h,sha256=b_ub73AMT7ri5Ph2ahZzU6hyrFfIymnEx3NM7lv01UM,6506
nvidia/cuda_nvcc/include/crt/sm_70_rt.hpp,sha256=3a_rU-Y0MSB4htBDFY4PCQ_jXiWFTe7WT1ZyhMuCJOA,7837
nvidia/cuda_nvcc/include/crt/sm_80_rt.h,sha256=1LmDr7ftuCDpdqAnyRjgAiD_y85vjF2xUUnN9EelFW8,7449
nvidia/cuda_nvcc/include/crt/sm_80_rt.hpp,sha256=o-rJu-jpehCeyABGgv-8dYRB7oJTCwuNdvSCq0VURdE,6705
nvidia/cuda_nvcc/include/crt/storage_class.h,sha256=dzcOZ16pLaN8ejqHaXw4iHbBJ6fXWxfaU-sj2QjYzzg,4791
nvidia/cuda_nvcc/include/fatbinary_section.h,sha256=NnuUfy358yGJx4enq0pBnetjv17UWa-nOlgYToUitrw,1809
nvidia/cuda_nvcc/include/nvPTXCompiler.h,sha256=2-Y1rcK1oRdk3w7b5ZLIIVUuQS4d6z49bDaQpe8ze5M,11859
nvidia/cuda_nvcc/nvvm/include/nvvm.h,sha256=SAIqraG9mMmwniceQJA0AMp5aooz8viPnDy1tF9Ip84,10993
nvidia/cuda_nvcc/nvvm/lib64/libnvvm.so,sha256=aOUJ5v2XgTrfIse5Dtlr5OeiHzIkNwbm8di61lO7zNo,26934808
nvidia/cuda_nvcc/nvvm/libdevice/libdevice.10.bc,sha256=uOk-C8sMI-HkeDR4KVV1urzD-edwPEWuw0vKu_nBRuQ,469572
nvidia_cuda_nvcc_cu116-11.6.124.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
nvidia_cuda_nvcc_cu116-11.6.124.dist-info/License.txt,sha256=rW9YU_ugyg0VnQ9Y1JrkmDDC-Mk_epJki5zpCttMbM0,59262
nvidia_cuda_nvcc_cu116-11.6.124.dist-info/METADATA,sha256=cgrCkFyz0fDTkCCiRWLZVKDdh-f-fqkLteTfapX0vrQ,1432
nvidia_cuda_nvcc_cu116-11.6.124.dist-info/RECORD,
nvidia_cuda_nvcc_cu116-11.6.124.dist-info/WHEEL,sha256=-kQi_VMfvRQozZJT7HUPMfY-5vLo0LVTmAylNJ3Ft98,106
nvidia_cuda_nvcc_cu116-11.6.124.dist-info/top_level.txt,sha256=GZffmzIAaXvkXFQ4Q5KPsz8Rda58-E0RvTKbX9zGXAs,17

Am I just missing another place where nvidia-cuda-nvcc has actually installed the binary?

Previously, I’ve has success installing nvcc via the conda-forge package, cudatoolkit-dev. However, that requires users to install conda first and requires a good 10 minutes of build time on my local machine which is less than ideal. This is also why I’d rather not use the new nvidia/cuda-nvcc conda package.

Is what I’m trying to do even possible? Or does the nvidia-cuda-nvcc package not actually provide nvcc? If nvidia-cuda-nvcc does provide the actual binary, how can I ensure it is visible in my environment? Can anyone point me to information on getting this setup working?

Cheers,
Hans

HI

Did you find any solution for this?
I am having similar situation where I need ‘nvcc’ to be available with PIP installation so that I can install pycuda…

Hey there, sorry no, haven’t found a solution.

It seems to me though that this pip wheel is simply broken. Hopefully they will be fixed next time they are updated.

FWIW it’s not an issue in Google Colab as the env there already has the development requirements pre-installed.

For now I’m just advising my users to install the CUDA toolkit with dev requirements manually :/

Thanks for the update…

For those using miniconda just copy de file libdevice.10.bc into the root folder of python application or notebook.

It´s works here using python=3.9, cudatoolkit=11.2, cudnn=8.1.0, and tensorflow==2.9

The pip wheel for nvcc is still broken, even for CUDA 12, right?