Packaging CUDA in spack, rpath vs link path with compat/stubs, driver vs runtime apis

Accelerated Computing CUDA CUDA Setup and Installation
1

Hello,

Reaching out on behalf of a spack PR trying to get gcc+nvptx to build, which is revealing our setup seems to have some discrepancies with how we treat the driver vs runtime api’s.

  1. Should libcuda.so ever show up in -L/path/ compiler invocations or be rpath’d into anything? (Our current answer is no, the setup_dependent_package just makes it available for the gcc package to see it in spec['cuda']).
  2. The runtime libs libcudart.so should always be available via -L{/path/} entries and rpath’d into binaries, and available in $LD_LIBRARY_PATH for consuming packages? (Currently our answer is yes, the libs property has that effect).
  3. Should we be masking out either compat or stubs directories when considering either or both of these APIs? E.g., for (2), currently in that PR we would be having both compat and stubs show up in e.g., $LD_LIBRARY_PATH. I think we want to mask it out for both driver and runtime side but do not know what these are for.
  4. Probably related to (3), do these answers change if we ever get to cross-compiling?

Thanks for any help, I hope this question makes sense! When comparing runfile vs system package manager installs, and where libcuda.so may show up, I got kind of turned around. It could also come from e.g., /usr/lib64 from an updated (non-CUDA) nvidia driver install right? (spack currently hides that directory). I’ve only toyed with the driver API for shiggles, but never really had to deal with the logistics of the build system :)

2

Maybe https://stackoverflow.com/a/52784819 helps at least a little bit, which also discusses the CUDA stubs libraries.