I found that pkg-config --libs cuda-11.0 gives me wrong path
$ pkg-config --libs cuda-11.0
-L/usr/local/cuda-11.0/x86_64-linux/lib -lcuda
But when I look at that location, it doesn’t exist.
$ ls /usr/local/cuda-11.0/x86_64-linux/lib
ls: cannot access '/usr/local/cuda-11.0/x86_64-linux/lib': No such file or directory
Problem is that this path is missing /targets/ between /usr/local/cuda-11.0 and x86_64-linux/lib.
$ ls /usr/local/cuda-11.0/targets/x86_64-linux/lib
libOpenCL.so libcublas.so.11 libcudart.so.11.0.167 libcuinj64.so libcurand.so.10 libcusparse.so libnppial.so libnppidei.so.11.0.0.191 libnppim.so.11 libnppitc.so libnvToolsExt.so.1.0.0 libnvperf_target.so
libOpenCL.so.1 libcublas.so.11.1.0.213 libcudart_static.a libcuinj64.so.11.0 libcurand.so.10.2.0.191 libcusparse.so.11 libnppial.so.11 libnppif.so libnppim.so.11.0.0.191 libnppitc.so.11 libnvblas.so libnvrtc-builtins.so
libOpenCL.so.1.0 libcublasLt.so libcufft.so libcuinj64.so.11.0.167 libcusolver.so libcusparse.so.11.0.0.191 libnppial.so.11.0.0.191 libnppif.so.11 libnppist.so libnppitc.so.11.0.0.191 libnvblas.so.11 libnvrtc-builtins.so.11.0
libOpenCL.so.1.0.0 libcublasLt.so.11 libcufft.so.10 libculibos.a libcusolver.so.10 liblapack_static.a libnppicc.so libnppif.so.11.0.0.191 libnppist.so.11 libnpps.so libnvblas.so.11.1.0.213 libnvrtc-builtins.so.11.0.167
libaccinj64.so libcublasLt.so.11.1.0.213 libcufft.so.10.1.3.191 libcupti.so libcusolver.so.10.4.0.191 libmetis_static.a libnppicc.so.11 libnppig.so libnppist.so.11.0.0.191 libnpps.so.11 libnvjpeg.so libnvrtc.so
libaccinj64.so.11.0 libcudadevrt.a libcufftw.so libcupti.so.11.0 libcusolverMg.so libnppc.so libnppicc.so.11.0.0.191 libnppig.so.11 libnppisu.so libnpps.so.11.0.0.191 libnvjpeg.so.11 libnvrtc.so.11.0
libaccinj64.so.11.0.167 libcudart.so libcufftw.so.10 libcupti.so.2020.1.0 libcusolverMg.so.10 libnppc.so.11 libnppidei.so libnppig.so.11.0.0.191 libnppisu.so.11 libnvToolsExt.so libnvjpeg.so.11.0.0.191 libnvrtc.so.11.0.167
libcublas.so libcudart.so.11.0 libcufftw.so.10.1.3.191 libcurand.so libcusolverMg.so.10.4.0.191 libnppc.so.11.0.0.191 libnppidei.so.11 libnppim.so libnppisu.so.11.0.0.191 libnvToolsExt.so.1 libnvperf_host.so stubs
This is inside NGC docker image nvcr.io/nvidia/tensorflow:20.06-tf2-py3 but also on the host machine if cuda-11.0 is installed with package manager. So, these files
/usr/lib/pkgconfig$ ls
accinj64-11.0.pc cuda-11.0.pc cufft-11.0.pc cuinj64-11.0.pc cusolver-11.0.pc nppc-11.0.pc nppial-11.0.pc nppicom-11.0.pc nppif-11.0.pc nppim-11.0.pc nppisu-11.0.pc npps-11.0.pc nvidia-ml-11.0.pc nvrtc-11.0.pc
cublas-11.0.pc cudart-11.0.pc cufftw-11.0.pc curand-11.0.pc cusparse-11.0.pc nppi-11.0.pc nppicc-11.0.pc nppidei-11.0.pc nppig-11.0.pc nppist-11.0.pc nppitc-11.0.pc nvToolsExt-11.0.pc nvjpeg-11.0.pc
contains wrong PATH information.
/usr/lib/pkgconfig$ cat cuda-11.0.pc
cudaroot=/usr/local/cuda-11.0
libdir=${cudaroot}/x86_64-linux/lib
includedir=${cudaroot}/x86_64-linux/include
Name: cuda
Description: CUDA Driver Library
Version: 11.0
Libs: -L${libdir} -lcuda
Cflags: -I${includedir}
So, simple workaround is to create a symlink inside /usr/local/cuda-11.0/
sudo ln -s targets/x86_64-linux/ x86_64-linux