With cuda 11.6 on rhel8 it is not possible anymore to have an nfs installation of cuda.
cuda 11.6 has a the GDS dependency, which is a driver part, but it installs for example:
libcufile-11-6
Description :
GPU Direct Storage runtime libraries
/etc/ld.so.conf.d/gds-11-6.conf
/usr/local/cuda-11.6/gds
/usr/local/cuda-11.6/gds/cufile.json
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcufile.so.1.2.0
/usr/local/cuda-11.6/targets/x86_64-linux/lib/libcufile_rdma.so.1.2.0
With 11.2 on rhel7 it was different. Here it was possible to install
nvidia-driver-latest-dkms
and put the cuda installatoin on a read-only mounted NFS share.
This is not possible anymore, because the nvidia-gds package writes
into this read only location.