In the /usr/local/bin/nvidia_entrypoint.sh script one of the first things done is a “find -L /usr -name libcuda.so.1”. If the user happens to bind mount a large local volume of their own under /usr this will result in huge startup times just to bootstrap the container.
In my case we have a /usr/pubsw which is a mount on our systems to a huge 1TB+ trove of 3rd party software. To clone our environment in the container we do a bind mount to the same location in docker/singularity when running the NGC container but then they spend over 15 minutes doing this find -L
Instead I think the entrypoint script should only search the directories that are in LD_LIBRARY_PATH or just test for /dev/nvidiactl