You need to recreate your initrd to make the changes to /etc/modprobe.d/nvidia.conf effective.
Note that there is no proper rights management for the nvidia device files. They are expected to be rw for everyone, otherwise things don’t work. That’s why nvidia-modprobe exists. It’s suid root and will (among other things) fixup everything to be 0666. nvidia-smi calls it if necessary.
Well I have little cluster and with old version of nvidia driver I can hide devices from users and I manupulate device permission in SGE prolog epilog script.
After upgrade, unfortunately (or not) devices permission now revert to the values given in the driver options.
I have to find another solution to share GPUs between users using old SGE scheduler.