NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver

I’ve Linux hpv-vm 5.15.0-1053-azure #61~20.04.1-Ubuntu SMP Tue Nov 21 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
GPU instance with

lspci
0002:00:00.0 3D controller: NVIDIA Corporation Device 2236 (rev a1)
ee9c:00:02.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] (rev 80)

I installed CUDA 11.2 and nvidia driver are:
libnvidia-common-470
libnvidia-gl-470
nvidia-driver-470

nvidia-bug-report.log.gz (44.9 KB)

nvidia-smi giving error:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
sudo systemctl status nvidia-persistenced.service
● nvidia-persistenced.service
   Loaded: not-found (Reason: Unit nvidia-persistenced.service not found.)
   Active: failed (Result: exit-code) since Fri 2024-01-12 08:50:20 UTC; 11min ago

Jan 12 08:50:19 hpv-vm nvidia-persistenced[2364]: Started (2364)
Jan 12 08:50:20 hpv-vm nvidia-persistenced[2364]: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that >
Jan 12 08:50:20 hpv-vm nvidia-persistenced[2363]: nvidia-persistenced failed to initialize. Check syslog for more details.
Jan 12 08:50:20 hpv-vm nvidia-persistenced[2364]: PID file unlocked.
Jan 12 08:50:20 hpv-vm systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
Jan 12 08:50:20 hpv-vm nvidia-persistenced[2364]: PID file closed.
Jan 12 08:50:20 hpv-vm nvidia-persistenced[2364]: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
Jan 12 08:50:20 hpv-vm nvidia-persistenced[2364]: Shutdown (2364)
Jan 12 08:50:20 hpv-vm systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
Jan 12 08:50:20 hpv-vm systemd[1]: Failed to start NVIDIA Persistence Daemon.
sudo lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0  63.5M  1 loop /snap/core20/2015
loop1     7:1    0  91.9M  1 loop /snap/lxd/24061
loop2     7:2    0  40.9M  1 loop /snap/snapd/20290
sda       8:0    0   256G  0 disk
├─sda1    8:1    0 255.9G  0 part /
├─sda14   8:14   0     4M  0 part
└─sda15   8:15   0   106M  0 part /boot/efi
sdb       8:16   0   360G  0 disk
└─sdb1    8:17   0   360G  0 part /mnt
sr0      11:0    1  1024M  0 rom

sudo fdisk -l
Device      Start       End   Sectors   Size Type
/dev/sda1  227328 536870878 536643551 255.9G Linux filesystem
/dev/sda14   2048     10239      8192     4M BIOS boot
/dev/sda15  10240    227327    217088   106M EFI System

Can someone help me in installing the right drivers for the above mentioned graphic card.