/dev/nvidia-caps* missing after boot on Ubuntu (AWS G5 instance)

Graphics / Linux Linux
1

Hi,

I installed NVIDIA drivers on my Linux instance (AWS G5 instance using NVIDIA TESLA A10), but /dev/nvidia-caps* devices fails to appear after boot.

I can workout the issue after boot with nvidia-smi or nvidia-modprobe (as documented on NVIDIA Multi-Instance GPU User Guide :: NVIDIA Data Center GPU Driver Documentation), but on other setup these devices appear on boot without manual action. For instance it works seamlessly with:

  • AWS G5 with NixOS 23.11 (same instance, different OS)
  • Paperspace P4000 with Ubuntu 22.04 (same OS, different Cloud)

Currently installed drivers:

$ sudo apt list --installed | grep nvidia

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnvidia-cfg1-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed,automatic]
libnvidia-compute-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed,automatic]
nvidia-compute-utils-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed,automatic]
nvidia-dkms-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed,automatic]
nvidia-firmware-535-server-535.183.01/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed,automatic]
nvidia-headless-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed]
nvidia-headless-no-dkms-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed]
nvidia-kernel-common-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed,automatic]
nvidia-kernel-source-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed]
nvidia-modprobe/jammy,now 470.103.01-1 amd64 [installed]
nvidia-utils-535-server/jammy-updates,jammy-security,now 535.183.01-0ubuntu0.22.04.1 amd64 [installed]

nvidia-persistenced is enabled:

ps -ef | grep nvidia-persistenced
root         441       1  0 10:46 ?        00:00:00 /usr/bin/nvidia-persistenced --user root --persistence-mode --verbose

I updated initramfs with update-initramfs -u

Am I missing something ? How can I make dev/nvidia-caps* devices appear on boot?

Thanks in advance