Inconsistent NVIDIA-SMI between two 8x GPU

Available memory showing in nvidia-SMI not consistent between 8x GPUs. This is after a fresh reboot so no tasks have been ran on the GPUs, all GPUs were purchased and installed at the same time into the server.
No displays are plugged into the server.
No VM software is installed.
Ubuntu 22.04 + CUDA/Drivers
As far as I’m aware no other software has been installed.

GPU 0 46068MiB
GPU 1 49140MiB
GPU 2 46068MiB
GPU 3 49140MiB
GPU 4 49140MiB
GPU 5 49140MiB
GPU 6 49140MiB
GPU 7 49140MiB

Is this expected behavior?
535.161.08
Cuda: 12.2

I know available memory can change based on loaded drivers/configurations in memory, but it’s odd that two GPU show a lower amount. 3GB is a lot.

The two gpus have ECC enabled which decreases available memory.

le sigh…
Was right in front of my face and I didn’t even check.

Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.