Hello everyone,
I have recently been involved in a project where I need to support an AI processing infrastructure. I have a Dell PowerEdge XE9689 server (HGX H200) running ESXi 8 Update 3. The GPUs are assigned to virtual machines using PCIe Passthrough.
The issue is that the model provided by the development team requires NVSwitch connectivity, but NVSwitch does not appear to be functioning inside the VMs. I have tested the same setup directly on the server in a bare-metal environment, and everything works correctly. NVIDIA Fabric Manager starts successfully and operates without any issues.
My question is: would using NVIDIA vGPU instead of PCIe Passthrough resolve this problem?
The error I receive when starting the model is:
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized.
Any guidance or suggestions would be greatly appreciated.