To the best of my knowledge, this is not possible right now as the ports are fixed both on the client- and server-side. I’ve asked the same question here: Configurable Ports for CloudXR - XR – VR/AR/MR / CloudXR (VR and AR Streaming) - NVIDIA Developer Forums. If at least the client-side ports would be configurable, I think this should be possible with a suitable network loadbalancer.
So you could probably build a service that wraps the vSphere APIs to orchestrate the VMs, right? Basically, your client application would talk to this service, asking for a VM and the service knows if there’s still availability by talking to vSphere and can directly return the ip address of the VM to be used so that your client application can directly connect.