And I try another bare metal with Ubuntu 20.04, kernel 5.13.0-27-generic, nvidia-driver 510.47, GPU is GTX Titan X and it works. But the same conbination of kernel and driver is not work on ESXi.
As generix said, it seems like ESXi can’t support nvidia-driver >= 470 greatly.
Furthermore, to confirm kernel 5.13 can work with low version nvidia-driver, I made a bare metal with Ubuntu 20.04, kernel 5.13.0-28-generic, nvidia-driver 465.31, GPU is RTX 3090, it still works.
Because the apt source stopped to provide nvidia-driver under 470, I download .run file from nvidia website and install manually. To prevent the driver auto upgrade, I stopped autoupgrade function of apt.
Hope it can helps! <3