Nvidia-modeset: ERROR: GPU:6: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])

Env info

Ubuntu 18.04
Driver Version: 418.xx
CUDA Version: 10.1
8 RTX 2080 Ti graphic cards

Error Info

Jul  3 10:24:01 klfy-SYS-4028GR-TR2 kernel: [   19.780981] nvidia-modeset: ERROR: GPU:6: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Jul  3 10:24:01 klfy-SYS-4028GR-TR2 kernel: [   19.788855] nvidia-modeset: ERROR: GPU:6: Failed to allocate display engine core DMA push buffer
Jul  3 10:24:09 klfy-SYS-4028GR-TR2 kernel: [   27.896161] nvidia-modeset: ERROR: GPU:6: Display engine push buffer channel allocation failed: 0x65 (Call timed out [NV_ERR_TIMEOUT])
Jul  3 10:24:09 klfy-SYS-4028GR-TR2 kernel: [   27.901129] nvidia-modeset: ERROR: GPU:6: Failed to allocate display engine window channels

The corresponding GPU card will show ERR!.

The log file:
nvidia-bug-report.log (2.9 MB)