Crash my ubuntu when using the GPU

THE UBUNTU CRASH AND SHUTDOWN when I use the graphic card in deep learning model training using PyTorch code. The issue occurs randomly with the followingmessages.

In addition, the /var/log/messages file show the message

My environments are as follows:
Ubuntu 18.04
NVIDIA graphic card RTX 3090

My Nvidia-bug-report.log file is here.
nvidia-bug-report.log.gz (2.4 MB)

Below is what I’ve tried.
I edited my /etc/gdm3/custom.conf file from #WaylandEnable=false to WaylandEnable=false.
I tried to run the sudo prime-select nvidia but the terminal showed messages Info: the nvidia profile is already set.

Is there any solution?