When I perform multi-card parallel training on 4-card 4090, it will get stuck or CPU soft lock

Graphics / Linux Linux
1

I have tested a variety of parallel pytorch programs on many machines, many of which will crash, but some codes will not crash on NGC 22.12-py3
The following are the versions of the driver, cuda, pytorch I have tested
Driver version：525.25.60.13/525.78.01
cuda versin：11.6/11.8/12.0
pytorch version：1.14/2.0/The master version of pytorch compiled with cuda12

d6ed8fa33f3c01add71299d302c3a87
d6ed8fa33f3c01add71299d302c3a871128×722 148 KB

5a9fd00f14cb4e08c853541a8c055dd
5a9fd00f14cb4e08c853541a8c055dd1920×1440 376 KB

image
image1485×660 119 KB

image
image1702×334 88.6 KB

I’m not sure if this is a driver or library related issue

2

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

3

nvidia-bug-report.log.gz (1.2 MB)
this is my bug report。thanks a million

4

please help me

5

please help me

6

The Xserver is crashing and restarting, please disable it. sudo systemctl disable display-manager
Then please install and set up nvidia-persistenced and check if that resolves your issue.