I use a CPU of AMD 5975WX, and four 4090 graphics cards. cuda version is cuda12, pytoch version is 2.0 [image] I noticed that the last call stack is on cuda [image] See attachment for bug report，My code is also in the attachment ex002_DataParallel.py (6.4 KB) nvidia-bug-report.log.gz (1.2 MB) …

(1) Post text as text, not as images. (2) Cut and paste relevant error messages instead of attaching some giant log file. Out of interest: What kind of power supply is used in this system? 3200 Watts?

cudalog.rtf (14.7 KB) ok, here is the wrong text report。 Deadlock in a single machine multi-gpu using dataparlel when cpu is AMD opened 03:35PM - 11 Feb 21 UTC (UTC) [ROBOTICSENGINEER] ROBOTICS…

As far as I understand the linked thread, the working hypothesis there is that the issue is due to an incompatibility between NVIDIA’s NCCL and AMD’s IOMMU. That is outside my area of expertise, and frankly, appears to have nothing to do with CUDA. Off-hand I do not know of a sub-forum for NCCL (in…

[image] 我用的CPU是AMD 5975WX，显卡是4块4090。cuda版本为cuda12，pytoch版本为2.0 Linux Then you should likely check with Supermicro if this is a supported setup. Manually fiddling with the ACS bit: https://forums.developer.nvidia.com/t/multi-gpu-peer-to-peer-access-failing-on-tesla-k80/397…

The motherboard I’m using is Supermicro’s M12SWA-TF, I’m sure the BIOS has been updated to the latest from the official website, and the ASC has been turned off. But my situation is still the same, and I am not sure whether the 4090 supports P2P, because I see that the link you shared is running sim…

The failures in your attachments indicate a problem with the motherboard. You should address this with Supermicro.

[25C416E52BE232B8F17EACFB7141D6B1] I don’t have this problem with 4 A100s, only 4090s currently have this problem The platform is still the original platform, I only changed the graphics card please help me

I have communicated with Supermicro technical staff, they recognized the problem of ACS, but after checking my settings, they confirmed that ASC has been turned off, and they asked me to take 4 A100s for experiment, it is indeed possible to run p2psimple, but when I change Back to 4090, the problem …

FWIW, I have 2 4090s and I’m having same problems training deep learning models on a system that worked fine with 2 3090s. I’ve tried all the bios settings (which I didn’t need with 3090s) I’ve tried NCCL_P2P_DISABLE=1 that sometimes works. I also get a very hard lockup training with Tensorflow 2…

Its a bummer. About to get my WRX80 WS next week with 2 4090s and 5975WX. Why is this issue not widespread? Aren’t workstation builders not seeing this issue when assembling/selling AMD WS with multiple 4090s?

Is turning amd_iommu off the solution in some configurations? [image] Standard nVidia CUDA tests fail with dual RTX 4090 Linux box Linux Did it work and scale with amd_iommu turned off?

Parallel training with 4 cards 4090 cannot be performed on AMD 5975WX， stuck at the beginning

Accelerated Computing CUDA CUDA Programming and Performance

akshu2023 February 6, 2023, 9:17pm 14

@jaybob20 , you might be aware from the other ticket. An issue is being worked on by nVidia.

Topic		Replies	Views
我用的CPU是AMD 5975WX，显卡是4块4090。cuda版本为cuda12，pytoch版本为2.0 Linux chinese	6	2159	December 29, 2022
Standard nVidia CUDA tests fail with dual RTX 4090 Linux box Linux	54	22903	April 29, 2024
P2P Communication Fails 1080ti->1080ti. IOMMU & ACS disabled Linux	1	1813	February 15, 2020
TAO5 - Detectnet_v2 - MultiGPU TAO API Stuck - EXTRA GPU TAO Toolkit	13	1355	November 7, 2023
Strange problem with CUDA on 3 GPUS (5090, 6000 Ada, RTX8000) CUDA Setup and Installation cuda , pytorch	2	1119	June 23, 2025
simpleP2P example and multi-GPU network training causes system freeze and ERR in nvidia-smi Linux	6	4764	February 7, 2019
CUDA P2P crash with threadripper CUDA Programming and Performance	5	1258	November 17, 2017
RTX4090, torch, kernel tried to execute NX-protected page Linux	7	1504	May 13, 2024
System hangs after executing p2p Bandwidth test on Tesla k40 NVIDIA gpus CUDA Programming and Performance	4	1554	February 3, 2018
Low P2P GPU bandwidth performance between GeForce GPUs CUDA Programming and Performance	20	2265	October 9, 2024

Parallel training with 4 cards 4090 cannot be performed on AMD 5975WX， stuck at the beginning

Related topics