Hi,
I’m testing the CUDA memory model described here:
According to the documentation, a thread_scope_system atomic operation is atomic only if one of several conditions is satisfied.
For GPU memory accessed by multiple GPUs, the relevant condition appears to be:
it affects an object in GPU memory, only GPU threads access it, and
cudaDeviceGetP2PAttribute(..., cudaDevP2PAttrNativeAtomicSupported, srcDev, dstDev) == 1
However, I’m seeing behavior that I don’t know how to interpret.
# Hardware
I have a machine with 8 GPUs.
For the GPU pair I tested:
src = GPU0
dst = GPU4
Peer access:
cudaDeviceCanAccessPeer(GPU0, GPU4) == 1
cudaDeviceCanAccessPeer(GPU4, GPU0) == 1
The P2P capability matrix reports:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
GPU0 X OK OK OK OK OK OK OK
GPU1 OK X OK OK OK OK OK OK
GPU2 OK OK X OK OK OK OK OK
GPU3 OK OK OK X OK OK OK OK
GPU4 OK OK OK OK X OK OK OK
GPU5 OK OK OK OK OK X OK OK
GPU6 OK OK OK OK OK OK X OK
GPU7 OK OK OK OK OK OK OK X
However, the Native Atomic capability matrix reports:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7
GPU0 X OK OK OK NS NS NS OK
GPU1 OK X OK OK NS NS OK NS
GPU2 OK OK X OK NS OK NS NS
GPU3 OK OK OK X OK NS NS NS
GPU4 NS NS NS OK X OK OK OK
GPU5 NS NS OK NS OK X OK OK
GPU6 NS OK NS NS OK OK X OK
GPU7 OK NS NS NS OK OK OK X
cudaDeviceGetP2PAttribute reports
cudaDeviceGetP2PAttribute(
&value,
cudaDevP2PAttrNativeAtomicSupported,
GPU0,
GPU4);
value == 0
# Test
I wrote a producer/consumer test using
cuda::atomic_ref<int, cuda::thread_scope_system>
with
memory_order_release
memory_order_acquire
Both the synchronization variable (`flag`) and the payload are allocated on the destination GPU.
The synchronization pattern is
writer
payload writes
↓
flag.store(memory_order_release)
-------------------------------
reader
flag.load(memory_order_acquire)
↓
read payload
The test runs for **200000 iterations**.
Results:
payload and flag are both on dst
src = GPU0
dst = GPU4
case 1
payload = dst
flag = dst
writer = GPU4
reader = GPU0
dst release -> src acquire
PASS
case 2
payload = dst
flag = dst
writer = GPU0
reader = GPU4
src release -> dst acquire
PASS
summary: 2/2 passed
No stale payloads, no mismatches, and no timeout were observed.
# Other Tests
I also tested remote atomic RMW operations (fetch_add, etc.).
Those behave exactly as expected according to the documentation when
cudaDevP2PAttrNativeAtomicSupported == 0
The only unexpected case is
-
memory_order_release -
memory_order_acquire -
store/load synchronization
# Minimal Example
The essential synchronization logic is:
Writer:
write_payload(ch, i);
flag_ref.store(i + 1, cuda::memory_order_release);
Reader:
wait_until(flag_ref.load(cuda::memory_order_acquire) >= i + 1);
read_payload(ch);
The full test program is attached below so that it can be compiled and run directly. If my test itself is incorrect, I would really appreciate any suggestions or corrections.
# Questions
-
Is this behavior expected?
Should a
thread_scope_systemrelease/acquire store/load test succeed even when?
cudaDevP2PAttrNativeAtomicSupported == 0
-
Does
cudaDevP2PAttrNativeAtomicSupportedonly affect remote read-modify-write atomics (fetch_add,exchange,compare_exchange, etc.), while release/acquire store/load synchronization may still work? -
Or is my test simply observing implementation behavior that is * **not guaranteed *** by the CUDA memory model?
-
If this case is not guaranteed , what kinds of failures should developers actually expect?
I’m mainly trying to understand the intended semantics of the CUDA memory model here.
Although my test passes consistently, I’m not sure whether this is guaranteed behavior or simply happens to work on this platform.
Thanks!
payload_flag_on_dst.log (1.1 KB)