I’ve been testing a distributed GPU setup and found that poor network performance can leave the GPUs idle while they wait for data, particularly when latency or packet loss increases.
Has anyone else experienced this? If so, what did you do to reduce the bottleneck?
Hello~
with only your description, it’s hard to clearly figure out what the problem is…
If you can provide more details (Lab Topo and tests scenariio and RDMA perftest results..and so on)
/HyungKwang
My original post was too vague. Let me go into my setup a little more:
I’m testing a distributed setup where GPU nodes communicate between separate locations over the public internet through an encrypted tunnel. I’ve seen GPU utilisation drop during network-dependent stages, particularly as latency and packet loss increase, and I’m trying to isolate exactly where the bottleneck begins.
I’m putting together a clearer topology and test results. Beyond RDMA perftest, which measurements would you recommend including to distinguish a network bottleneck from a workload or configuration issue?