How to change GPU topology?

hello

I am using 8 RTX 6000 ADA GPUs in supermicro’s 421ge-tnrt product.

I want to allocate 4 GPUs to CPU1 and 4 GPUs to CPU2 using Pytorch DDP.

However, when I checked with the nvidia-smi topo -m command, it outputs SYS and does not allocate to PXB. Is there a way to allocate 4 GPUs at a time while outputting to PXB??

OS : Ubuntu 22.04 and 20.04

    GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity

GPU0 X NODE NODE NODE SYS SYS SYS SYS 0-35,72-107 0
GPU1 NODE X PIX PIX SYS SYS SYS SYS 0-35,72-107 0
GPU2 NODE PIX X PIX SYS SYS SYS SYS 0-35,72-107 0
GPU3 NODE PIX PIX X SYS SYS SYS SYS 0-35,72-107 0
GPU4 SYS SYS SYS SYS X PIX NODE NODE 36-71,108-143 1
GPU5 SYS SYS SYS SYS PIX X NODE NODE 36-71,108-143 1
GPU6 SYS SYS SYS SYS NODE NODE X PIX 36-71,108-143 1
GPU7 SYS SYS SYS SYS NODE NODE PIX X 36-71,108-143 1

Short answer: not at all without buying different hardware.
The topology is the hardware layout, in your case I’m interpreting this as
two cpus with two pci root complexes each
gpu 0 is connected directly to root complex 0 on cpu 0
gpu 1,2,3 connected to the same pci switch on root complex 1 on cpu 0

gpu 4,5 are connected to one pci switch on root complex 0 on cpu 1
gpu 6,7 connected to another pci switch on root complex 1 on cpu 1

So if you want to push data from gpu 0 to gpu4 it needs to traverse the interconnect from cpu 0 to cpu1 (SYS).

PXB would mean all gpus were on different pci switches on the same root complex.