Does anyone have experience setting IRQ affinity masks for NVIDIA GPUs? I have a NUMA system with two NUMA nodes of 6 cores each. Each NUMA node is connected to 4 GTX-470 GPUs. Of course, CPUs in one node can talk to GPUs in the other, but the bandwidth is less.
I’d like to configure the IRQ affinity masks such that interrupts (IRQs) from the GPUs in a particular NUMA node are delivered to CPUs in the same node.
Example:
NUMA Node #1 has CPUs 0…5 and GPUs 0…3
NUMA Node #2 has CPUs 6…11 and GPUs 4…7
I want an interrupt raised by GPU 0 to be delivered to CPUs 0, 1, 2, 3, 4, or 5.
Generally speaking, any interrupts raised by GPUs 0…3 should be delivered to CPUs 0…5. Any interrupts raised by GPUs 4…7 should be delivered to CPUs 6…11.
Here’s a basic primer on how to set affinity masks.
I can set CPU affinity masks in Linux by setting the proper bits to /proc/irq/<IRQ#>/smp_affinity. The trick is how to determine the proper IRQ#s and mapping those to GPUs. On my 8 GPU system, /proc/interrupts shows:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11
24: 680 50 79 76 465 0 0 0 28 0 0 0 IO-APIC-fasteoi nvidia, nvidia
30: 492 84 62 52 0 0 0 0 17 422 0 0 IO-APIC-fasteoi nvidia, nvidia
48: 630 99 68 480 0 5 0 0 18 0 0 0 IO-APIC-fasteoi nvidia, nvidia
54: 673 34 83 72 7 0 22 0 453 0 0 0 IO-APIC-fasteoi nvidia, nvidia
Okay, so my 8 GPUs are using IRQs 24, 30, 48, and 54. Now how do I know what GPUs are using which IRQ lines? Does each GPU use more than one IRQ line? Is there a way that I can tell the NVIDIA driver to disconnect GPUs selectively (and thus determine IRQs by elimination)? (Also, is there a way to prevent the devices from sharing IRQ lines?)
Thanks!