How to read /proc/interrupts?

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hello,

when I have a look at our Ubuntu 12.04 LTS (Kernel 3.5) /proc/interrupts output I see several interrupts for the installed ConnectX infiniband card:

73: 999+ 999+ 999+ 999+ … mlx4-ib-1-0@PCI Bus 0000:09 …

74: 0000 0000 0000 0000 … mlx4-ib-1-1@PCI Bus 0000:09 …

75: 0000 0000 0000 0000 … mlx4-ib-1-2@PCI Bus 0000:09 …

76: 0000 0000 0000 0000… mlx4-ib-1-3@PCI Bus 0000:09 …

If my interpretation of the availabale documentation is right, each of these interrupts matches one ring buffer of port 1 on the adapter. At least from what you can read at https://www.kernel.org/doc/Documentation/networking/scaling.txt https://www.kernel.org/doc/Documentation/networking/scaling.txt The port is running as in IPoIB datagram mode on a NFS storage filer and is accessed from several clients attached to the same infiniband switch.

With this setup I’m wondering why only interrupts on device mlx4-ib-1-0@PCI Bus 0000:09 are counting up. I used number 999+ above to make it more readable. Is there any option to get interrupts on the other queues? The idea would be to spread them over several CPUs. If yes or no would there be any benefit of doing it?

Best regards.

Markus

2

I understand the multiple interrupt channels are to provide better concurrent performance as you suspect. In my experience using them has not been straightforward.

I am not sure how some of the OFED ULPs like IPoIB use the interrupt channels but the selection of the interrupts channel is made at the verbs layer. After the queue pair is created it can be modified to allow it to target a specific interrupt channel. (in fact it might be in the create_qp call but I am pretty sure that it is done with modify_qp)

Keep in mind that IPoIB is a complete software implementation of the ethernet stack. The interrupts shows are IB interrupts that work the IB queue pair completion channel not traditional ethernet interrupts.

For the ULPs there would have to be some way to expose the modify queue pair interface to the user and I am not sure if that is the case.

Keep digging there may be someone who knows how the interrupt channels can be used in the ULPs.