UD Pingpong and MTU on RoCE

Infrastructure & Networking Software And Drivers Mellanox OFED
1

With the MTU set for the ethernet device to be 9000 (jumbo frames), I can only send 4012 bytes using ibv_ud_pingpong. Is there a reason why its 4012?

I understand there are ethernet headers to be added with the payload but I though they were 40 bytes for a UD QP.

If I send a 4096-byte sized packet on a RC QP I see that the PSN only gets incremented once.

What am I missing here with regard to UD?

The only reference I found for the ibv_ud_pingpong and 4012 bytes was here: 6.5 Technical Notes Red Hat Enterprise Linux 6 | Red Hat Customer Portal 6.5 Technical Notes Red Hat Enterprise Linux 6 | Red Hat Customer Portal

I’m using ConnectX-3 HCAs on Ubuntu 14.04-based hosts connected back-to-back.

Thanks!