Testing MCX455A bandwidth between Dell servers

I am testing MCX455A bandwidth between Dell servers using the commands:

Server: Ib_write-bw –d mlx5_0 –i 1 –a –F

and

Client: Ib_write-bw –d mlx5_0 –i 1 –a –F

The results looks fine until # bytes = 65536

Then I get the message:

mlx5: usb1 : got completion with errors

00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000

00000000 00008813 08000029 40807dd3

Problems with warm up

This test used to work to completion and I don’t think I’ve changed the configuration. I have the same problem with multiple cards (I have 13).

Thank you. That fixed it. Can you explain why?

–all / -a flag is a feature that runs traffic on all message size from 2^1 to 2^23

When not using this flag, the default message size is 64KB.

Which means that if the server side doesn’t have the ‘-a’ flag set, it will prepare it’s resources for 64KB messages… And when the client tries to send 128KB messages, it will fail.

Are you sure you used ‘-a’ flag on both client and server?

Can you please try and put the -a flag as the last flag in the command line on the server?

For the client it should be the last flag before the IP of the server.