How to tune mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance

How to tune Linux OFED mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance of IB FDR and QDR adapters for memory intensive / high bandwidth MPI communication.

Reference to the doc or information is appreciated.

Mellanox have a very comprehensive server tuning guide. see this: http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf

Yari,

Thanks for the pointer. I looked at the the tuning guide, it is good, but targeted for ethernet over IB performance and doesn’t delve much into details of IB driver settings, although there is a section about autotuning.

But you may have some more, see the list of params here for kernel modules:

We see that tuning/increasing the default params in /sys/module/mlx4_core/parameters resolve issues, we even see some performance boost for our use case. (when we increase log_num_mtt, log_mtts_per_seg).

We really need to understand how to obtain max performance tuning these and other driver params. I doubt the auto-tuners would resolve it for us…

ll /sys/module/mlx4_core/parameters/

-rw-r–r-- 1 root root 4096 Mar 11 17:23 block_loopback

-rw-r–r-- 1 root root 4096 Mar 11 17:23 debug_level

-rw-r–r-- 1 root root 4096 Mar 11 17:23 enable_pre_t11_mode

-r–r–r-- 1 root root 4096 Mar 11 17:23 enable_qinq

-r–r–r-- 1 root root 4096 Mar 11 17:23 enable_qos

-r–r–r-- 1 root root 4096 Mar 11 17:23 fast_drop

-r–r–r-- 1 root root 4096 Mar 11 17:23 high_rate_steer

-rw-r–r-- 1 root root 4096 Mar 11 17:23 internal_err_reset

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_mtts_per_seg

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_cq

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_mac

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_mcg

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_mgm_entry_size

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_mpt

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_mtt

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_qp

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_num_srq

-r–r–r-- 1 root root 4096 Mar 11 17:23 log_rdmarc_per_qp

-r–r–r-- 1 root root 4096 Mar 11 17:23 msi_x

-r–r–r-- 1 root root 4096 Mar 11 17:23 pfcrx

-r–r–r-- 1 root root 4096 Mar 11 17:23 pfctx

-r–r–r-- 1 root root 4096 Mar 11 17:23 probe_vf

-r–r–r-- 1 root root 4096 Mar 11 17:23 set_4k_mtu

-r–r–r-- 1 root root 4096 Mar 11 17:23 sr_iov

-r–r–r-- 1 root root 4096 Mar 11 17:23 use_prio

ll /sys/module/mlx4_en/parameters

total 0

-r–r–r-- 1 root root 4096 Mar 11 17:40 enable_sys_tune

-r–r–r-- 1 root root 4096 Mar 11 17:40 inline_thold

-r–r–r-- 1 root root 4096 Mar 11 17:40 mem_node

-r–r–r-- 1 root root 4096 Mar 11 17:40 num_rx_rings

-r–r–r-- 1 root root 4096 Mar 11 17:40 udp_rss

Thanks,

Mous.

Hi Mous,

the list of parameters you present is applicable for various modes and applications.

The Tuning guide i provided you is not only for IPoIB, it is also more general purposes IB only applications (That uses RDMA). One god example is the log_num_mtt setting.

I am not really familiar with your application and what type of resources it specifically need, You can either open a ticket with Mellanox support or expend more here about what exactly are you looking to do.

Thanks!