Infiniband on Linux... defaults to datagram mode somehow?

Infrastructure & Networking Adapters And Cables InfiniBand/VPI Adapter Cards
#1

RHEL v5.5

kernel 2.6.18-194.el5

What I have discovered, is that if you don’t have “ONBOOT=yes” in your ifcfg-ib# script… the interface will come up in “datagram” mode, regardless of what is in /etc/infiniband/openib.conf. Can anyone tell me why this happens or if its by design even?

cat openib.conf

Start HCA driver upon boot

ONBOOT=yes

Node description

NODE_DESC=$(hostname -s)

Wait for NODE_DESC_TIME_BEFORE_UPDATE sec before node_desc update

NODE_DESC_TIME_BEFORE_UPDATE=10

Load UCM module

UCM_LOAD=no

Load RDMA_CM module

RDMA_CM_LOAD=yes

Load RDMA_UCM module

RDMA_UCM_LOAD=yes

Increase ib_mad thread priority

RENICE_IB_MAD=no

Run sysctl performance tuning script

RUN_SYSCTL=yes

Load MTHCA

MTHCA_LOAD=yes

Load QIB

QIB_LOAD=yes

Load MLX4 modules

MLX4_LOAD=yes

Load MLX4_EN module

MLX4_EN_LOAD=yes

Start FC service

MLX4_FC_LOAD=no

Load MLX4_VNIC module

MLX4_VNIC_LOAD=no

Load CXGB3 modules

CXGB3_LOAD=yes

Load NES modules

NES_LOAD=yes

Load IPoIB

IPOIB_LOAD=yes

Enable IPoIB Connected Mode

SET_IPOIB_CM=yes

Load SDP module

SDP_LOAD=no

Load SRP module

SRP_LOAD=no

Load ISER module

ISER_LOAD=no

Load RDS module

RDS_LOAD=no

Enable SRP High Availability daemon

SRPHA_ENABLE=no

SRP_DAEMON_ENABLE=no

#2

Interesting. I’m more used to RHEL 6.x instead of RHEL 5.x.

With RHEL 6, settings are in the /etc/rdma/ directory, not /etc/infiniband/.

Is there an /etc/rdma/ directory on your system?

#3

It does not have that directory.

#4

Well, I think its just what comes with the kernel source…

################################################################################

module ib_ipoib

filename: /lib/modules/2.6.18-194.el5/updates/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

license: Dual BSD/GPL

description: IP-over-InfiniBand net driver

author: Roland Dreier

srcversion: 22D6DA2213921B790D1BD7D

depends: ib_cm,ipv6,ib_core,ib_core,ib_sa,ipoib_helper

vermagic: 2.6.18-194.el5 SMP mod_unload gcc-4.1

parm: max_nonsrq_conn_qp:Max number of connected-mode QPs per interface (applied only if shared receive queue is not available) (int)

parm: mcast_debug_level:Enable multicast debug tracing if > 0 (int)

parm: send_queue_size:Number of descriptors in send queue (int)

parm: recv_queue_size:Number of descriptors in receive queue (int)

parm: lro:Enable LRO (Large Receive Offload) (bool)

parm: lro_max_aggr:LRO: Max packets to be aggregated (default = 64) (int)

parm: debug_level:Enable debug tracing if > 0 (int)

parm: mc_sendonly_timeout:Enable debug tracing if > 0 (int)

################################################################################

module ib_cm

filename: /lib/modules/2.6.18-194.el5/updates/kernel/drivers/infiniband/core/ib_cm.ko

license: Dual BSD/GPL

description: InfiniBand CM

author: Sean Hefty

srcversion: 08CBDB36AB5BDE6F83763F9

depends: ib_core,ib_mad,ib_sa,ib_core

vermagic: 2.6.18-194.el5 SMP mod_unload gcc-4.1

parm: max_timeout:Maximum IB CM per message timeout (default=21, or ~8 seconds) (int)

################################################################################

This is an HP Ibrix storage server, and its sort of “locked down” as far as packages go. I was just curious if anyone had an idea as to why it was falling to UD mode. Thanks for any help you guys can provide.

#5

Which Infiniband packages are you using?

Are you using the ones that come with RHEL 5.5, or are you using OFED packages?

#6

Are you using MLNX OFED 2.0?

Starting from MLNX OFED 2.0, the default mode is UD.

From the User Manual http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v2.0-2.0.5.pdf section 1.3.4:

The IP over IB (IPoIB) driver is a network interface implementation over InfiniBand. IPoIB

encapsulates IP datagrams over an InfiniBand connected or datagram transport service. IPoIB

pre-appends the IP datagrams with an encapsulation header, and sends the outcome over the

InfiniBand transport service. The transport service is Unreliable Datagram (UD) by default, but it

may also be configured to be Reliable Connected (RC). The interface supports unicast, multicast

and broadcast. For details, see Chapter 4.3, “IP over InfiniBand”.

IPoIB UD performance was significantly improved in MLNX OFED 2.0, thus it was set to be the default mode.

You can still use IPoIB CM mode if you wish.

This still doesn’t explain why the config in openibd.conf is not picked up, I suggest you check /etc/init.d/openibd to see if the config file is parsed as expected.

#7

Sorry for the delay, I was travelling internationally, but am back home now.

I’ve just spun up RHEL 5.x in a VM locally to see where things are installed.

It looks like Infiniband support on RHEL 5.x stores everything in files under /etc/ofed/, unlike RHEL 6.x.

There is an /etc/ofed/openib.conf file in there. Have you tried changing the setting in there yet?

#8

I am not using any add on packages. Only what comes with RHEL v5.5. Thanks in advance for any insight.