Support for "INBOX drivers?" for 18.04/connected mode?

Infrastructure & Networking Software And Drivers Mellanox OFED
,
1

I saw at http://www.mellanox.com/page/products_dyn?product_family=26 http://www.mellanox.com/page/products_dyn?product_family=26

"Linux Inbox Drivers

Mellanox Adapters’ Linux VPI Drivers for Ethernet and InfiniBand are also available Inbox in all the major distributions, RHEL, SLES, Ubuntu and more. Inbox drivers enable Mellanox High performance for Cloud, HPC, Storage, Financial Services and more with the Out of box experience of Enterprise grade Linux distributions."

I’ve used this for several generations of Mellanox cards of for last decade with a wide variety of linux distributions. Just config with /etc/network/interfaces, ifconfig ib0 works, datagram/connected mode works, IPoIB works.

From what I can tell with 18.04 it doesn’t work. The default (ubuntu supplied/INBOX) drivers:

# cat /sys/class/net/ib0/mode

datagram

# echo connected > /sys/class/net/ib0/mode

-bash: echo: write error: Invalid argument

I found docs that with the MLNX_OFED drivers that you just:

# cat ib_ipoib.conf

options ib_ipoib ipoib_enhanced=0

I get:

[ 57.573664] ib_ipoib: unknown parameter ‘ipoib_enhanced’ ignored

Is there any documentation for the “INBOX drivers”? Anyone know how to get connected mode working? I’m guess the 18.04 drivers are too old to have the ability to disable enhanced mode, but to new to have the connected mode working by default.

2

Hi Bill,

Here is the link to the Release Notes for Ubuntu 18.04 Inbox Driver. Section 2

Changes and New Features lists the support for Enhanced IPoIB for ConnectX-4

cards. Also, I am attaching the link for the User Manual for your reference.

The documentation was quite helpful, thanks.

Which card are you using?

root@k1:~# lspci | grep -i mellanox

06:00.0 Infiniband controller: Mellanox Technologies MT27700 Family

Also, you mentioned you are receiving the following

error ib_ipoib: unknown parameter ‘ipoib_enhanced’ ignored. Is

this after testing with MLNX_OFED? If yes, which OFED version did you test it with?

I was tried what comes with ubuntu 18.04 and with

“mlnx-en-4.4-2.0.7.0-ubuntu18.04-x86_64”. I’m open to a solution with either.

Do you have any recommendations for getting ipoib connected mode to work with

either the native ubuntu 18.04 drivers or the MLNX_OFED drivers?

I tried this because it was recommended in the Mellanox forum:

root@k2:/etc/modprobe.d# cat ib_ipoib.conf

options ib_ipoib ipoib_enhanced=0

But when I boot the above mentioned error shows up.

When I install the MLNX_OFED driver, I don’t see any errors. Here’s the summary:

Device #1:

3

I had the wrong package installed, MLNX_OFED fixed the problem and connected mode now works.

Is there any way to get connected mode working with the drivers that come with Ubuntu 18.04?

4

Hi Bill,

It would be great if you could run the following commands to validate if the parameter ipoib_enhanced is supported on your system with inbox driver:

  1. #find /lib/modules -iname ‘ib_ipoib’

(Example output from host running Ubuntu 16.04)

/lib/modules/4.4.0-134-generic/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

/lib/modules/4.4.0-116-generic/updates/dkms/ib_ipoib.ko

The first line denotes Inbox driver and second line denotes mlnx driver

2.#modinfo /lib/modules/4.4.0-134-generic/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

Check which parameters are supported(You will see in output, lines that start with “parm:”).

Compare the same by running modinfo with one which has mlnx driver

#modinfo /lib/modules/4.4.0-116-generic/updates/dkms/ib_ipoib.ko

Please share the outputs.

5

Hi Bill,

Here is the link to the Release Notes for Ubuntu 18.04 Inbox Driver. Section 2 Changes and New Features lists the support for Enhanced IPoIB for ConnectX-4 cards. Also, I am attaching the link for the User Manual for your reference. Which card are you using? Also, you mentioned you are receiving the following error [ 57.573664] ib_ipoib: unknown parameter ‘ipoib_enhanced’ ignored. Is this after testing with MLNX_OFED? If yes, which OFED version did you test it with? Also, for any questions related Inbox Driver, it would be great to reach out to OS Vendor.

http://www.mellanox.com/pdf/prod_software/Ubuntu_18.04_Inbox_Driver_Release_Notes.pdf http://www.mellanox.com/pdf/prod_software/Ubuntu_18.04_Inbox_Driver_Release_Notes.pdf

http://www.mellanox.com/pdf/prod_software/Ubuntu_18_04_Inbox_Driver_User_Manual.pdf http://www.mellanox.com/pdf/prod_software/Ubuntu_18_04_Inbox_Driver_User_Manual.pdf