Troubleshooting the iSER enable on Mellanox ConnectX-2 EN and running iSCSI traffic

, ,
1

Hi,

I have folowing card on my system.

lspci | grep -i mellanox

02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

I am running CentOs 6.5.

uname -a

Linux RV-189 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/issue

CentOS release 6.5 (Final)

Kernel \r on an \m

I am following the link . https://community.mellanox.com/s/article/howto-setup-rdma-connection-using-inbox-driver--rhel--ubuntu-x

HowTo setup RoCE connection using Inbox driver (RHEL, Ubuntu) https://community.mellanox.com/s/article/howto-setup-rdma-connection-using-inbox-driver--rhel--ubuntu-x

I do not see the enp5s0 or any other interface which can describe the Mellanox card on my system.

#ifconfig enp5s0 12.12.12.1/24 up

Can someone pl let me know what am I missing?

Thanks in advance,

Anil

2

Ophir,

This <Infrastructure & Networking - NVIDIA Developer Forums Infrastructure & Networking - NVIDIA Developer Forums > is the exact link what I

was following. I installed packages from there and got stuck up at

following section.

Configure port parameters:

Configure IP Address and enable the port.

*It can be done via console scripts such as nmtui, ifconfig (not permanent)

or other method*

For example:

#ifconfig enp5s0 12.12.12.1/24 <http://12.12.12.1/24 http://12.12.12.1/24 > up

I tried to install these packages again and it came up with following

response:

yum -y groupinstall “InfiniBand Support”

Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

presto, refresh-packagekit, security, tmprepo, verify,

: versionlock

Loading support for CentOS kernel ABI

Loading mirror speeds from cached hostfile

Setting up Group Process

Checking for new repos for mirrors

Package libibverbs-utils-1.1.7-1.el6.x86_64 already installed and latest

version

Package libibverbs-1.1.7-1.el6.x86_64 already installed and latest version

Package librdmacm-utils-1.0.17-1.el6.x86_64 already installed and latest

version

Package libibcm-1.0.5-3.el6.x86_64 already installed and latest version

Package rdma-3.10-3.el6.noarch already installed and latest version

Package librdmacm-1.0.17-1.el6.x86_64 already installed and latest version

Package libmlx4-1.0.5-4.el6.1.x86_64 already installed and latest version

Package libipathverbs-1.2-4.el6.x86_64 already installed and latest version

Package libmthca-1.0.6-3.el6.x86_64 already installed and latest version

Package libnes-1.1.3-1.el6.x86_64 already installed and latest version

Package libcxgb3-1.3.1-1.el6.x86_64 already installed and latest version

Package ibsim-0.5-7.el6.x86_64 already installed and latest version

Package dapl-2.0.34-1.el6.x86_64 already installed and latest version

Package libibmad-1.3.9-1.el6.x86_64 already installed and latest version

Package libibumad-1.3.8-1.el6.x86_64 already installed and latest version

Package rds-tools-2.0.6-3.el6.x86_64 already installed and latest version

Package ibutils-1.5.7-8.el6.x86_64 already installed and latest version

Package ibacm-1.0.8-0.git7a3adb7.el6.x86_64 already installed and latest

version

Warning: Group infiniband does not have any packages.

No packages in any requested group available to install or update

yum -y install perftest infiniband-diags

Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

presto, refresh-packagekit, security, tmprepo, verify,

: versionlock

Loading support for CentOS kernel ABI

Loading mirror speeds from cached hostfile

Setting up Install Process

Package perftest-2.0-2.el6.x86_64 already installed and latest version

Package infiniband-diags-1.5.12-5.el6.x86_64 already installed and latest

version

Nothing to do

So it seems like theses packages are there on my system.

If you think, I need to do following extra, I can do that.

*You can also install MLNX_OFED, see Mellanox website for the driver

package.*

Click here, and follow the links.

*Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux

(MLNX_OFED)*

<http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers >

On Tue, Sep 16, 2014 at 12:02 PM, ophirmaor <socialmedia@mellanox.com mailto:socialmedia@mellanox.com >

3

eth4 and eth5 will be the interfaces you need to configure. correct.

Do you have two hosts? if so, you can configure one host (with one Mellanox adapter) as target (you can use TGT for example) and another server equipped with Mellanox adapter as the initiator.

you can follow HowTo Configure TGT enabled with iSER Transport for RHEL https://community.mellanox.com/s/article/howto-configure-tgt-enabled-with-iser-transport-for-rhel for that.

Ophir.

4

Ophir,

I downloaded driver from your web site and probably it seems to behave much

better now. I am giving you output of the commands what you asked earlier.

ibdev2netdev

mlx4_0 port 2 ==> eth4 (Down)

mlx4_0 port 1 ==> eth5 (Down)

ofed_info | head

MLNX_OFED_LINUX-2.3-1.0.1 (OFED-2.3-1.0.0):

ar_mgr:

ar_mgr/e338fbc05dd6b480f8505791e50700b7/ar_mgr-1.0-0.23.g20d8d45.tar.gz

bupc:

bupc/6b0f0c8b7d7ff2e258e600d65f6da291/bupc-2.18.0-421.src.rpm

cc_mgr:

cc_mgr/4a29eb085479e51b6bb278efcddfe379/cc_mgr-1.0-0.22.g20d8d45.tar.gz

So interface eth4 and eth5 can be used for iSER, correct?

I have two cards and want to set one card in one system as iSER Initiator

and other card as iSER target.

Can I use these two cards in the same system as Initiator and target?

Thanks,

Anil

5

Hi Anil,

Do you have MLNX_OFED installed? did you use OFED inbox driver?

Note that the example is from RHEL7 and over there the interface name is different (not eth1,eth2 …) but enp5s0 or similar.

try

#yum -y groupinstall “InfiniBand Support”

#yum -y install perftest infiniband-diags

  • for the inbox driver, it may be the easiest.

Please send me the output of

#ibdev2netdev

#ofed_info | head

you didn’t reach the level of testing ISER, first you need to make sure that RoCE is running, and for that you need RDMA support.

Ophir.

6

Run

yum install “InfiniBand Support”

See this post

HowTo Setup RDMA Connection using Inbox Driver (RHEL, Ubuntu) https://community.mellanox.com/s/article/howto-setup-rdma-connection-using-inbox-driver--rhel--ubuntu-x

You can also install MLNX_OFED, see Mellanox website for the driver package.

Click here, and follow the links.

Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED) http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

Thanks,

Ophir.

7

Thanks Ophir for the reply.

Hi Anil,

Do you have MLNX_OFED installed? did you use OFED inbox driver?

Anil> Whether MLNX_OFED should be available as inbox? I tried your commands

and they did not run, please see the output given below.

yum install “InfiniBand Group”

Loaded plugins: aliases, changelog, downloadonly, fastestmirror, kabi,

presto, refresh-packagekit,

: security, tmprepo, verify, versionlock

Loading support for CentOS kernel ABI

Loading mirror speeds from cached hostfile

Setting up Install Process

No package InfiniBand Group available.

Error: Nothing to do

ibdev2netdev

bash: ibdev2netdev: command not found

ibdev2netdev

bash: ibdev2netdev: command not found

ofed_info | head

bash: ofed_info: command not found

you didn’t reach the level of testing ISER, first you need to make sure

that RoCE is running, and for that you need RDMA support.

Anil>> I knew, I am missing something. yes, I want to make sure that RoCE

is running first

Thanks,

Anil