OK, I’ve been trying to setup eIPoIB. I have my infiniband network up, ib0 is setup for ipoib I can see the new eth2 interface (virtual eIPoIB device) but nothing in the vifs, so I can’t ping from compute node to compute node.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-39-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Thu May 9 22:31:11 EST 2013 System load: 1.83 Users logged in: 0 Usage of /: 2.7% of 59.39GB IP address for eth0: 192.168.10.101 Memory usage: 0% IP address for ib0: 10.10.10.101 Swap usage: 0% IP address for eth2: 20.20.20.101 Processes: 112 Graph this data and manage this system at https://landscape.canonical.com/ Last login: Thu May 9 21:55:22 2013 from maas.local ubuntu@blade01:~$ sudo su - root@blade01:~# cat /sys/class/net/eth2/eth/vifs root@blade01:~# root@blade01:~# root@blade01:~# ibstat CA 'mlx4_0' CA type: MT25418 Number of ports: 2 Firmware version: 2.8.0 Hardware version: a0 Node GUID: 0x001b78ffff33ee58 System image GUID: 0x001b78ffff33ee5b Port 1: State: Active Physical state: LinkUp Rate: 20 Base lid: 5 LMC: 0 SM lid: 3 Capability mask: 0x02510868 Port GUID: 0x001b78ffff33ee59 Link layer: InfiniBand Port 2: State: Active Physical state: LinkUp Rate: 20 Base lid: 2 LMC: 0 SM lid: 1 Capability mask: 0x02510868 Port GUID: 0x001b78ffff33ee5a Link layer: InfiniBand root@blade01:~# ifconfig ib0 ib0 Link encap:UNSPEC HWaddr A0-00-01-00-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:10.10.10.101 Bcast:10.10.10.255 Mask:255.255.255.0 inet6 addr: fe80::21b:78ff:ff33:ee59/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:6 errors:0 dropped:0 overruns:0 frame:0 TX packets:30 errors:0 dropped:8 overruns:0 carrier:0 collisions:0 txqueuelen:1024 RX bytes:1515 (1.5 KB) TX bytes:5752 (5.7 KB) root@blade01:~# ping 10.10.10.102 PING 10.10.10.102 (10.10.10.102) 56(84) bytes of data. 64 bytes from 10.10.10.102: icmp_req=2 ttl=64 time=2.24 ms 64 bytes from 10.10.10.102: icmp_req=3 ttl=64 time=0.033 ms ^C --- 10.10.10.102 ping statistics --- 3 packets transmitted, 2 received, 33% packet loss, time 2000ms rtt min/avg/max/mdev = 0.033/1.141/2.249/1.108 ms root@blade01:~# ifconfig eth2 eth2 Link encap:Ethernet HWaddr 00:1b:78:33:ee:59 inet addr:20.20.20.101 Bcast:20.20.20.255 Mask:255.255.255.0 inet6 addr: fe80::21b:78ff:fe33:ee59/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) root@blade01:~# ping 20.20.20.102 PING 20.20.20.102 (20.20.20.102) 56(84) bytes of data. ^C --- 20.20.20.102 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2015ms root@blade01:~# cat /sys/class/net/eth2/eth/vifs root@blade01:~# ???? root@blade01:~# ethtool -i eth2 driver: eth_ipoib version: 1.0.0 firmware-version: 1 bus-info: ib0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no root@blade01:~#
Thanks for the info , I don’t have access to my lab on the weekend but I’ll have to check it out when back in the office. Do you have the details of what you changed on your ubuntu system ?
In the meantime, with the ipoibd daemon startup bit that’s rejecting the OS… does it seem like a binary file or does it look like a shell script gets run first? Kind of thinking the OS check might be in a shell script before the proper daemon gets launched (reasonably common). If that’s the case you might be able to edit the script for now to accept your OS version. (make a backup of it first, etc)
My question now is why wasn’t the sub ib0.1 interface enslaving setup for me as the user manual says the below:
“The IPoIB daemon (ipoibd) detects the new VIFs and creates a new IPoIB instances, as a result number of IPoIB interfaces (ibX.Y) are shown as being created/destroyed, and are being enslaved to the corresponding ethX interface to serve any active VIF in the system according to the set configuration, This process is done automatically by the ipoibd service.”
or do you just have to manually do the PIF but the VIF’s are auto setup ?
I’m trying too to get eipoib to work on ubuntu 12.04.
In the end, did you succeed to have eipoib working? Can you add your infiniband hca as an interface for a bridge device?
I’m stuck in compiling eth_ipoib kernel module…
Did you start from OFED download right? which version?
Can you please paste here the procedure or better your linux history of commands to download, edit, and compile both the kernel module and the ipoibd daemon?
ipoibd is in the public beta is broken on ubuntu. I needed to hack it in order to make it run. Even when it is properly configured, the daemon immediately exits due to a check if the operating system is supported. Ubuntu is not. If you comment out this guard, the daemon starts, but has a bunch of hardcoded paths to executables that the daemon is calling that are in different locations on ubuntu than on redhat. Once these are fixed, the daemon works properly.
I would also like to get eIPoIB up&running. I have a couple of servers with CentOS 6.4 (2.6.32-358.6.2.el6.x86_64) and ConnectX-3 dual-port adapters. I’ve installed latest OFED-2 (I had to add my kernel support with “./mlnx_add_kernel_support.sh -m . -v”). IPoIB works fine.
Then I followed the instructions in the manual and enabled eIPoIB by adding “E_IPOIB_LOAD=yes” to /etc/infiniband/openib.conf and restarted InfiniBand drivers by /etc/init.d/openibd restart.
Then manual says “When eth_ipoib is loaded,”… ok, how do I know if “eth_ipoib is loaded”? Is this supposed to be a kernel module?
modprobe eth_ipoib
FATAL: Module eth_ipoib not found
Also, OFED didn’t install /etc/init.d/ipoibd, but I found one in /usr/src/ofa_kernel-2.0/ofed_scripts/ipoibd …
Obviously something hasn’t been installed correctly? Do I have to compile&install it manually from /usr/src/ofa_kernel-2.0/drivers/net/eipoib/?
Hi, Iliyasa … have found a solution for building eIPoIB modules and drivers on centos63/centos64? I would really like to get this working so any help would be very much appreciated! Thanks!