Hi All,
I am trying to compile DPDK with Mellanox driver support and test pktgen on Ubuntu 18.04/16.04 on Azure. I have created the VM Ubuntu 18.04/16.04 with two interfaces with accelerated networking enabled. I have compiled DPDK with MLX4/5 enabled successfully followed by PKTGEN with appropriate targets. However when the Mellanox interfaces are bound to mlx4_core using dpdk-devbind.py the interfaces doesn’t show up as DPDK network devices and rather show up as using Network Devices. However when the interfaces are bound to igb_uio driver the interfaces show up as DPDK compatible
network devices.
DPDK version used: dpdk-19.02
PKTGEN: pktgen-3.6.5
After installing all dependencies, DPDK is compiled with the following steps
Make config T=x86_64-native-linuxapp-gcc
To enable MLX driver:
sed -ri ‘s,(MLX._PMD=)n,\1y,’ build/.config.
and
make install T=x86_64-native-linuxapp-gcc DESTDIR=<PKTGEN_DIR>
Please note that I have installed libibverbs and have not installed full version of OFED i.e. MLNX_OFED_LINUX.
Installation in /home/<>/dpdk-19.02// complete
Similarly PKTGEN is also compiled and built with the already compiled DPDK SDK.
What I notice is that these interfaces show up as DPDK compatible
interfaces when bound to igb_uio but when bound to mlx4_uio, they show up as Kernel
Output of dpdk-devbind.py with -b with igb_uio and mlx4_core
When bound to igb_uio
Network devices using DPDK-compatible driver
============================================
0001:00:02.0 ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004’ drv=igb_uio unused=
0002:00:02.0 ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004’ drv=igb_uio unused=
0003:00:02.0 ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004’ drv=igb_uio unused=
When bound to mlx4_core
Network devices using kernel driver
===================================
0001:00:02.0 ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004’ if=rename20 drv=mlx4_core unused=igb_uio
0002:00:02.0 ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004’ if=enP2s2 drv=mlx4_core unused=igb_uio
0003:00:02.0 ‘MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1004’ if=enP3s3 drv=mlx4_core unused=igb_uio
Please note that the devices are using kernel driver and not DPDK MLX driver.
~/dpdk-19.02$ lspci
0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03)
0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01)
0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA
0001:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
0002:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
0003:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
dpdk-19.02$ ifconfig -a
enP1p0s2 Link encap:Ethernet HWaddr 00:0d:3a:f2:08:6b
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:9773 errors:0 dropped:0 overruns:0 frame:0
TX packets:15939 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7116204 (7.1 MB) TX bytes:3611453 (3.6 MB)
enP2p0s2 Link encap:Ethernet HWaddr 00:0d:3a:f2:f5:2c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1690 (1.6 KB)
enP3p0s2 Link encap:Ethernet HWaddr 00:0d:3a:f2:f0:7c
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1690 (1.6 KB)
eth0 Link encap:Ethernet HWaddr 00:0d:3a:f2:08:6b
inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::20d:3aff:fef2:86b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11519 errors:0 dropped:0 overruns:0 frame:0
TX packets:15414 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8502266 (8.5 MB) TX bytes:3579927 (3.5 MB)
eth1 Link encap:Ethernet HWaddr 00:0d:3a:f2:f5:2c
inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0
inet6 addr: fe80::20d:3aff:fef2:f52c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:762 (762.0 B) TX bytes:1690 (1.6 KB)
Do I need to do anything else to get this working? I mean bind Mellanox interfaces
As DPDK compatible interfaces? Packet generation is not working which I think is due
To this issue. In summary while the compilation goes through, DPDK MLX drivers doesn’t
Seem to get loaded and attach to the interfaces.
I have referred to https://doc.dpdk.org/guides/nics/mlx4.html as well while working on compiling DPDK.
Other thing I would like to know is if installing MLNX_OFED_LINUX is a must in this environment. I am facing
Issues while installing OFED drivers but DPDK compilation seems to go through fine with just libibverbs. Please
Let me know if you would like to know info. Any help will be greatly appreciated.