Why are no RDMA devices returned by ibv_get_device_list?

Infrastructure & Networking Software And Drivers Mellanox OFED
1

I have a box with a couple of ConnectX-4 Lx cards running FreeBSD 12. I have downloaded http://content.mellanox.com/Drivers/freebsd_mlx5en_v3_5_0.tar.gz, built and installed the kernel and userspace components (using “./build.sh -5 -I -i -V -t”), and rebooted. ibstat shows the presence of my devices but a simple test program that calls ibv_get_device_list receives an empty list. What have I missed?

r640fbsd05 m/ 0 309$ kldstat

Id Refs Address Size Name

1 9 0xffffffff80200000 243cd00 kernel

2 1 0xffffffff8263e000 32980 mlx5ib.ko

3 2 0xffffffff82671000 9dd68 ibcore.ko

4 5 0xffffffff8270f000 261c0 linuxkpi.ko

5 3 0xffffffff82736000 65080 mlx5.ko

6 1 0xffffffff82b12000 138a0 mlx5en.ko

r640fbsd05 m/ 0 311$ pciconf -lv | grep -A4 mlx

mlx5_core0@pci0:25:0:0: class=0x020000 card=0x002515b3 chip=0x101515b3 rev=0x00 hdr=0x00

vendor = ‘Mellanox Technologies’

device = ‘MT27710 Family [ConnectX-4 Lx]’

class = network

subclass = ethernet

mlx5_core1@pci0:25:0:1: class=0x020000 card=0x002515b3 chip=0x101515b3 rev=0x00 hdr=0x00

vendor = ‘Mellanox Technologies’

device = ‘MT27710 Family [ConnectX-4 Lx]’

class = network

subclass = ethernet

–

mlx5_core2@pci0:94:0:0: class=0x020000 card=0x000115b3 chip=0x101715b3 rev=0x00 hdr=0x00

vendor = ‘Mellanox Technologies’

device = ‘MT27800 Family [ConnectX-5]’

class = network

subclass = ethernet

mlx5_core3@pci0:94:0:1: class=0x020000 card=0x000115b3 chip=0x101715b3 rev=0x00 hdr=0x00

vendor = ‘Mellanox Technologies’

device = ‘MT27800 Family [ConnectX-5]’

class = network

subclass = ethernet

r640fbsd05 m/ 0 314$

r640fbsd05 m/ 0 314$ sysctl sys.class.infiniband.mlx5_0

sys.class.infiniband.mlx5_0.cong.stats.np_cnp_sent: 0

sys.class.infiniband.mlx5_0.cong.stats.np_ecn_marked_roce_packets: 0

sys.class.infiniband.mlx5_0.cong.stats.accumulators_period: 1000000

sys.class.infiniband.mlx5_0.cong.stats.time_stamp: 3335618974

sys.class.infiniband.mlx5_0.cong.stats.rp_cnp_handled: 0

sys.class.infiniband.mlx5_0.cong.stats.rp_cnp_ignored: 0

sys.class.infiniband.mlx5_0.cong.stats.sum_flows: 0

sys.class.infiniband.mlx5_0.cong.stats.rp_cur_flows: 0

sys.class.infiniband.mlx5_0.cong.stats.syndrome: 0

sys.class.infiniband.mlx5_0.cong.conf.np_cnp_prio: 0

sys.class.infiniband.mlx5_0.cong.conf.np_cnp_prio_mode: 0

sys.class.infiniband.mlx5_0.cong.conf.np_cnp_dscp: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_gd: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_initial_alpha_value: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_rate_reduce_monitor_period: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_dce_tcp_rtt: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_dce_tcp_g: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_rate_to_set_on_first_cnp: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_min_rate: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_min_dec_fac: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_hai_rate: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_ai_rate: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_threshold: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_byte_reset: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_time_reset: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_clamp_tgt_rate_ati: 0

sys.class.infiniband.mlx5_0.cong.conf.rp_clamp_tgt_rate: 0

sys.class.infiniband.mlx5_0.reg_pages: 0

sys.class.infiniband.mlx5_0.fw_pages: 34645

sys.class.infiniband.mlx5_0.board_id: DEL2810000034

sys.class.infiniband.mlx5_0.hca_type: MT4117

sys.class.infiniband.mlx5_0.hw_rev: 0

sys.class.infiniband.mlx5_0.default_roce_mode_port1: RoCE v2

sys.class.infiniband.mlx5_0.ports.1.hw_counters.lifespan: 10

sys.class.infiniband.mlx5_0.ports.1.hw_counters.local_ack_timeout_err: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.implied_nak_seq_err: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.packet_seq_err: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.rnr_nak_retry_err: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.duplicate_request: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.out_of_sequence: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.out_of_buffer: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.rx_atomic_requests: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.rx_read_requests: 0

sys.class.infiniband.mlx5_0.ports.1.hw_counters.rx_write_requests: 0

sys.class.infiniband.mlx5_0.ports.1.pkeys.0: 0xffff

sys.class.infiniband.mlx5_0.ports.1.gids.255: 0000:0000:0000:0000:0000:0000:0000:0000

…

sys.class.infiniband.mlx5_0.ports.1.gids.4: 0000:0000:0000:0000:0000:0000:0000:0000

sys.class.infiniband.mlx5_0.ports.1.gids.3: 0000:0000:0000:0000:0000:ffff:0a01:56e2

sys.class.infiniband.mlx5_0.ports.1.gids.2: 0000:0000:0000:0000:0000:ffff:0a01:56e2

sys.class.infiniband.mlx5_0.ports.1.gids.1: fe80:0000:0000:0000:526b:4bff:feb1:2ec6

sys.class.infiniband.mlx5_0.ports.1.gids.0: fe80:0000:0000:0000:526b:4bff:feb1:2ec6

sys.class.infiniband.mlx5_0.ports.1.counters.multicast_xmit_packets: 0

sys.class.infiniband.mlx5_0.ports.1.counters.multicast_rcv_packets: 0

sys.class.infiniband.mlx5_0.ports.1.counters.unicast_xmit_packets: 0

sys.class.infiniband.mlx5_0.ports.1.counters.unicast_rcv_packets: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_rcv_packets: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_xmit_wait: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_xmit_packets: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_rcv_data: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_xmit_data: 0

sys.class.infiniband.mlx5_0.ports.1.counters.VL15_dropped: 0

sys.class.infiniband.mlx5_0.ports.1.counters.excessive_buffer_overrun_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.local_link_integrity_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_rcv_constraint_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_xmit_constraint_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_xmit_discards: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_rcv_switch_relay_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_rcv_remote_physical_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.port_rcv_errors: 0

sys.class.infiniband.mlx5_0.ports.1.counters.link_downed: 0

sys.class.infiniband.mlx5_0.ports.1.counters.link_error_recovery: 0

sys.class.infiniband.mlx5_0.ports.1.counters.symbol_error: 0

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.types.255: IB/RoCE v1

…

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.types.3: RoCE v2

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.types.2: IB/RoCE v1

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.types.1: RoCE v2

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.types.0: IB/RoCE v1

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.ndevs.3: mce0

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.ndevs.2: mce0

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.ndevs.1: mce0

sys.class.infiniband.mlx5_0.ports.1.gid_attrs.ndevs.0: mce0

sys.class.infiniband.mlx5_0.ports.1.link_layer: Ethernet

sys.class.infiniband.mlx5_0.ports.1.phys_state: 5: LinkUp

sys.class.infiniband.mlx5_0.ports.1.rate: 10 Gb/sec (1X QDR)

sys.class.infiniband.mlx5_0.ports.1.cap_mask: 0x04010000

sys.class.infiniband.mlx5_0.ports.1.sm_sl: 0

sys.class.infiniband.mlx5_0.ports.1.sm_lid: 0x0

sys.class.infiniband.mlx5_0.ports.1.lid_mask_count: 0

sys.class.infiniband.mlx5_0.ports.1.lid: 0x0

sys.class.infiniband.mlx5_0.ports.1.state: 4: ACTIVE

sys.class.infiniband.mlx5_0.fw_ver: 14.21.3012

sys.class.infiniband.mlx5_0.node_desc: MT4117 ConnectX4 Mellanox Technologies

sys.class.infiniband.mlx5_0.node_guid: 506b:4b03:00b1:2ec6

sys.class.infiniband.mlx5_0.sys_image_guid: 506b:4b03:00b1:2ec6

sys.class.infiniband.mlx5_0.node_type: 1: CA

r640fbsd05 m/ 0 315$ cat x.c

#include <infiniband/verbs.h>

#include <stdio.h>

int main() {

int c;

ibv_get_device_list(&c);

printf(“ibv_get_device_list returns %d devices\n”, c);

}

r640fbsd05 m/ 0 323$ ibstat

CA ‘mlx5_0’

CA type: MT4117

Number of ports: 1

Firmware version: 14.21.3012

Hardware version: 0

Node GUID: 0x506b4b0300b12ec6

System image GUID: 0x506b4b0300b12ec6

Port 1:

State: Active

Physical state: LinkUp

Rate: 10

Base lid: 0

LMC: 0

SM lid: 0

Capability mask: 0x04010000

Port GUID: 0x526b4bfffeb12ec6

Link layer: Ethernet

…

2

Err pardon. Obviously, not two ConnectX-4 Lx cards but one of those and one ConnectX5.

3

I omitted a critical piece of information: The command used to build my test program. It was

cc -o x x.c -libverbs

This will always result in no devices being returned.

First, libibverbs uses pthread_once to run its initialization code. I do not know what magic FreeBSD does with pthread calls, but failing to link with libpthreads will not result in a linking error. The call will fail at runtime. Unfortunately, libibverbs does not check its return value; therefore, the failure is silent.

Second, libibverbs expects userspace driver libraries to register themselves with it, which they do in their init functions. If no drivers are found, no devices will be returned because there must be a driver suitable for any device to be used. Therefore, failing to link with libmlx5 (or some other appropriate driver library) will also cause this kind of failure.

Furthermore, statically linking the library may cause problems. This can be detected by setting IBV_SHOW_WARNINGS=1 in the environment prior to calling ibv_get_device_list.

And finally, beware: On FreeBSD 12, libibverbs and libmlx5 were built and installed from the FBSD source tree into /lib/ but the tarball from Mellanox’s website installs to /usr/lib/. If you build it yourself, you may need to alter the install location or move the original copy out of the way.