NVMe-oF Host Connection Failed

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi everyone,
I’m facing a strange issue with NVMe-oF that I haven’t been able to fix easily. TLDR with the error: rdma connection establishment failed (-104) and Failed to write to /dev/nvme-fabrics: Connection reset by peer

I’m generally familiar with NVMe-oF setup, following this guide here: EnterpriseSupport My setup is a single node with multiple RDMA NICs and SSDs. With this configuration, I’ve been able to set up an NVMe-oF target on one NIC and let the host connect from another NIC. The setup and all my commands have worked multiple times before.

However, after a reboot today, I was no longer able to let the host connect using the same setup and scripts. The target setup shows no issues, and the host can discover the connection. But the actual connect fails, with error messages appearing both in the logs and in dmesg.

$ nvme discover -t rdma -a 192.168.1.20 -s 4420

Discovery Log Number of Records 2, Generation counter 2
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: unrecognized
treq:    not specified, sq flow control disable supported
portid:  0
trsvcid: 4420
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  192.168.1.20
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000
=====Discovery Log Entry 1======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  0
trsvcid: 4420
subnqn:  zero
traddr:  192.168.1.20
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000

$ bash nvmeof_client.sh zero 192.168.1.20 4420 192.168.1.21 # my own connect script
Failed to write to /dev/nvme-fabrics: Connection reset by peer

$ dmesg
[ 2322.783144] nvmet: adding nsid 10 to subsystem zero
[ 2322.785800] nvmet_rdma: enabling port 0 (192.168.1.20:4420)
[ 2327.757810] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.
[ 2327.758099] nvme nvme10: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.20:4420
[ 2327.758248] nvme nvme10: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 2331.064689] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.
[ 2331.064957] nvme nvme10: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.20:4420
[ 2331.065117] nvme nvme10: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 2334.104665] nvme nvme10: rdma connection establishment failed (-104)

I found [a related issue https://stackoverflow.com/questions/78981318/nvme-discover-failed-failed-fo-write-to-dev-nvme-fabrics-connection-reset-by , but in my case I’ve managed the separate RDMA interfaces independently from my Ethernet addresses, so I don’t think that applies here. For reference, I’ll share the address list (this node actually has 8 NICs):

$ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp226s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5c:ff:35:fb:ae:93 brd ff:ff:ff:ff:ff:ff
3: enp97s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 3c:ec:ef:b4:74:52 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.51/24 brd 10.0.2.255 scope global enp97s0f0
       valid_lft forever preferred_lft forever
    inet6 2001:41b8:830:16e2:3eec:efff:feb4:7452/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fe80::3eec:efff:feb4:7452/64 scope link 
       valid_lft forever preferred_lft forever
4: enp97s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 3c:ec:ef:b4:74:53 brd ff:ff:ff:ff:ff:ff
5: enx060d30559abe: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:0d:30:55:9a:be brd ff:ff:ff:ff:ff:ff
6: enp12s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d1:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 scope global enp12s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:d1ba/64 scope link 
       valid_lft forever preferred_lft forever
7: enp18s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:e4:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.21/24 scope global enp18s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:e422/64 scope link 
       valid_lft forever preferred_lft forever
8: enp75s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d7:aa brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.22/24 scope global enp75s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:d7aa/64 scope link 
       valid_lft forever preferred_lft forever
9: enp84s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d9:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.23/24 scope global enp84s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:d946/64 scope link 
       valid_lft forever preferred_lft forever
10: enp141s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:e4:3a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.24/24 scope global enp141s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:e43a/64 scope link 
       valid_lft forever preferred_lft forever
11: enp148s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:e4:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global enp148s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:e412/64 scope link 
       valid_lft forever preferred_lft forever
12: enp186s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:da:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.26/24 scope global enp186s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:da12/64 scope link 
       valid_lft forever preferred_lft forever
13: enp204s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d9:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.27/24 scope global enp204s0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:fe16:d95e/64 scope link 
       valid_lft forever preferred_lft forever
14: enp225s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:ce:f6:2d:08:06 brd ff:ff:ff:ff:ff:ff
15: enp225s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:ce:f6:2d:08:07 brd ff:ff:ff:ff:ff:ff
16: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:50:6d:46:cd brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:50ff:fe6d:46cd/64 scope link 
       valid_lft forever preferred_lft forever
18: veth953756b@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether b6:f5:f5:21:b7:2b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::b4f5:f5ff:fe21:b72b/64 scope link 
       valid_lft forever preferred_lft forever

I also checked the firewall:

systemctl status ufw
● ufw.service - Uncomplicated firewall
     Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2025-12-15 17:50:45 UTC; 1h 1min ago
       Docs: man:ufw(8)
    Process: 7770 ExecStart=/lib/ufw/ufw-init start quiet (code=exited, status=0/SUCCESS)
   Main PID: 7770 (code=exited, status=0/SUCCESS)
        CPU: 2ms

Dez 15 17:50:45 dgx01.lab.<...>.de systemd[1]: Starting Uncomplicated firewall...
Dez 15 17:50:45 dgx01.lab.<...>.de systemd[1]: Finished Uncomplicated firewall.

$ ufw status verbose
Status: inactive
Status: inactive

I can provide more details about the setup if necessary. Thanks!

2

This is getting even stranger. I have 8 NICs and 8 SSDs, and I was previously able to set up 8 NVMe‑oF connections. The setup is essentially a loopback configuration: NVMe‑oF connections between two NICs on the same node.

Right now, I can only bring up NVMe‑oF between two NICs; the other six all fail. It’s very odd because the drivers should be working, but it looks like some SSDs or NICs may be behaving incorrectly.

Below are the dmesg logs for both the failed setups and the successful ones:

## failed

[Fr Dez 19 13:06:57 2025] nvme nvme10: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.24:4424
[Fr Dez 19 13:06:57 2025] nvme nvme10: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fr Dez 19 13:07:00 2025] nvme nvme10: rdma connection establishment failed (-104)
[Fr Dez 19 13:08:03 2025] nvmet: adding nsid 15 to subsystem five
[Fr Dez 19 13:08:03 2025] nvmet_rdma: enabling port 5 (192.168.1.25:4425)
[Fr Dez 19 13:08:08 2025] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.


## success
[Fr Dez 19 13:10:13 2025] nvme nvme10: rdma connection establishment failed (-512)
[Fr Dez 19 13:10:14 2025] nvmet: adding nsid 16 to subsystem seven
[Fr Dez 19 13:10:14 2025] nvmet_rdma: enabling port 6 (192.168.1.26:4426)
[Fr Dez 19 13:10:18 2025] nvmet: creating discovery controller 1 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.
[Fr Dez 19 13:10:18 2025] nvme nvme10: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.26:4426
[Fr Dez 19 13:10:18 2025] nvme nvme10: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fr Dez 19 13:10:19 2025] nvmet: creating nvm controller 1 for subsystem seven for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.
[Fr Dez 19 13:10:19 2025] nvme nvme10: creating 63 I/O queues.
[Fr Dez 19 13:10:19 2025] mlx5_ib: mlx5_set_umr_free_mkey: Translation mode supported only when access_mode is MTT or PA
[Fr Dez 19 13:10:32 2025] nvme nvme10: mapped 63/0/0 default/read/poll queues.
[Fr Dez 19 13:10:32 2025] nvme nvme10: new ctrl: NQN "seven", addr 192.168.1.26:4426
[Fr Dez 19 13:11:05 2025] nvmet: adding nsid 17 to subsystem eight
[Fr Dez 19 13:11:05 2025] nvmet_rdma: enabling port 7 (192.168.1.27:4427)
[Fr Dez 19 13:11:10 2025] nvmet: creating discovery controller 2 for subsystem nqn.2014-08.org.nvmexpress.discovery for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.
[Fr Dez 19 13:11:10 2025] nvme nvme11: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.1.27:4427
[Fr Dez 19 13:11:10 2025] nvme nvme11: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[Fr Dez 19 13:11:10 2025] nvmet: creating nvm controller 2 for subsystem eight for NQN nqn.2014-08.org.nvmexpress:uuid:c6028c49-2d50-4692-8cbd-2679629e5a0a.
[Fr Dez 19 13:11:10 2025] nvme nvme11: creating 63 I/O queues.
[Fr Dez 19 13:11:23 2025] nvme nvme11: mapped 63/0/0 default/read/poll queues.
[Fr Dez 19 13:11:23 2025] nvme nvme11: new ctrl: NQN "eight", addr 192.168.1.27:4427
3

since you are seeing a “rdma connection establishment failed (-104)”

# 1. Check RDMA device to network interface mapping

rdma link show

# 2. Check resource usage

rdma resource show

# 3. Check for device errors

dmesg | grep -i “mlx5\|rdma\|infiniband” | grep -i “error\|fail” | tail -30

# 4. Check which RDMA device corresponds to which IP

for dev in enp12s0np0 enp18s0np0 enp75s0np0 enp84s0np0 enp141s0np0 enp148s0np0 enp186s0np0 enp204s0np0; do

echo “=== $dev ===”

ethtool -i $dev | grep “bus-info”

ls -l /sys/class/net/$dev/device/infiniband* 2>/dev/null || echo “No RDMA device found”

done

$ lsmod | grep rdma

$ uname -r

4

Hi @kmodukuri , thanks again for always helping out. I got the command, and I’ll append the output later.

The issue seems more complex than expected. I can run RDMA over the NICs, and I can set up NVMe‑oF on 2 NICs, so the driver itself appears to be working. For the IP configuration, we set up all 8 NICs in the same way.

I still need help from your side, because I can’t figure out why some NVMe‑oF setups work while others fail at the exact same time.

There are the output:

$ rdma link show
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev enp12s0np0 
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev enp18s0np0 
link mlx5_2/1 state ACTIVE physical_state LINK_UP netdev enp75s0np0 
link mlx5_3/1 state ACTIVE physical_state LINK_UP netdev enp84s0np0 
link mlx5_4/1 state ACTIVE physical_state LINK_UP netdev enp141s0np0 
link mlx5_5/1 state ACTIVE physical_state LINK_UP netdev enp148s0np0 
link mlx5_6/1 state ACTIVE physical_state LINK_UP netdev enp186s0np0 
link mlx5_7/1 state ACTIVE physical_state LINK_UP netdev enp204s0np0 
link mlx5_8/1 state DOWN physical_state DISABLED netdev enp225s0f0np0 
link mlx5_9/1 state DOWN physical_state DISABLED netdev enp225s0f1np1 

$ rdma resource show
0: mlx5_0: pd 3 cq 130 qp 1 cm_id 0 mr 0 ctx 0 srq 2 
1: mlx5_1: pd 3 cq 130 qp 1 cm_id 0 mr 0 ctx 0 srq 2 
2: mlx5_2: pd 3 cq 130 qp 1 cm_id 1 mr 0 ctx 0 srq 2 
3: mlx5_3: pd 3 cq 130 qp 1 cm_id 1 mr 0 ctx 0 srq 2 
4: mlx5_4: pd 3 cq 130 qp 1 cm_id 1 mr 0 ctx 0 srq 2 
5: mlx5_5: pd 3 cq 130 qp 1 cm_id 1 mr 0 ctx 0 srq 2 
6: mlx5_6: pd 5 cq 130 qp 129 cm_id 129 mr 24256 ctx 0 srq 2 
7: mlx5_7: pd 5 cq 130 qp 129 cm_id 129 mr 24256 ctx 0 srq 2 
8: mlx5_8: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 0 srq 2 
9: mlx5_9: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 0 srq 2 

$ sudo dmesg | grep -i “mlx5\|rdma\|infiniband” | grep -i “error\|fail” | tail -30
$ EMPTY



$ bash forum_20Dec.sh 
=== enp12s0np0 ===
bus-info: 0000:0c:00.0
/sys/class/net/enp12s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_0

/sys/class/net/enp12s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm0
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad0

/sys/class/net/enp12s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs0
=== enp18s0np0 ===
bus-info: 0000:12:00.0
/sys/class/net/enp18s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_1

/sys/class/net/enp18s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm1
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad1

/sys/class/net/enp18s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs1
=== enp75s0np0 ===
bus-info: 0000:4b:00.0
/sys/class/net/enp75s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_2

/sys/class/net/enp75s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm2
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad2

/sys/class/net/enp75s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs2
=== enp84s0np0 ===
bus-info: 0000:54:00.0
/sys/class/net/enp84s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_3

/sys/class/net/enp84s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm3
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad3

/sys/class/net/enp84s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs3
=== enp141s0np0 ===
bus-info: 0000:8d:00.0
/sys/class/net/enp141s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_4

/sys/class/net/enp141s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm4
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad4

/sys/class/net/enp141s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs4
=== enp148s0np0 ===
bus-info: 0000:94:00.0
/sys/class/net/enp148s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_5

/sys/class/net/enp148s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm5
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad5

/sys/class/net/enp148s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs5
=== enp186s0np0 ===
bus-info: 0000:ba:00.0
/sys/class/net/enp186s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_6

/sys/class/net/enp186s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm6
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad6

/sys/class/net/enp186s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs6
=== enp204s0np0 ===
bus-info: 0000:cc:00.0
/sys/class/net/enp204s0np0/device/infiniband:
total 0
drwxr-xr-x 8 root root 0 Dez 15 17:50 mlx5_7

/sys/class/net/enp204s0np0/device/infiniband_mad:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 issm7
drwxr-xr-x 3 root root 0 Dez 15 17:50 umad7

/sys/class/net/enp204s0np0/device/infiniband_verbs:
total 0
drwxr-xr-x 3 root root 0 Dez 15 17:50 uverbs7

$ lsmod | grep rdma
nvme_rdma              45056  2
nvmet_rdma             57344  6
nvmet                 151552  37 nvmet_rdma
nvme_fabrics           32768  1 nvme_rdma
rdma_ucm               28672  0
rdma_cm               122880  3 nvme_rdma,nvmet_rdma,rdma_ucm
iw_cm                  49152  1 rdma_cm
ib_cm                 131072  2 rdma_cm,ib_ipoib
ib_uverbs             135168  2 rdma_ucm,mlx5_ib
ib_core               434176  11 rdma_cm,ib_ipoib,nvidia_peermem,nvme_rdma,nvmet_rdma,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvme                   57344  19 nvmet,nvmet_rdma
nvme_core             143360  23 nvmet,nvme,nvme_rdma,nvme_fabrics
mlx_compat             69632  17 rdma_cm,ib_ipoib,mlxdevm,nvmet,nvme,nvme_rdma,nvmet_rdma,iw_cm,nvme_core,nvme_fabrics,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

$ uname -r
5.15.0-1091-nvidia
5

ok looks like RDMA is okay. maybe something on the NVMeOF protocol or config issue

Enable RDMA and NVMe debug logs

echo ‘module nvme_rdma +p’ > /sys/kernel/debug/dynamic_debug/control

echo ‘module nvmet_rdma +p’ > /sys/kernel/debug/dynamic_debug/control

echo ‘module rdma_cm +p’ > /sys/kernel/debug/dynamic_debug/control

Monitor Dmesg while connecting

dmesg -w

6

Thanks again for all the help I’ve received. I still think we need to continue working on this together, but first, I want to wish you a happy holiday!

I still find the situation strange because I’m seeing **different behaviors** (see the output below).

  • The setup hasn’t changed: I already have two NVMe‑oF connections working (nvme10 and nvme11), and I’m trying to continue monitoring the failures from last time. I haven’t rebooted the system since my previous post. The target is configured, and I’m simply reconnecting using my script.

Here are the different cases (address error and hanging) I’m seeing:

$ bash nvmeof_client.sh two 192.168.1.22 4422 192.168.1.23 # 192.168.1.22 is target, 192.168.1.23 is the host trying to connect the target

Failed to write to /dev/nvme-fabrics: Connection reset by peer
Failed to connect to target!




# in Dmesg
[Mo Dez 22 19:08:45 2025] RDMA CM: ADDR_ERROR: failed to resolve IP. status -110
[Mo Dez 22 19:08:45 2025] nvme nvme12: address error (1): status -110 id 00000000eab8bbd7
[Mo Dez 22 19:08:45 2025] nvme nvme12: CM error event 1
[Mo Dez 22 19:08:45 2025] nvme nvme12: rdma connection establishment failed (-104)

$ bash nvmeof_client.sh one 192.168.1.21 4421 192.168.1.20 # 192.168.1.21 is target, 192.168.1.20 is the host trying to connect the target

# THIS COMMASND HANGS!




# in Dmesg
[Mo Dez 22 19:09:33 2025] nvme nvme12: address resolved (0): status 0 id 00000000f1635065
[Mo Dez 22 19:09:33 2025] nvme nvme12: route resolved  (2): status 0 id 00000000f1635065

I have a similar setup on the other NICs. Some of them fail with an address error, and others just hang.

Then there are some IP setups, and I think the address should work:

$ ibstatus
Infiniband device 'mlx5_0' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:d1ba
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_1' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:e422
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_2' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:d7aa
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_3' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:d946
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_4' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:e43a
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_5' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:e412
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_6' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:da12
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_7' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe16:d95e
	base lid:	0x0
	sm lid:		0x0
	state:		4: ACTIVE
	phys state:	5: LinkUp
	rate:		200 Gb/sec (4X HDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_8' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe2d:0806
	base lid:	0x0
	sm lid:		0x0
	state:		1: DOWN
	phys state:	3: Disabled
	rate:		40 Gb/sec (4X QDR)
	link_layer:	Ethernet

Infiniband device 'mlx5_9' port 1 status:
	default gid:	fe80:0000:0000:0000:bace:f6ff:fe2d:0807
	base lid:	0x0
	sm lid:		0x0
	state:		1: DOWN
	phys state:	3: Disabled
	rate:		40 Gb/sec (4X QDR)
	link_layer:	Ethernet


$ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp226s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5c:ff:35:fb:ae:93 brd ff:ff:ff:ff:ff:ff
3: enp97s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 3c:ec:ef:b4:74:52 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.51/24 brd 10.0.2.255 scope global enp97s0f0
       valid_lft forever preferred_lft forever
    inet6 2001:41b8:830:16e2:3eec:efff:feb4:7452/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 86395sec preferred_lft 14395sec
    inet6 fe80::3eec:efff:feb4:7452/64 scope link 
       valid_lft forever preferred_lft forever
4: enp97s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 3c:ec:ef:b4:74:53 brd ff:ff:ff:ff:ff:ff
5: enx060d30559abe: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:0d:30:55:9a:be brd ff:ff:ff:ff:ff:ff
6: enp12s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d1:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 scope global enp12s0np0
       valid_lft forever preferred_lft forever
7: enp18s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:e4:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.21/24 scope global enp18s0np0
       valid_lft forever preferred_lft forever
8: enp75s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d7:aa brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.22/24 scope global enp75s0np0
       valid_lft forever preferred_lft forever
9: enp84s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d9:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.23/24 scope global enp84s0np0
       valid_lft forever preferred_lft forever
10: enp141s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:e4:3a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.24/24 scope global enp141s0np0
       valid_lft forever preferred_lft forever
11: enp148s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:e4:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.25/24 scope global enp148s0np0
       valid_lft forever preferred_lft forever
12: enp186s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:da:12 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.26/24 scope global enp186s0np0
       valid_lft forever preferred_lft forever
13: enp204s0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:16:d9:5e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.27/24 scope global enp204s0np0
       valid_lft forever preferred_lft forever
14: enp225s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:ce:f6:2d:08:06 brd ff:ff:ff:ff:ff:ff
15: enp225s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b8:ce:f6:2d:08:07 brd ff:ff:ff:ff:ff:ff
16: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:50:6d:46:cd brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:50ff:fe6d:46cd/64 scope link 
       valid_lft forever preferred_lft forever
18: veth953756b@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether b6:f5:f5:21:b7:2b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::b4f5:f5ff:fe21:b72b/64 scope link 
       valid_lft forever preferred_lft forever

What I did:

echo 'module nvme_rdma +p' > /sys/kernel/debug/dynamic_debug/control
echo 'module nvmet_rdma +p' > /sys/kernel/debug/dynamic_debug/control
echo 'module rdma_cm +p' > /sys/kernel/debug/dynamic_debug/control
7

Got some help from RAPIDS. We think the issue is that the RDMA connection between some of the NICs is faulty, which prevents NVMe‑oF from being set up on those interfaces. I still have two NICs that work correctly, which is why I can set up NVMe‑oF on them.

# Bad NIC

$ ib_write_bw -F -d mlx5_0 -a

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_0
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 4
 Max inline data : 0[B]
 rdma_cm QPs	: OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0074 PSN 0xad2658 RKey 0x203d00 VAddr 0x007f58edba8000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:20
ethernet_read_keys: Couldn't read remote address
 Unable to read to socket/rdma_cm
 Failed to exchange data between server and clients


$ ib_write_bw -d mlx5_1 192.168.1.20
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_1
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	: OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
Failed to modify QP 84 to RTR
 Unable to Connect the HCA's through the link

# Good NICs where the NVMeoF comes from

$ ib_write_bw -F -d mlx5_6 -a

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_6
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	: OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0204 PSN 0xcfbf4c RKey 0x208c00 VAddr 0x007f253277c000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:26
 remote address: LID 0000 QPN 0x0202 PSN 0xb22e74 RKey 0x20fa00 VAddr 0x007fe306732000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:27
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      5000             13634.99            13221.16		  0.211539
---------------------------------------------------------------------------------------


$ ib_write_bw -d mlx5_7 192.168.1.26
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : mlx5_7
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 1
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 3
 Max inline data : 0[B]
 rdma_cm QPs	: OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0202 PSN 0xb22e74 RKey 0x20fa00 VAddr 0x007fe306732000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:27
 remote address: LID 0000 QPN 0x0204 PSN 0xcfbf4c RKey 0x208c00 VAddr 0x007f253277c000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:01:26
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 2942.585000 != 2250.000000. CPU Frequency is not max.
 65536      5000             13634.99            13221.16		  0.211539
---------------------------------------------------------------------------------------

Here are some system details:

  • The OS is DGX Server Version 6.3.2 (GNU/Linux 5.15.0‑1091‑nvidia x86_64).
  • $ nvidia-smi --version
NVIDIA-SMI version  : 580.95.05
NVML version        : 580.95
DRIVER version      : 580.95.05
CUDA Version        : 13.0
8

The issue might simply be that I was a bit careless. But I believe RDMA was working for a while. I’ll continue working on the fix (next year).