Ping between BlueField-3 DPU and host does not work (communication via tmfifo_net0 works fine)

Infrastructure & Networking BlueField
1

Hello,

I’m experiencing an issue with the network connection between a BlueField-3 DPU and its host system, and I would appreciate your advice or suggestions.

Issue Summary

When I try to ping between the host and the DPU using their data network interfaces, there is no response.
However, communication through the control interface (tmfifo_net0) works correctly.

System Overview

  • The host and DPU are connected within the same server via PCIe.

  • The ping test is performed from the host’s enp179s0f0np0 (IP: 172.16.0.2)
    to the DPU’s enp3s0f0s0 (IP: 172.16.0.6).

Host Configuration

The interface status and IP configuration are as follows:

3: enp179s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a0:88:c2:36:4e:98 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.2/28 brd 172.16.0.15 scope global enp179s0f0np0

Routing table:

$ ip route show
172.16.0.0/28 dev enp179s0f0np0 proto kernel scope link src 172.16.0.2

BlueField operating mode on host

sudo cat /dev/rshim0/misc

BF_MODE         DPU mode
DEV_INFO        BlueField-3(Rev 1)

→ The host detects the BlueField-3 as being in DPU mode.

DPU (BlueField-3) Configuration

Below are the outputs of ip link show and ip a on the DPU:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: oob_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether a0:88:c2:36:4e:bc brd ff:ff:ff:ff:ff:ff
3: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:1a:ca:ff:ff:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.6.6/30 brd 192.168.6.7 scope global noprefixroute tmfifo_net0
       valid_lft forever preferred_lft forever
4: p0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether a0:88:c2:36:4e:a8 brd ff:ff:ff:ff:ff:ff
5: p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether a0:88:c2:36:4e:a9 brd ff:ff:ff:ff:ff:ff
6: ovs-system: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether be:8f:d0:59:51:1b brd ff:ff:ff:ff:ff:ff
7: ovsbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether a0:88:c2:36:4e:a8 brd ff:ff:ff:ff:ff:ff
8: ovsbr2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a0:88:c2:36:4e:a9 brd ff:ff:ff:ff:ff:ff
9: pf0hpf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 128
    link/ether ae:16:07:16:ed:45 brd ff:ff:ff:ff:ff:ff
10: pf1hpf: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 128
    link/ether 7e:d1:56:16:4a:e6 brd ff:ff:ff:ff:ff:ff
11: en3f0pf0sf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 128
    link/ether c2:70:01:d4:82:bf brd ff:ff:ff:ff:ff:ff
12: enp3s0f0s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:ca:cb:ac:21:12 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.6/28 brd 172.16.0.15 scope global enp3s0f0s0
       valid_lft forever preferred_lft forever
13: en3f1pf1sf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 128
    link/ether da:2f:af:3e:77:46 brd ff:ff:ff:ff:ff:ff
14: enp3s0f1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:fa:70:ff:35:5d brd ff:ff:ff:ff:ff:ff

Routing table on the DPU:

$ ip route show
172.16.0.0/28 dev enp3s0f0s0 proto kernel scope link src 172.16.0.6

OVS Configuration

sudo ovs-vsctl show
7f3236ee-04dd-43c7-b69c-8a42d3cc4154
    Bridge ovsbr1
        Port en3f0pf0sf0
            Interface en3f0pf0sf0
        Port pf0hpf
            Interface pf0hpf
        Port p0
            Interface p0
        Port ovsbr1
            Interface ovsbr1
                type: internal
    Bridge ovsbr2
        Port p1
            Interface p1
        Port en3f1pf1sf0
            Interface en3f1pf1sf0
        Port ovsbr2
            Interface ovsbr2
                type: internal
        Port pf1hpf
            Interface pf1hpf
    ovs_version: "3.1.0057"

DPU Operating Mode (ECPF Mode) Verification

I also checked whether the DPU is operating in ECPF Mode, and confirmed it as follows:

mst status

BlueField3 ... net-en3f0pf0sf0, net-p0, net-pf0hpf

mlxconfig (CPU-related fields)

HOST_PRIV_NV_INTERNAL_CPU                   DEVICE_DEFAULT(0) 
INTERNAL_CPU_ESWITCH_MANAGER                ECPF(0)             
INTERNAL_CPU_IB_VPORT0                      ECPF(0)             
INTERNAL_CPU_MODEL                          EMBEDDED_CPU(1)     
INTERNAL_CPU_OFFLOAD_ENGINE                 ENABLED(0)          
INTERNAL_CPU_PAGE_SUPPLIER                  ECPF(0)             
INTERNAL_CPU_RSHIM                          ENABLED(0)          
INT_CPU_AUTO_SHUTDOWN                       DEVICE_DEFAULT(0)

→ DPU is in ECPF Mode, and the host recognizes the device in DPU Mode, which appears consistent with BlueField architecture.

What Has Been Verified

  • Reinstalled the BFB image (BlueField OS).

  • Reset mlxconfig to default settings.

  • IP configuration and OVS bridge setup appear correct.

  • When sending ARP requests from the host (enp179s0f0np0), no ARP packets are observed on any interface on the DPU side.
    In particular, I confirmed that even the representor interface pf0hpf does not receive any ARP frames (verified using tcpdump on all DPU interfaces, including pf0hpf, en3f0pf0sf0, p0, p1, enp3s0f0s0, etc.).

  • Communication over the management channel (tmfifo_net0, 192.168.6.6/30) works fine.

Environment Information

  • DPU: BlueField-3

  • DOCA Version: 3.1.0

  • Host OS / DPU OS: Ubuntu 22.04

  • Connection Type: PCIe (within the same server)

  • Issue: Ping between host ↔ DPU (data port) fails

  • Control Port (tmfifo_net0): Working correctly

Question

Even though the link status, IP configuration, routing, and OVS bridge setup all appear correct,
the host’s ARP requests do not reach the DPU (pf0hpf), and ICMP ping also fails.

Could this issue be related to OVS configuration, DOCA 3.1.0 behavior, or PCIe connection-specific restrictions?
Are there any known limitations or additional configurations required for host-to-DPU communication in this setup?

Any guidance or troubleshooting suggestions would be greatly appreciated.

Thank you very much for your time and support.

Additional Investigation: Representor Devices

Since this problem might be related to representor device mapping or configuration,
I have checked both sides using devlink and ethtool.

Host-side output

y-jinbo@1gpu:~$ sudo devlink dev show
pci/0000:b3:00.0
pci/0000:b3:00.1

y-jinbo@1gpu:~$ sudo devlink port show
pci/0000:b3:00.0/65535: type eth netdev enp179s0f0np0 flavour physical port 0 splittable false
  function:
    hw_addr 00:00:00:00:00:00
pci/0000:b3:00.1/131071: type eth netdev enp179s0f1np1 flavour physical port 1 splittable false
  function:
    hw_addr 00:00:00:00:00:00

y-jinbo@1gpu:~$ sudo ethtool -i enp179s0f0np0
driver: mlx5_core
version: 25.07-0.9.7
firmware-version: 32.46.1006 (MT_0000000884)
bus-info: 0000:b3:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

DPU-side output

ubuntu@localhost:~$ sudo devlink dev show
pci/0000:03:00.0
pci/0000:03:00.1
auxiliary/mlx5_core.sf.2
auxiliary/mlx5_core.sf.3

ubuntu@localhost:~$ sudo devlink port show
pci/0000:03:00.0/262143: type eth netdev p0 flavour physical port 0 splittable false
  function:
    hw_addr 00:00:00:00:00:00
pci/0000:03:00.0/196608: type eth netdev pf0hpf flavour pcipf controller 1 pfnum 0 external true splittable false
  function:
    hw_addr a0:88:c2:36:4e:98
pci/0000:03:00.0/229408: type eth netdev en3f0pf0sf0 flavour pcisf controller 0 pfnum 0 sfnum 0 splittable false
  function:
    hw_addr 02:ca:cb:ac:21:12 state active opstate attached
pci/0000:03:00.1/327679: type eth netdev p1 flavour physical port 1 splittable false
  function:
    hw_addr 00:00:00:00:00:00
pci/0000:03:00.1/262144: type eth netdev pf1hpf flavour pcipf controller 1 pfnum 1 external true splittable false
  function:
    hw_addr a0:88:c2:36:4e:99
pci/0000:03:00.1/294944: type eth netdev en3f1pf1sf0 flavour pcisf controller 0 pfnum 1 sfnum 0 splittable false
  function:
    hw_addr 02:fa:70:ff:35:5d state active opstate attached
auxiliary/mlx5_core.sf.2/6619136: type eth netdev enp3s0f0s0 flavour virtual splittable false
auxiliary/mlx5_core.sf.3/8716288: type eth netdev enp3s0f1s0 flavour virtual splittable false

ubuntu@localhost:~$ sudo ethtool -i enp3s0f0s0
driver: mlx5_core
version: 25.07-0.9.7
firmware-version: 32.46.1006 (MT_0000000884)
bus-info: mlx5_core.sf.2

ubuntu@localhost:~$ sudo ethtool -i pf0hpf
driver: mlx5e_rep
version: 5.15.0-1074-bluefield
firmware-version: 32.46.1006 (MT_0000000884)

ubuntu@localhost:~$ sudo ethtool -i en3f0pf0sf0
driver: mlx5e_rep
version: 5.15.0-1074-bluefield
firmware-version: 32.46.1006 (MT_0000000884)
2

Hi, I’m the original poster of this thread.
I’ve done some additional debugging and would like to share the results and ask a more specific question about the eSwitch / PF representor path.

1. Host PF ↔ PF representor mapping is correct

On the host, my data interface is:

$ ip -4 addr show dev enp179s0f0np0
3: enp179s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
    link/ether a0:88:c2:36:4e:98 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.2/28 ...

On the DPU, devlink shows:

$ sudo devlink port show
pci/0000:03:00.0/196608: type eth netdev pf0hpf flavour pcipf controller 1 pfnum 0 external true splittable false
  function:
    hw_addr a0:88:c2:36:4e:98

So the MAC address of the host PF and the PF representor pf0hpf match exactly, which makes me confident that:

Host PF → pf0hpf mapping is correct.

2. Host ARP definitely goes out, but never appears on pf0hpf

On the host, when I ping the DPU data IP (172.16.0.6), I see ARP going out:

$ sudo tcpdump -ni enp179s0f0np0 arp or icmp
ARP, Request who-has 172.16.0.6 tell 172.16.0.2
ARP, Request who-has 172.16.0.6 tell 172.16.0.2
ARP, Request who-has 172.16.0.6 tell 172.16.0.2

However, on the DPU side:

$ sudo tcpdump -ni pf0hpf arp or icmp
$ sudo tcpdump -ni en3f0pf0sf0 arp or icmp

I see no packets at all (no ARP, no ICMP).

Even when I send packets from the Arm-side SF (enp3s0f0s0) towards the host, nothing appears on the SF representor en3f0pf0sf0 either.
The bridge FDB also never learns the host MAC dynamically.

This makes me think that the drop is happening before Linux, somewhere in the PF → eSwitch → representor path.

3. Linux / OVS / TC don’t seem to be dropping anything

Current eSwitch mode:

$ sudo devlink dev eswitch show pci/0000:03:00.0
pci/0000:03:00.0: mode switchdev inline-mode none encap-mode basic

OVS flows are minimal:

$ sudo ovs-ofctl dump-flows ovsbr1
cookie=0x0, table=0, priority=0 actions=NORMAL

No ingress tc filters on the relevant ports:

$ sudo tc -s filter show dev p0 ingress
$ sudo tc -s filter show dev pf0hpf ingress
$ sudo tc -s filter show dev en3f0pf0sf0 ingress
# all empty

OVS offloaded flows are also empty.

So from what I can see, there is no explicit drop policy at the Linux / OVS / TC level that would explain why PF→rep traffic disappears.

4. Even with a plain Linux bridge (no OVS), the issue persists

To completely remove OVS from the equation, I created a very simple Linux bridge on the DPU:

sudo ovs-vsctl del-port ovsbr1 pf0hpf
sudo ovs-vsctl del-port ovsbr1 en3f0pf0sf0

sudo ip link add br-host-arm type bridge
sudo ip link set pf0hpf master br-host-arm
sudo ip link set enp3s0f0s0 master br-host-arm
sudo ip link set br-host-arm up
sudo ip link set pf0hpf up
sudo ip link set enp3s0f0s0 up

Even in this minimal setup:

  • ARP from the host still never appears on pf0hpf

  • The Linux bridge FDB does not learn the host MAC

So it doesn’t look like an OVS configuration issue.

5. Firmware / BFB status and resets

On the DPU:

$ sudo flint -d /dev/mst/mt41692_pciconf0 q | egrep 'FW Version|PSID'
FW Version:            32.46.1006
PSID:                  MT_0000000884

BFB in use:

  • bf-bundle-3.1.0-76_25.07_ubuntu-22.04_prod.bfb (DOCA 3.1)

I have already:

  1. Run mlxconfig -y reset

  2. Reflashed the above BFB

  3. Performed a full cold boot of the whole server (system-level reset)

  4. Verified that both BlueField-3 cards in this server show the exact same behavior

Before a certain cold boot, I could successfully communicate between host and DPU (Arm) over the data path at least once.
After that cold boot, with (as far as I know) the same configuration, both cards started showing this “no packets on pf0hpf” behavior.

6. Follow-up questions

Given all this, my current assumption is:

This is no longer a plain Linux/OVS misconfiguration,
but something in the hardware data path from host PF to the eSwitch / PF representor,
or some FW / policy that blocks host↔DPU traffic in this mode.

So as a follow-up to my original post, I would like to ask:

  1. Does this diagnosis make sense from your perspective?
    (That the problem is probably in the PF → eSwitch → representor path or its FW/policy, not in my Linux configuration?)

  2. For BlueField-3 + DOCA 3.1.0 + FW 32.46.1006 in ECPF mode, are there any known issues or policies where:

    • L2 traffic from the host PF never shows up on pf0hpf, or

    • P-series / this PSID might enforce stronger host isolation by default?

  3. Are there any additional debug tools or commands (beyond devlink, tc, ovs-*, tcpdump) that I can use to inspect how the eSwitch is actually forwarding (or not forwarding) the host PF traffic?

Any guidance on how to further debug or confirm if this is a FW/BSP/board-level problem would be greatly appreciated.

3

Let me understand you , you want to connect SF to host ?

4

Thank you for your support.
Regarding the issue where the host-side port could not connect to the SF, I actually managed to resolve it. However, I still don’t understand why it was resolved, so if you have time, I would appreciate your help in figuring it out.

When I posted the previous forum thread, I was assuming that I would not be using the external ports on the BlueField-3, so I did not plug a cable into the port corresponding to p0 (that’s why p0 was showing NO-CARRIER, although I did run ip link set up to bring it UP).

After that, I shut down the entire host including the BlueField once, then booted it again with the BlueField’s physical ports p0 and p1 connected to each other with a single cable. In this configuration, the host port and the SF were able to connect properly, and everything worked as expected.

I didn’t change any configuration before or after performing this procedure.

5

Can you share output

mlxconfig -d /dev/mst/mt41686_pciconf0 q

6

y-jinbo@1gpu:~$ sudo mlxconfig -d /dev/mst/mt41692_pciconf0 q

Device #1:

Device type: BlueField3
Name: 900-9D3B6-00CV-A_Ax
Description: NVIDIA BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled
Device: /dev/mst/mt41692_pciconf0

Configurations: Next Boot
ACCURATE_TX_SCHEDULER False(0)
ADVANCED_PCI_SETTINGS False(0)
RO ADVANCED_TESTABILITY False(0)
AES_XTS_TWEAK_INC_64 False(0)
ALLOW_RD_COUNTERS True(1)
ATS_ENABLED False(0)
AUTO_POWER_SAVE_LINK_DOWN_P1 False(0)
AUTO_POWER_SAVE_LINK_DOWN_P2 False(0)
BAR_PAGE_ALIGNMENT DEVICE_DEFAULT(0)
BOOT_DBG_LOG False(0)
BOOT_INTERRUPT_DIS False(0)
BOOT_LACP_DIS True(1)
BOOT_PKEY 0
BOOT_UNDI_NETWORK_WAIT 0
BOOT_VLAN_EN False(0)
BOOT_VLAN 1
CLAMP_TGT_RATE_AFTER_TIME_INC_P1 True(1)
CLAMP_TGT_RATE_AFTER_TIME_INC_P2 True(1)
CLAMP_TGT_RATE_P1 False(0)
CLAMP_TGT_RATE_P2 False(0)
CNP_802P_PRIO_P1 6
CNP_802P_PRIO_P2 6
CNP_DSCP_P1 48
CNP_DSCP_P2 48
CQE_COMPRESSION BALANCED(0)
CRYPTO_POLICY UNRESTRICTED(1)
RO CUSTOMIZATION_NUMBER Array[0..15]
DCBX_CEE_P1 True(1)
DCBX_CEE_P2 True(1)
DCBX_IEEE_P1 True(1)
DCBX_IEEE_P2 True(1)
DCBX_WILLING_P1 True(1)
DCBX_WILLING_P2 True(1)
DCE_TCP_G_P1 1019
DCE_TCP_G_P2 1019
DCE_TCP_RTT_P1 1
DCE_TCP_RTT_P2 1
DCR_LIFO_SIZE 16384
DELAY_HOST_OS_INIT DEVICE_DEFAULT(0)
DO_NOT_CLEAR_PORT_STATS_P1 False(0)
DO_NOT_CLEAR_PORT_STATS_P2 False(0)
DPA_AUTHENTICATION False(0)
DUP_MAC_ACTION_P1 LAST_CFG(0)
DUP_MAC_ACTION_P2 LAST_CFG(0)
DYNAMIC_VF_MSIX_TABLE False(0)
ESWITCH_HAIRPIN_DESCRIPTORS Array[0..7]
ESWITCH_HAIRPIN_TOT_BUFFER_SIZE Array[0..7]
ETH_ETS_GRANULARITY_P1 DEVICE_DEFAULT(0)
ETH_ETS_GRANULARITY_P2 DEVICE_DEFAULT(0)
ETS_SCHED_MODE_P1 device_default(0)
ETS_SCHED_MODE_P2 device_default(0)
EXP_ROM_NVME_UEFI_x86_ENABLE True(1)
EXP_ROM_PXE_ENABLE True(1)
EXP_ROM_UEFI_ARM_ENABLE True(1)
EXP_ROM_UEFI_x86_ENABLE True(1)
EXP_ROM_VIRTIO_BLK_UEFI_ARM_ENABLE True(1)
EXP_ROM_VIRTIO_BLK_UEFI_x86_ENABLE True(1)
EXP_ROM_VIRTIO_NET_PXE_ENABLE True(1)
EXP_ROM_VIRTIO_NET_UEFI_ARM_ENABLE True(1)
EXP_ROM_VIRTIO_NET_UEFI_x86_ENABLE True(1)
FLEX_IPV4_OVER_VXLAN_PORT 0
FLEX_PARSER_PROFILE_ENABLE 0
HAIRPIN_DATA_BUFFER_LOCK False(0)
HIDE_PORT2_PF False(0)
HOST_CHAINING_CACHE_DISABLE False(0)
HOST_CHAINING_DESCRIPTORS Array[0..7]
HOST_CHAINING_MODE DISABLED(0)
HOST_CHAINING_TOTAL_BUFFER_SIZE Array[0..7]
IB_CC_SHAPER_COALESCE_P1 DEVICE_DEFAULT(0)
IB_CC_SHAPER_COALESCE_P2 DEVICE_DEFAULT(0)
IB_ETS_GRANULARITY_P1 DEVICE_DEFAULT(0)
IB_ETS_GRANULARITY_P2 DEVICE_DEFAULT(0)
IB_ROUTING_MODE_P1 LID(1)
IB_ROUTING_MODE_P2 LID(1)
ICM_CACHE_MODE DEVICE_DEFAULT(0)
INITIAL_ALPHA_VALUE_P1 1023
INITIAL_ALPHA_VALUE_P2 1023
INTERNAL_CPU_ESWITCH_MANAGER ECPF(0)
INTERNAL_CPU_IB_VPORT0 ECPF(0)
INTERNAL_CPU_MODEL EMBEDDED_CPU(1)
INTERNAL_CPU_OFFLOAD_ENGINE ENABLED(0)
INTERNAL_CPU_PAGE_SUPPLIER ECPF(0)
INTERNAL_CPU_RSHIM ENABLED(0)
INT_CPU_AUTO_SHUTDOWN DEVICE_DEFAULT(0)
INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0)
IP_OVER_VXLAN_EN False(0)
IP_VER IPv4(0)
KEEP_ETH_LINK_UP_P1 True(1)
KEEP_ETH_LINK_UP_P2 True(1)
KEEP_IB_LINK_UP_P1 False(0)
KEEP_IB_LINK_UP_P2 False(0)
KEEP_LINK_UP_ON_BOOT_P1 False(0)
KEEP_LINK_UP_ON_BOOT_P2 False(0)
KEEP_LINK_UP_ON_STANDBY_P1 False(0)
KEEP_LINK_UP_ON_STANDBY_P2 False(0)
LAG_RESOURCE_ALLOCATION DEVICE_DEFAULT(0)
LARGE_MTU_TWEAK_64 False(0)
LEGACY_BOOT_PROTOCOL PXE(1)
LINK_TYPE_P1 ETH(2)
LINK_TYPE_P2 ETH(2)
LLDP_NB_DCBX_P1 False(0)
LLDP_NB_DCBX_P2 False(0)
LLDP_NB_RX_MODE_P1 OFF(0)
LLDP_NB_RX_MODE_P2 OFF(0)
LLDP_NB_TX_MODE_P1 OFF(0)
LLDP_NB_TX_MODE_P2 OFF(0)
LOAD_BALANCE_MODE_P1 TRANSPORT(2)
LOAD_BALANCE_MODE_P2 TRANSPORT(2)
LOG_DCR_HASH_TABLE_SIZE 11
LOG_MAX_OUTSTANDING_READ_ATOMIC 0
LOG_MAX_OUTSTANDING_WQE 7
LOG_MAX_QUEUE 17
LOG_TX_PSN_WINDOW 9
LRO_LOG_TIMEOUT0 6
LRO_LOG_TIMEOUT1 7
LRO_LOG_TIMEOUT2 8
LRO_LOG_TIMEOUT3 13
MAX_PACKET_LIFETIME 0
MCTP_I3C_DISABLE False(0)
MEMIC_ATOMIC MEMIC_ATOMIC_ENABLE(2)
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
MIN_TIME_BETWEEN_CNPS_P1 4
MIN_TIME_BETWEEN_CNPS_P2 4
MKEY_BY_NAME False(0)
MKEY_BY_NAME_RANGE DEVICE_DEFAULT(0)
MODULE_SPLIT_M0 Array[0..15]
MODULE_SPLIT_M1 Array[0..15]
MPFS_MC_LOOPBACK_DISABLE_P1 False(0)
MPFS_MC_LOOPBACK_DISABLE_P2 False(0)
MPFS_UC_LOOPBACK_DISABLE_P1 False(0)
MPFS_UC_LOOPBACK_DISABLE_P2 False(0)
MULTIPATH_DSCP DEVICE_DEFAULT(0)
MULTI_PCI_RESOURCE_SHARING DEVICE_DEFAULT(0)
MULTI_PORT_VHCA_EN False(0)
NON_PREFETCHABLE_PF_BAR False(0)
NUM_OF_PFC_P1 8
NUM_OF_PFC_P2 8
NUM_OF_PF 2
NUM_OF_PLANES_P1 0
NUM_OF_PLANES_P2 0
NUM_OF_TC_P1 _8_TCs(0)
NUM_OF_TC_P2 _8_TCs(0)
NUM_OF_VFS 16
NUM_OF_VL_P1 _4_VLs(3)
NUM_OF_VL_P2 _4_VLs(3)
NUM_PF_MSIX 63
NUM_PF_MSIX_VALID True(1)
NUM_VF_MSIX 11
NVME_EMULATION_CLASS_CODE 67586
NVME_EMULATION_DEVICE_ID 24577
NVME_EMULATION_ENABLE False(0)
NVME_EMULATION_MAX_QUEUE_DEPTH 0
NVME_EMULATION_NUM_MSIX 0
NVME_EMULATION_NUM_PF 1
NVME_EMULATION_NUM_VF_MSIX 0
NVME_EMULATION_NUM_VF 0
NVME_EMULATION_REVISION_ID 0
NVME_EMULATION_SUBSYSTEM_ID 0
NVME_EMULATION_SUBSYSTEM_VENDOR_ID 0
NVME_EMULATION_VENDOR_ID 5555
OFF_BOARD_SERIALIZER False(0)
P2P_ORDERING_MODE DEVICE_DEFAULT(0)
PARTIAL_RESET_EN False(0)
PCC_HANDLE_CORE_UTIL DEVICE_DEFAULT(0)
PCC_INT_EN False(0)
PCC_INT_NP_RTT_DATA_MODE RTT_V0(64)
PCC_INT_NP_RTT_DSCP_EN False(0)
PCC_INT_NP_RTT_DSCP 26
PCC_INT_SYSTEM_RTT 0
PCC_NP_HANDLE_CORE_UTIL DEVICE_DEFAULT(0)
PCIE_CONGESTION_MONITOR DEVICE_DEFAULT(0)
PCIE_CREDIT_TOKEN_TIMEOUT 0
PCIE_IN_BAND_VDM_DISABLE False(0)
PCIE_SMBUS_DISABLE False(0)
PCI_ATOMIC_MODE PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
PCI_BUS00_ASPM False(0)
PCI_BUS00_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS00_SPEED PCI_GEN_1(0)
PCI_BUS00_SWITCH_INDEX 0
PCI_BUS00_WIDTH PCI_INACTIVE(0)
PCI_BUS01_ASPM False(0)
PCI_BUS01_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS01_SPEED PCI_GEN_1(0)
PCI_BUS01_SWITCH_INDEX 0
PCI_BUS01_WIDTH PCI_INACTIVE(0)
PCI_BUS02_ASPM False(0)
PCI_BUS02_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS02_SPEED PCI_GEN_1(0)
PCI_BUS02_SWITCH_INDEX 0
PCI_BUS02_WIDTH PCI_INACTIVE(0)
PCI_BUS03_ASPM False(0)
PCI_BUS03_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS03_SPEED PCI_GEN_1(0)
PCI_BUS03_SWITCH_INDEX 0
PCI_BUS03_WIDTH PCI_INACTIVE(0)
PCI_BUS04_ASPM False(0)
PCI_BUS04_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS04_SPEED PCI_GEN_1(0)
PCI_BUS04_SWITCH_INDEX 0
PCI_BUS04_WIDTH PCI_INACTIVE(0)
PCI_BUS05_ASPM False(0)
PCI_BUS05_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS05_SPEED PCI_GEN_1(0)
PCI_BUS05_SWITCH_INDEX 0
PCI_BUS05_WIDTH PCI_INACTIVE(0)
PCI_BUS06_ASPM False(0)
PCI_BUS06_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS06_SPEED PCI_GEN_1(0)
PCI_BUS06_SWITCH_INDEX 0
PCI_BUS06_WIDTH PCI_INACTIVE(0)
PCI_BUS07_ASPM False(0)
PCI_BUS07_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS07_SPEED PCI_GEN_1(0)
PCI_BUS07_SWITCH_INDEX 0
PCI_BUS07_WIDTH PCI_INACTIVE(0)
PCI_BUS10_ASPM False(0)
PCI_BUS10_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS10_SPEED PCI_GEN_1(0)
PCI_BUS10_SWITCH_INDEX 0
PCI_BUS10_WIDTH PCI_INACTIVE(0)
PCI_BUS11_ASPM False(0)
PCI_BUS11_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS11_SPEED PCI_GEN_1(0)
PCI_BUS11_SWITCH_INDEX 0
PCI_BUS11_WIDTH PCI_INACTIVE(0)
PCI_BUS12_ASPM False(0)
PCI_BUS12_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS12_SPEED PCI_GEN_1(0)
PCI_BUS12_SWITCH_INDEX 0
PCI_BUS12_WIDTH PCI_INACTIVE(0)
PCI_BUS13_ASPM False(0)
PCI_BUS13_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS13_SPEED PCI_GEN_1(0)
PCI_BUS13_SWITCH_INDEX 0
PCI_BUS13_WIDTH PCI_INACTIVE(0)
PCI_BUS14_ASPM False(0)
PCI_BUS14_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS14_SPEED PCI_GEN_1(0)
PCI_BUS14_SWITCH_INDEX 0
PCI_BUS14_WIDTH PCI_INACTIVE(0)
PCI_BUS15_ASPM False(0)
PCI_BUS15_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS15_SPEED PCI_GEN_1(0)
PCI_BUS15_SWITCH_INDEX 0
PCI_BUS15_WIDTH PCI_INACTIVE(0)
PCI_BUS16_ASPM False(0)
PCI_BUS16_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS16_SPEED PCI_GEN_1(0)
PCI_BUS16_SWITCH_INDEX 0
PCI_BUS16_WIDTH PCI_INACTIVE(0)
PCI_BUS17_ASPM False(0)
PCI_BUS17_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS17_SPEED PCI_GEN_1(0)
PCI_BUS17_SWITCH_INDEX 0
PCI_BUS17_WIDTH PCI_INACTIVE(0)
PCI_BUS20_ASPM False(0)
PCI_BUS20_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS20_SPEED PCI_GEN_1(0)
PCI_BUS20_SWITCH_INDEX 0
PCI_BUS20_WIDTH PCI_INACTIVE(0)
PCI_BUS21_ASPM False(0)
PCI_BUS21_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS21_SPEED PCI_GEN_1(0)
PCI_BUS21_SWITCH_INDEX 0
PCI_BUS21_WIDTH PCI_INACTIVE(0)
PCI_BUS22_ASPM False(0)
PCI_BUS22_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS22_SPEED PCI_GEN_1(0)
PCI_BUS22_SWITCH_INDEX 0
PCI_BUS22_WIDTH PCI_INACTIVE(0)
PCI_BUS23_ASPM False(0)
PCI_BUS23_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS23_SPEED PCI_GEN_1(0)
PCI_BUS23_SWITCH_INDEX 0
PCI_BUS23_WIDTH PCI_INACTIVE(0)
PCI_BUS24_ASPM False(0)
PCI_BUS24_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS24_SPEED PCI_GEN_1(0)
PCI_BUS24_SWITCH_INDEX 0
PCI_BUS24_WIDTH PCI_INACTIVE(0)
PCI_BUS25_ASPM False(0)
PCI_BUS25_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS25_SPEED PCI_GEN_1(0)
PCI_BUS25_SWITCH_INDEX 0
PCI_BUS25_WIDTH PCI_INACTIVE(0)
PCI_BUS26_ASPM False(0)
PCI_BUS26_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS26_SPEED PCI_GEN_1(0)
PCI_BUS26_SWITCH_INDEX 0
PCI_BUS26_WIDTH PCI_INACTIVE(0)
PCI_BUS27_ASPM False(0)
PCI_BUS27_HIERARCHY_TYPE PCIE_ENDPOINT(0)
PCI_BUS27_SPEED PCI_GEN_1(0)
PCI_BUS27_SWITCH_INDEX 0
PCI_BUS27_WIDTH PCI_INACTIVE(0)
PCI_DOWNSTREAM_PORT_OWNER Array[0..15]
PCI_SWITCH0_UPSTREAM_PORT_BUS 0
PCI_SWITCH0_UPSTREAM_PORT_PEX 0
PCI_SWITCH1_UPSTREAM_PORT_BUS 0
PCI_SWITCH1_UPSTREAM_PORT_PEX 0
PCI_SWITCH2_UPSTREAM_PORT_BUS 0
PCI_SWITCH2_UPSTREAM_PORT_PEX 0
PCI_SWITCH_EMULATION_ENABLE False(0)
PCI_SWITCH_EMULATION_NUM_PORT 16
PCI_WR_ORDERING per_mkey(0)
PCORE0_REVERSAL False(0)
PCORE1_REVERSAL False(0)
PCORE2_REVERSAL False(0)
PER_PF_NUM_SF False(0)
PF_BAR2_ENABLE True(1)
PF_BAR2_SIZE 3
PF_DEVICE_ID_ENABLE False(0)
PF_DEVICE_ID 41692
PF_LOG_BAR_SIZE 5
PF_NUM_OF_VF_VALID False(0)
PF_NUM_PF_MSIX 63
PF_NUM_PF_MSIX_VALID False(0)
PF_SD_GROUP 0
PF_SF_BAR_SIZE 0
PF_TOTAL_SF 0
PHY_AUTO_NEG_P1 DEVICE_DEFAULT(0)
PHY_AUTO_NEG_P2 DEVICE_DEFAULT(0)
PHY_ETH_RECOVERY_P1 DEVICE_DEFAULT(0)
PHY_ETH_RECOVERY_P2 DEVICE_DEFAULT(0)
PHY_FEC_OVERRIDE_P1 DEVICE_DEFAULT(0)
PHY_FEC_OVERRIDE_P2 DEVICE_DEFAULT(0)
PHY_RATE_MASK_OVERRIDE_P1 False(0)
PHY_RATE_MASK_OVERRIDE_P2 False(0)
PLDM_FW_UPDATE_DISABLE False(0)
PORT_OWNER True(1)
PRIO_TAG_REQUIRED_EN False(0)
PROG_PARSE_GRAPH False(0)
QOS_TRUST_STATE_P1 TRUST_PCP(1)
QOS_TRUST_STATE_P2 TRUST_PCP(1)
RATE_REDUCE_MONITOR_PERIOD_P1 4
RATE_REDUCE_MONITOR_PERIOD_P2 4
RATE_TO_SET_ON_FIRST_CNP_P1 0
RATE_TO_SET_ON_FIRST_CNP_P2 0
RBT_DISABLE False(0)
RDE_DISABLE False(0)
RDMA_SELECTIVE_REPEAT_EN False(0)
REAL_TIME_CLOCK_ENABLE False(0)
RENEG_ON_CHANGE True(1)
RESET_WITH_HOST_ON_ERRORS False(0)
ROCE_ADAPTIVE_ROUTING_EN False(0)
ROCE_CC_CNP_MODERATION_P1 DEVICE_DEFAULT(0)
ROCE_CC_CNP_MODERATION_P2 DEVICE_DEFAULT(0)
ROCE_CC_DCQCN_COMPATIBILITY_MODE DEVICE_DEFAULT(0)
ROCE_CC_IFA2_TA DEVICE_DEFAULT(0)
ROCE_CC_LEGACY_DCQCN_SW False(0)
ROCE_CC_PRIO_MASK_P1 255
ROCE_CC_PRIO_MASK_P2 255
ROCE_CC_SHAPER_COALESCE_P1 DEVICE_DEFAULT(0)
ROCE_CC_SHAPER_COALESCE_P2 DEVICE_DEFAULT(0)
ROCE_CC_STEERING_EXT DISABLED(1)
ROCE_CONTROL ROCE_ENABLE(2)
ROCE_NEXT_PROTOCOL 254
ROCE_RTT_RESP_DSCP_MODE_P1 DEVICE_DEFAULT(0)
ROCE_RTT_RESP_DSCP_MODE_P2 DEVICE_DEFAULT(0)
ROCE_RTT_RESP_DSCP_P1 0
ROCE_RTT_RESP_DSCP_P2 0
RPG_AI_RATE_P1 5
RPG_AI_RATE_P2 5
RPG_BYTE_RESET_P1 32767
RPG_BYTE_RESET_P2 32767
RPG_GD_P1 11
RPG_GD_P2 11
RPG_HAI_RATE_P1 50
RPG_HAI_RATE_P2 50
RPG_MAX_RATE_P1 0
RPG_MAX_RATE_P2 0
RPG_MIN_DEC_FAC_P1 50
RPG_MIN_DEC_FAC_P2 50
RPG_MIN_RATE_P1 1
RPG_MIN_RATE_P2 1
RPG_THRESHOLD_P1 1
RPG_THRESHOLD_P2 1
RPG_TIME_RESET_P1 300
RPG_TIME_RESET_P2 300
RT_PPS_ENABLED_ON_POWERUP False(0)
SAFE_MODE_ENABLE True(1)
SAFE_MODE_THRESHOLD 10
SILENT_MODE False(0)
SM_DISABLE False(0)
SRIOV_EN True(1)
SRIOV_IB_ROUTING_MODE_P1 LID(1)
SRIOV_IB_ROUTING_MODE_P2 LID(1)
STEERING_CACHE_REFRESH 0
STRAP_SD_OR_MH False(0)
STRICT_VF_MSIX_NUM False(0)
SWITCH_COMPT_FEATURE_MASK 0x0(0)
SWP_L4_CHECKSUM_MODE DEVICE_DEFAULT(0)
TRACER_ENABLE True(1)
TUNNEL_ECN_COPY_DISABLE False(0)
TUNNEL_IP_PROTO_ENTROPY_DISABLE False(0)
TX_PCI_DATA_FETCH_LATENCY 40
TX_SCHEDULER_BURST 0
TX_SCHEDULER_FWS_REACTIVITY DIRECT(1)
TX_SCHEDULER_LOCALITY_FACTOR 0
TX_SCHEDULER_LOCALITY_MODE DEVICE_DEFAULT(0)
UCTX_EN True(1)
UEFI_HII_EN True(1)
UEFI_LOGS DISABLED(0)
UNKNOWN_UPLINK_MAC_FLOOD_P1 False(0)
UNKNOWN_UPLINK_MAC_FLOOD_P2 False(0)
UPSTREAM_PORT0_GEN3_PRESET0 0
UPSTREAM_PORT0_GEN3_PRESET1 0
UPSTREAM_PORT0_GEN4_PRESET0 0
UPSTREAM_PORT0_GEN4_PRESET1 0
UPSTREAM_PORT0_GEN5_PRESET0 0
UPSTREAM_PORT0_GEN5_PRESET1 0
UPSTREAM_PORT0_GEN6_PRESET0 0
UPSTREAM_PORT0_GEN6_PRESET1 0
UPSTREAM_PORT0_PCIE_BUS 0
UPSTREAM_PORT0_TX_AMP 0
UPSTREAM_PORT1_GEN3_PRESET0 0
UPSTREAM_PORT1_GEN3_PRESET1 0
UPSTREAM_PORT1_GEN4_PRESET0 0
UPSTREAM_PORT1_GEN4_PRESET1 0
UPSTREAM_PORT1_GEN5_PRESET0 0
UPSTREAM_PORT1_GEN5_PRESET1 0
UPSTREAM_PORT1_GEN6_PRESET0 0
UPSTREAM_PORT1_GEN6_PRESET1 0
UPSTREAM_PORT1_PCIE_BUS 0
UPSTREAM_PORT1_TX_AMP 0
UPSTREAM_PORT2_GEN3_PRESET0 0
UPSTREAM_PORT2_GEN3_PRESET1 0
UPSTREAM_PORT2_GEN4_PRESET0 0
UPSTREAM_PORT2_GEN4_PRESET1 0
UPSTREAM_PORT2_GEN5_PRESET0 0
UPSTREAM_PORT2_GEN5_PRESET1 0
UPSTREAM_PORT2_GEN6_PRESET0 0
UPSTREAM_PORT2_GEN6_PRESET1 0
UPSTREAM_PORT2_PCIE_BUS 0
UPSTREAM_PORT2_TX_AMP 0
UPSTREAM_PORT3_GEN3_PRESET0 0
UPSTREAM_PORT3_GEN3_PRESET1 0
UPSTREAM_PORT3_GEN4_PRESET0 0
UPSTREAM_PORT3_GEN4_PRESET1 0
UPSTREAM_PORT3_GEN5_PRESET0 0
UPSTREAM_PORT3_GEN5_PRESET1 0
UPSTREAM_PORT3_GEN6_PRESET0 0
UPSTREAM_PORT3_GEN6_PRESET1 0
UPSTREAM_PORT3_PCIE_BUS 0
UPSTREAM_PORT3_TX_AMP 0
UPT_EMULATION_ENABLE False(0)
USER_PROGRAMMABLE_CC False(0)
VF_LOG_BAR_SIZE 1
VF_MIGRATION_MODE DEVICE_DEFAULT(0)
VF_NODNIC_ENABLE False(0)
VF_VPD_ENABLE False(0)
VIRTIO_BLK_EMULATION_ENABLE False(0)
VIRTIO_BLK_EMULATION_NUM_MSIX 2
VIRTIO_BLK_EMULATION_NUM_PF 0
VIRTIO_BLK_EMULATION_NUM_VF_MSIX 0
VIRTIO_BLK_EMULATION_NUM_VF 0
VIRTIO_BLK_EMULATION_SUBSYSTEM_ID 4162
VIRTIO_BLK_EMU_SUBS_VENDOR_ID 6900
VIRTIO_FS_EMULATION_ENABLE False(0)
VIRTIO_FS_EMULATION_NUM_MSIX 2
VIRTIO_FS_EMULATION_NUM_PF 0
VIRTIO_FS_EMULATION_NUM_VF_MSIX 0
VIRTIO_FS_EMULATION_NUM_VF 0
VIRTIO_FS_EMULATION_SUBSYSTEM_ID 4186
VIRTIO_FS_EMU_SUBSYSTEM_VENDOR_ID 6900
VIRTIO_NET_EMULATION_ENABLE False(0)
VIRTIO_NET_EMULATION_NUM_MSIX 2
VIRTIO_NET_EMULATION_NUM_PF 0
VIRTIO_NET_EMULATION_NUM_VF_MSIX 0
VIRTIO_NET_EMULATION_NUM_VF 0
VIRTIO_NET_EMULATION_SUBSYSTEM_ID 4161
VIRTIO_NET_EMU_SUBSYSTEM_VENDOR_ID 6900
VL15_BUFFER_SIZE_P1 0
VL15_BUFFER_SIZE_P2 0
VL_BUFFER_ALLOCATION_P1 FULL_DYNAMIC(1)
VL_BUFFER_ALLOCATION_P2 FULL_DYNAMIC(1)
VL_ISOLATION_MODE_P1 DEVICE_DEFAULT(0)
VL_ISOLATION_MODE_P2 DEVICE_DEFAULT(0)
ZERO_TOUCH_TUNING_ENABLE False(0)

The ‘RO’ shows parameters which are for read only and cannot be changed

I ran mlxconfig reset, so I believe the settings have reverted to their defaults.

7

Hello,

Thank you for your thread.

Reviewing your responses it appears that you have been able to resolve the issue.

For further investigation the following guide may be helpful to review:

If a deeper investigation is needed to find the root cause of the issue, a ticket would need to be opened through the NVIDIA Enterprise support portal found at the following link:

Thank you,

NVEX Networking Technical Support Team