Connectivity Issues between Two NIC Ports on the Same Machine

Infrastructure & Networking BlueField
, ,
1

Description:

I am experiencing connectivity issues between two network interface cards (NICs) on the same server, and I’m unable to establish a ping between them. I’m seeking assistance to resolve this issue.

Environment:

  • Operating System: Ubuntu 20.04.5 LTS
  • NIC 1: enp130s0np0 (192.168.3.1/24)
  • NIC 2: enp3s0f0np0 (192.168.3.2/24)
  • NIC 2 Type: Nvidia Bluefield (with embedded CPU configuration)
  • Both NICs are directly connected via an Ethernet cable.
$ ip route
default via 10.190.172.1 dev eno1 proto static 
default via 10.190.172.1 dev eno1 proto dhcp src 10.190.172.107 metric 100 
10.190.172.0/22 dev eno1 proto kernel scope link src 10.190.174.31 
10.190.172.1 dev eno1 proto dhcp scope link src 10.190.172.107 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.3.0/24 dev enp130s0np0 proto kernel scope link src 192.168.3.1 
192.168.3.0/24 dev enp3s0f1np1 proto kernel scope link src 192.168.3.3 
192.168.3.0/24 dev enp3s0f0np0 proto kernel scope link src 192.168.3.2 
192.168.100.0/24 dev tmfifo_net0 proto kernel scope link src 192.168.100.1 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Issue Details:

Despite the direct connection, attempts to ping from one NIC to the other result in “Destination Host Unreachable” errors from both ends.

  • Ping from enp3s0f0np0 to enp130s0np0:
PING 192.168.3.1 (192.168.3.1) from 192.168.3.2 enp3s0f0np0: 56(84) bytes of data.
From 192.168.3.2 icmp_seq=1 Destination Host Unreachable
...
  • Ping from enp130s0np0 to enp3s0f0np0:
PING 192.168.3.2 (192.168.3.2) from 192.168.3.1 enp130s0np0: 56(84) bytes of data.
From 192.168.3.1 icmp_seq=1 Destination Host Unreachable
...

Configuration of NIC 2 (Nvidia Bluefield):

  • Mode: Embedded PCU
  • Relevant configurations that might affect connectivity are provided below:
HOST_CHAINING_MODE: DISABLED
INTERNAL_CPU_MODEL: EMBEDDED_CPU
SRIOV_EN: True
NUM_OF_VFS: 10
...

Request:

I am looking for guidance on why these connectivity issues might be occurring and how to resolve them. I suspect there might be a configuration or hardware-related issue, especially concerning the special properties of the Nvidia Bluefield card in embedded PCU mode.

Thank you in advance for any suggestions or guidance you can provide!

2

Hi,

Thanks for your question.
Most likely the issue is not with the Bluefield adapter itself.

Based on the network configuration you shared, there are a few interfaces on the same server, which are configured with addresses in the same subnet 192.168.3.0/24

This configuration may cause various network issues (ARP, routing issues), as the kernel may reply to ARP via wrong interface, and as a result no connectivity.

In this case you may configure a routing table per interface, or just not using same subnet on a few ports on the same server.

Please find the below explanations:

#All the settings are very well documented in below link
#Chapter 2. Working with sysctl and kernel tunables Red Hat Enterprise Linux 7 | Red Hat Customer Portal

#default is usually 0
#value 1 is used if there is static source based routing

in order to force ARPs for each interface be answered based on

#whether or not the kernel would route a packet from the ARP’d IP out that interface
sysctl -w net.ipv4.conf.all.arp_filter=1
sysctl -w net.ipv4.conf.default.arp_filter=1

#value 1 or 2 can work, the difference if the subnet is checked as well
sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.default.arp_ignore=1

#either 1 or 2, 1 works better as value 2 ignores IP addr in src packet
sysctl -w net.ipv4.conf.all.arp_announce=1
sysctl -w net.ipv4.conf.default.arp_announce=1

#rp_filter can be set to 0 or 2, just not 1
sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.default.rp_filter=2

Best Regards,
Anatoly

1 Like
3

Thank you very much, Anatoly! I appreciate your detailed advice and will implement these settings to see if they help resolve the issue.