Mellanox Technologies MT2892 Family [ConnectX-6 Dx] SRIOV using Openstack - No IP inside the VM

[vinubama.k]

Hello,

Im using SRIOV using Openstack(version: Antelope) which is deployed using kolla-ansible deployment (single node setup).
Ethernet card details:

root@poc-dev-3:/home/user# lspci -nnv -s a2:00.0
a2:00.0 Ethernet controller [0200]: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] [15b3:101d]
	Subsystem: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] [15b3:0040]
	Physical Slot: 5
	Flags: bus master, fast devsel, latency 0, IRQ 18, NUMA node 2, IOMMU group 161
	Memory at e0000000 (64-bit, prefetchable) [size=32M]
	Capabilities: [60] Express Endpoint, MSI 00
	Capabilities: [48] Vital Product Data
	Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
	Capabilities: [c0] Vendor Specific Information: Len=18 <?>
	Capabilities: [40] Power Management version 3
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
	Capabilities: [1c0] Secondary PCI Express
	Capabilities: [320] Lane Margining at the Receiver <?>
	Capabilities: [370] Physical Layer 16.0 GT/s <?>
	Capabilities: [420] Data Link Feature <?>
	Kernel driver in use: mlx5_core
	Kernel modules: mlx5_core

OS Version: Ubuntu 22.04.5 LTS
Grub changes:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt intel_iommu=on"

SDN used: OVN(enabled with DHCP)
Number of VF’s created using the following command:

echo '7' > /sys/class/net/ens3np0/device/sriov_numvfs

root@poc-dev-3:/home/user# cat /sys/class/net/ens3np0/device/sriov_totalvfs
8

Other Info:

1. Enabled SRIOV & VT-d in the BIOS configuration
2. Enabled iommu=pt intel_iommu=on on the grub changes and rebooted the server
3. Enabled neutron_sriov in the globals.yml file(which is used in kolla-ansible’s deployment)
4. Created a physical_network_function(physnet) called sriovtenant1 which is mapped to ens3np0 port where SRIOV is enabled
5. Made all the necessary changes in nova-compute(pci devices are listed passthrough_whitelist = [{“physical_network”: “sriovtenant1”, “devname”: “ens3np0”}]
6. Similarlly for ml2_conf.ini

[ml2]
type_drivers = flat,vlan,vxlan,geneve
tenant_network_types = geneve
mechanism_drivers = ovn,sriovnicswitch
extension_drivers = port_security

[ml2_type_vlan]
network_vlan_ranges = sriovtenant1

[ml2_type_flat]
flat_networks = physnet1, sriovtenant1

7. Enabled PCI passthrough filter in nova-scheduler
8. ip link show ens3np0

root@poc-dev-3:/home/user# ip link show ens3np0
6: ens3np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 88:e9:a4:1f:2f:62 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 4     link/ether fa:16:3e:e0:03:16 brd ff:ff:ff:ff:ff:ff, vlan 805, spoof checking on, link-state enable, trust off, query_rss off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 7     link/ether 22:56:d7:64:61:89 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state enable, trust off, query_rss off
    altname enp15s0np0

9. Created sriov-net and used the VLAN network type (provided the same VLAN ID where the port ens3np0 was connected to switch) and created the subnets & port with vnic type as direct
10. Launched an instance using the port created from sriov-net, But there is no IP address allocated in the VM
11. No IP Inside the VM
    
    
12. Even Static IP allocation is also not happening

Help me resolve this issue

[samerka]

Hi ,

Looking at your SR-IOV setup, the issue appears to be related to how networking works with SR-IOV direct passthrough. Let me help you troubleshoot this step by step.

Root Cause Analysis

The fundamental issue: With SR-IOV vnic_type=direct, the virtual function bypasses the neutron networking stack, so DHCP doesn’t work automatically. This is expected behavior, not a bug.

Troubleshooting Steps

1. Verify VF Assignment

Check which VF was assigned to your VM:

# Check nova logs
grep -i sriov /var/log/nova/nova-compute.log

# Check which VFs are bound
lspci | grep -i virtual

2. Guest VM Driver Check

Inside the VM, verify the SR-IOV interface is detected:

# In the VM
lspci | grep -i mellanox
ip link show
dmesg | grep mlx

3. Network Configuration Options

Option A: Static IP Configuration (Recommended)

In the VM, manually configure the interface:

# Inside VM - replace with your network details
sudo ip addr add 192.168.1.100/24 dev <interface_name>
sudo ip route add default via 192.168.1.1
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Option B: Cloud-init/Metadata Configuration

Create the port with a fixed IP in OpenStack:

# Create port with fixed IP
openstack port create \
  --network sriov-net \
  --vnic-type direct \
  --fixed-ip subnet=<subnet_name>,ip-address=192.168.1.100 \
  sriov-port

# Launch instance with this port
openstack server create \
  --port sriov-port \
  --flavor <flavor> \
  --image <image> \
  vm-name

4. VLAN Configuration Verification

Verify VLAN settings match:

# Check physical switch VLAN configuration
# Ensure the VLAN ID used in neutron matches the switch port configuration

# Check neutron network VLAN
openstack network show sriov-net -f value -c provider:segmentation_id

5. Advanced Troubleshooting

Check PCI Device Binding:

# Verify PCI passthrough
nova show <instance-id> | grep pci
virsh dumpxml <instance> | grep hostdev

Driver Issues in Guest:

# In VM - install/update Mellanox drivers if needed
sudo apt update
sudo apt install mlnx-ofed-all # or appropriate driver package

Alternative: Use macvtap instead of direct

If you need DHCP functionality:

# Create port with macvtap (allows some neutron features)
openstack port create \
  --network sriov-net \
  --vnic-type macvtap \
  sriov-macvtap-port

Configuration Corrections

1. Update nova.conf for better debugging:

[pci]
passthrough_whitelist = [{"vendor_id": "15b3", "product_id": "101e", "physical_network": "sriovtenant1"}]
alias = [{"vendor_id": "15b3", "product_id": "101e", "name": "connectx6dx"}]

2. Ensure proper neutron ml2 config:

[ml2_type_vlan]
network_vlan_ranges = sriovtenant1:1:4094

[securitygroup]
enable_security_group = true
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Quick Test

1. Manually configure IP in VM as shown in Option A above
2. Test connectivity to verify SR-IOV is working
3. If successful, implement automated IP assignment via cloud-init or custom scripts

The lack of automatic IP assignment is normal behavior for SR-IOV direct passthrough. Focus on static configuration or cloud-init metadata to provide network settings to your VMs.

In case you have further questions , please contact OpenStack support for best practices or refer to the below

Nova-neutron-sriov - OpenStack

Thanks
Samer