Intel X520-DA2 10GBE PCIe card with Jetson AGX Orin with DPDK

[vkakade]

I am trying to get the Intel X520-DA2 dual port 10GBE PCIe card working with Jetson AGX Orin and unable to DPDK working with it due to the following errors during DPDK EAL initialization.
The same card works fine without DPDK and I am able to stream data using it to another device. I found some similar posts here: where they saw similar errors but the difference being I am able to use this card just fine without DPDK on the Jetson.

100 GbE on a Jetson AGX Orin dev kit

What is our best chance to use a 100 GbE NIC (with DPDK) in a Jetson AGX Orin dev kit? So far, we tried: an NVIDIA MCX653105A-ECAT: not detected with lspci after booting (not even after echo 1 >/sys/bus/pci/rescan). an Intel e810: works fine with the ice driver, but not with DPDK, as the vfio_pci driver complains that IOMMU group 12 is not viable a QNAP QXG-25G2SF-CX6: works (also with DPDK), but needs multiple boot attempts to be detected, and is only dual 25 GbE. We use JetPack 5.0.2 with t…

DPDK version 19.11
Driver used vfio-pci

On a typical linux host desktop I would just get this to work by enabling IOMMU support in BIOS and add kernel args so tried to follow a similar process here except that there is nothing to be done in the BIOS/bootloader to support IOMMU afaik for Jetson and we have support for that out of the box.

Currently my kernel args looks like this after my modifications on Jetson.
APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 console=tty0 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb iommu=pt iommu=on hugepages=2048

EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-32768kB
EAL: No available hugepages reported in hugepages-64kB
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0005:01:00.0 on NUMA socket 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL:   0005:01:00.0 VFIO group is not viable! Not all devices in IOMMU group bound to VFIO or unbound
EAL: Requested device 0005:01:00.0 cannot be used
EAL: PCI device 0005:01:00.1 on NUMA socket 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL:   0005:01:00.1 VFIO group is not viable! Not all devices in IOMMU group bound to VFIO or unbound
EAL: Requested device 0005:01:00.1 cannot be used

I looked into the IOMMU group error above and tried a few things like unbinding the non-NIC PCI devices but that didn’t help unfortunately.

$ ls /sys/kernel/iommu_groups/11/devices/
0005:00:00.0  0005:01:00.0  0005:01:00.1  141a0000.pcie
$ lspci -nk
0001:00:00.0 0604: 10de:229e (rev a1)
        Kernel driver in use: pcieport
0001:01:00.0 0280: 10ec:c822
        Subsystem: 1a3b:3751
        Kernel driver in use: vfio-pci
        Kernel modules: rtl8822ce
0005:00:00.0 0604: 10de:229a (rev a1)
        Kernel driver in use: pcieport
0005:01:00.0 0200: 8086:10fb (rev 01)
        Subsystem: 8086:0003
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbe
0005:01:00.1 0200: 8086:10fb (rev 01)
        Subsystem: 8086:0003
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbe

[kayccc]

We don’t have experience about DPDK and would see if other users can provide suggestion.