IOMMU Group is not viable

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
, ,
1

Hi, I am encountered a issue when i try to attach my pcie device to vfio-pci driver in Jetson Xavier NX development kit.

labuser@lab69:~$ lspci -v
0004:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad1 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 24
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 00001000-00001fff [size=4K]
Memory behind bridge: 40000000-400fffff [size=1M]
Prefetchable memory behind bridge: [disabled]
Capabilities:
Kernel driver in use: pcieport

0004:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter
Subsystem: AzureWave RTL8822CE 802.11ac PCIe Wireless Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 270
I/O ports at 1000 [size=256]
Memory at 1740000000 (64-bit, non-prefetchable) [size=64K]
Capabilities:
Kernel driver in use: rtl88x2ce
Kernel modules: rtl8822ce

0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 26
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: 40000000-400fffff [size=1M]
Prefetchable memory behind bridge: 0000001c00000000-0000001c000fffff [size=1M]
Capabilities:
Kernel driver in use: pcieport

0005:01:00.0 Processing accelerators: Device 1e58:0001 (rev 01)
Subsystem: Allied Telesis, Inc (Wrong ID) Device 0010
Flags: fast devsel, IRQ 255
Memory at 1f40080000 (64-bit, non-prefetchable) [disabled] [size=16K]
Memory at 1f40000000 (64-bit, non-prefetchable) [disabled] [size=512K]
Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=1M]
Capabilities:

I want to attach pcie device with 1e58:0001 id to vfio-pci driver.
I implemented by following steps:

labuser@lab69:~$ sudo -s
root@lab69:/home/labuser# modprobe vfio-pci
root@lab69:/home/labuser# echo 1e58 0001 > /sys/bus/pci/drivers/vfio-pci/new_id
root@lab69:/home/labuser# lspci -v
0004:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad1 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 24
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: 00001000-00001fff [size=4K]
Memory behind bridge: 40000000-400fffff [size=1M]
Prefetchable memory behind bridge: [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [168] Physical Layer 16.0 GT/s <?> Capabilities: [18c] Lane Margining at the Receiver <?>
Capabilities: [1ac] L1 PM Substates
Capabilities: [1bc] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2bc] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [2f4] Data Link Feature <?> Capabilities: [300] Precision Time Measurement Capabilities: [30c] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport

0004:01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter
Subsystem: AzureWave RTL8822CE 802.11ac PCIe Wireless Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 270
I/O ports at 1000 [size=256]
Memory at 1740000000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Device Serial Number 00-e0-4c-ff-fe-c8-22-01
Capabilities: [158] Latency Tolerance Reporting
Capabilities: [160] L1 PM Substates
Kernel driver in use: rtl88x2ce
Kernel modules: rtl8822ce

0005:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 26
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: [disabled]
Memory behind bridge: 40000000-400fffff [size=1M]
Prefetchable memory behind bridge: 0000001c00000000-0000001c000fffff [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [168] Physical Layer 16.0 GT/s <?> Capabilities: [190] Lane Margining at the Receiver <?>
Capabilities: [1c0] L1 PM Substates
Capabilities: [1d0] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [2d0] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
Capabilities: [308] Data Link Feature <?> Capabilities: [314] Precision Time Measurement Capabilities: [320] Vendor Specific Information: ID=0004 Rev=1 Len=054 <?>
Kernel driver in use: pcieport

0005:01:00.0 Processing accelerators: Device 1e58:0001 (rev 01)
Subsystem: Allied Telesis, Inc (Wrong ID) Device 0010
Flags: fast devsel, IRQ 26
Memory at 1f40080000 (64-bit, non-prefetchable) [disabled] [size=16K]
Memory at 1f40000000 (64-bit, non-prefetchable) [disabled] [size=512K]
Memory at 1c00000000 (64-bit, prefetchable) [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] Secondary PCI Express
Capabilities: [168] Latency Tolerance Reporting
Capabilities: [170] L1 PM Substates
Capabilities: [180] Dynamic Power Allocation <?>
Kernel driver in use: vfio-pci

I attached my device to vfio-pci driver

root@lab69:/home/labuser/usha/vfio_bins# ./a.out
Container opened
open IOMMU group
ERROR: group not viable 0

This is the error i am facing. I checked iommu groups i found 2 pci bridges in that folder.

root@lab69:/home/labuser/usha/vfio_bins# ls /sys/kernel/iommu_groups/7/devices/
0005:00:00.0 0005:01:00.0 141a0000.pcie

What should i do now? Do i need to isolate my device?

3

I noticed the same issue as well, but don’t know why Orin SMMU groups all pcie(PF and VF) in the same IOMMU group, maybe Orin doesn’t support PCIe SR-IOV

4

Thanks.

Could u please suggest any other way to read pcie device?

6

Hi,
We don’t have much experience about using the driver, but would suggest physically separate the devices into different groups for a try.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.