BAR Allocation Failed & IOMMU Conflicts: Dual GPU (RTX 5060 + 4060) on Ryzen 5800X/B550 - "No Space" Errors

Accelerated Computing GPU-Accelerated Libraries
1

Hi everyone,

I am hitting a wall trying to configure a dual GPU setup on a B550/Ryzen platform running Ubuntu. I am facing a persistent Memory Mapped I/O (MMIO) resource allocation issue where the kernel cannot assign BAR addresses for the second GPU, leading to can't assign; no space errors.

Hardware Specs:

  • CPU: AMD Ryzen 7 5800X

  • Mobo: B550 Chipset [INSERT YOUR MOTHERBOARD MODEL HERE]

  • GPU 1: NVIDIA RTX 4060

  • GPU 2: NVIDIA RTX 5060 (New gen)

  • PSU: 850W

  • OS: Ubuntu (Kernel 6.x generic)

  • Driver: Manual install of NVIDIA-Linux-x86_64-580.126.09.run (Required for 5060 support)

The Problem: When booting with default BIOS settings, only one GPU is enumerated correctly. The second GPU fails with BAR allocation errors in dmesg. When I attempt to fix this by enabling “Above 4G Decoding” in BIOS, the system fails to mount the NVMe drive or panics during boot unless I use specific kernel parameters, and even then, I lose the proprietary driver binding.

Logs & Symptoms:

1. The BAR Error (dmesg output): Without 4G decoding, the kernel clearly sees the device but cannot map the memory:

Bash

[ 0.666193] pci 0000:07:00.0: BAR 1 [mem size 0x10000000 64bit pref]: can't assign; no space
[ 0.666195] pci 0000:07:00.0: BAR 1 [mem size 0x10000000 64bit pref]: failed to assign
[ 0.666196] pci 0000:07:00.0: VF BAR 2 [mem size 0x10000000 64bit pref]: can't assign; no space
NVRM: BAR1 is 0M @ 0x0 (PCI:0000:04:00.0)

2. The IOMMU/Disk Conflict: If I Enable Above 4G Decoding and Disable CSM (UEFI only), the boot process hangs with filesystem errors, losing access to the NVMe block devices:

Bash

/var/lib/snapd/snaps/firefox_2356.snap: Can't lookup blockdev
[FAILED] Failed to mount Mount unit for firefox.
[FAILED] Failed to mount Mount unit for snapd.

What I have tried so far:

  1. BIOS Settings:

    • Above 4G Decoding: Enabled

    • Re-Size BAR: Auto/Enabled

    • CSM: Disabled

  2. Kernel Parameters:

    • Tried pci=realloc: Did not resolve the no space error.

    • Tried iommu=soft: This allows the system to boot with “Above 4G Decoding” enabled (bypassing the blockdev error), BUT nvidia-smi fails to detect devices, or the system falls back to llvmpipe (software rendering) with a zoomed-in resolution.

  3. Display Manager:

    • Disabled Wayland in /etc/gdm3/custom.conf (WaylandEnable=false).

  4. Driver Installation:

    • Purged all apt nvidia packages.

    • Running the .run file for driver 580.126.09.

    • Installation completes successfully in Recovery Mode (Root Shell), but upon rebooting with iommu=soft, the driver doesn’t seem to bind correctly to the cards.

The Question: How can I properly expose the BAR address space for both cards on a B550/Ryzen platform without breaking the NVMe/IOMMU controller? Is there a specific combination of GRUB parameters (e.g., pci=noalloc, iommu=pt, amd_iommu=on) that allows 4G decoding to coexist with a functional NVIDIA driver on this chipset?

Any insights on how to escape this loop would be appreciated.

2

Update: Tested all suggestions - NVMe/Blockdev Error persists

Thank you all for the suggestions. I have methodically tried all three proposed solutions, but unfortunately, they all result in the exact same boot failure. It seems that enabling “Above 4G Decoding” breaks the NVMe controller’s ability to mount filesystems, regardless of the IOMMU parameters or BAR sizing.

Here is exactly what I tested:

Attempt 1 (iommu=off):

BIOS: Above 4G [Enabled], CSM [Disabled].

GRUB: iommu=off

Result: Boot failure. “Can’t lookup blockdev”.

Attempt 2 (pci=big_root_window):

BIOS: Above 4G [Enabled], CSM [Disabled].

GRUB: iommu=soft pci=big_root_window

Result: Boot failure. “Can’t lookup blockdev”.

Attempt 3 (Re-Size BAR Disabled):

BIOS: Above 4G [Enabled], Re-Size BAR [Disabled], CSM [Disabled].

GRUB: iommu=soft

Result: Boot failure. “Can’t lookup blockdev”.

The Error Log (Identical in all 3 cases):

The system sees the drive initially (/dev/nvme0n1p2: clean…), but immediately loses access to block devices required for mounting snaps and the desktop environment:

/dev/nvme0n1p2: clean, 2954993/30498816 files, 87996494/121965088 blocks

[ 3.603861] /var/lib/snapd/snaps/firefox_2356.snap: Can’t lookup blockdev

[ 3.603916] /var/lib/snapd/snaps/gnome-3-38-2004_119.snap: Can’t lookup blockdev

[FAILED] Failed to mount Mount unit for firefox, revision 2356.

[FAILED] Failed to mount Mount unit for snapd, revision 18357.

[FAILED] Failed to start Process error reports when automatic reporting is enabled.

It appears impossible to keep the NVMe drive functional while “Above 4G Decoding” is Enabled on this B550/Ryzen setup with the current kernel parameters.

Is there any other specific IOMMU parameter (like amd_iommu=fullflush or iommu=pt) that specifically targets B550 NVMe conflicts? Or is moving the NVMe drive to a Chipset-controlled M.2 slot (instead of the CPU-controlled one) a potential workaround?

3

I want to share the solution that finally worked for me to get a dual GPU setup (RTX 4060 + RTX 5060) running on Ubuntu with a B550 Motherboard.

The Symptoms:

  1. Boot Loop/NVMe Error: Enabling “Above 4G Decoding” (required for the GPUs) caused Can't lookup blockdev errors on my NVMe drive.

  2. Missing GPU: After fixing boot, nvidia-smi would only show the RTX 4060. The RTX 5060 was visible in lspci but failed to initialize in the driver.

  3. Logs: dmesg showed errors stating: “NVRM: installed in this system requires use of the NVIDIA open kernel modules.”

THE SOLUTION:

Part 1: Hardware/BIOS (Fixing the NVMe crash) The B550 chipset had a PCIe lane conflict.

  • Fix: I physically removed my SATA SSDs. This freed up enough resources/lanes.

  • BIOS Settings: Above 4G Decoding [Enabled], Re-Size BAR [Enabled], CSM [Disabled].

Part 2: The Driver (Fixing the Missing RTX 5060) The latest 580.xx driver gave me issues. The fix was using the 570.xx driver and forcing Open Kernel Modules. The RTX 50-series (Blackwell) appears to require the Open modules to initialize correctly on Linux.

Steps:

  1. Clean Install: I completely purged the old drivers (--uninstall and apt purge nvidia*).

  2. Driver Version: I used NVIDIA-Linux-x86_64-570.211.01.run.

  3. The Command: You must install it with this specific flag:

    Bash

    sudo ./NVIDIA-Linux-x86_64-570.211.01.run --kernel-module-type=open

  4. Installation Prompts:

    • 32-bit libraries: Yes

    • Register with DKMS: Yes

    • Update Xconfig: Yes

Result: Both cards are now fully recognized and working.

Plaintext

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.211.01             Driver Version: 570.211.01      CUDA Version: 12.8    |
|-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA GeForce RTX 4060        Off |   00000000:04:00.0  On |                  N/A |
|   1  NVIDIA GeForce RTX 5060        Off |   00000000:07:00.0 Off |                  N/A |
+-----------------------------------------+------------------------+----------------------+

I hope this helps anyone else struggling with the new 50-series on Linux!