VGPU migration under VFIO. nvidia-vgpu-mgr: Error saving page in pipelined mode on 550.90.05 driver (Debian12,libvirt 10.5.0 qemu 9.1.1)

Virtual GPU Forums General Discussion
1

Good day!
I tried to migrate my VM(Debian12) on Nvidia A10 on PowerEdge R740xd.
Install drivers 550.90.05 vgpu-kvm on host kernel 6.6.28 and 550.90.07 grid in VM.
In my libvirt VM XML description I use VFIO because kernel 6.6, not mdev like in docs.
So I have acceleration in VM and nvidia-smi told me that OK.
I use virtfn22 and 598 profile.

Now I start migrate

u@a246:~$ virsh -c qemu+ssh://u@a246/system migrate d12-nfs --live --verbose --parallel --unsafe qemu+ssh://u@a247/system

Migration: [100,00 %]Error: internal error: QEMU unexpectedly closed the monitor (vm=‘d12-nfs’): Socket connected unix:/var/lib/libvirt/qemu/1-d12-nfs/monitor.sock,server=on

2024-11-27T06:54:38.654160Z qemu-system-x86_64: -device {“driver”:“vfio-pci”,“host”:“0000:af:03.2”,“id”:“hostdev0”,“bus”:“pci.7”,“addr”:“0x0”}: warning: vfio 0000:af:03.2: Could not enable error recovery for the device
2024-11-27T06:54:39.636449Z qemu-system-x86_64: error while loading state section id 78(0000:00:02.6:00.0/vfio)
2024-11-27T06:54:39.637563Z qemu-system-x86_64: load of migration failed: Input/output error

But on src side I have

nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 30303030-613a-3a66-3033-2e3200000000 GPU PCI id 00:af:03.2 config params vgpu_type_id=598
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=598
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_env_log: Successfully updated env symbols!
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): detected a VF at 0:af:3.2
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): gpu-pci-id : 0xaf00
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vgpu_type : NVS
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): Framebuffer: 0x38000000
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): Virtual Device Id: 0x2236:0x14c0
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): FRL Value: 60 FPS
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: ######## vGPU Manager Information: ########
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: Driver Version: 550.90.05
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vGPU BAR1 size 256 MB
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vGPU supported range: (0x70001, 0x140001)
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): Init frame copy engine: syncing…
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vGPU migration enabled
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vGPU manager is running in SRIOV mode.
nov 27 09:52:06 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: display_init inst: 0 successful
nov 27 09:52:18 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: ######## Guest NVIDIA Driver Information: ########
nov 27 09:52:18 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: Driver Version: 550.90.07
nov 27 09:52:18 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: vGPU version: 0x140001
nov 27 09:52:18 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vGPU license state: Unlicensed (Unrestricted)
nov 27 09:52:23 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): vGPU license state: Licensed
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_env_log: (0x0): Plugin migration stage change none → pre_copy. QEMU migration state: PRECOPY_ACTIVE
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): Start pre-copy vGPU state …
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_env_log: (0x0): Plugin migration stage change pre_copy → none. QEMU migration state: RUNNING
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: error: vmiop_log: (0x0): Error saving page in pipelined mode
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: error: vmiop_log: (0x0): Failed to copy dirty fb pages.
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: error: vmiop_log: (0x0): Failed to save FB pages marked dirty. 0x1
nov 27 09:54:37 a246 nvidia-vgpu-mgr[19722]: notice: vmiop_log: (0x0): Migration Ended
On dst side
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid 30303030-613a-3a66-3033-2e3200000000 GPU PCI id 00:af:03.2 config params vgpu_type_id=598
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=598
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_env_log: Successfully updated env symbols!
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): detected a VF at 0:af:3.2
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): gpu-pci-id : 0xaf00
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): vgpu_type : NVS
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Framebuffer: 0x38000000
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Virtual Device Id: 0x2236:0x14c0
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): FRL Value: 60 FPS
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: ######## vGPU Manager Information: #######
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: Driver Version: 550.90.05
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): vGPU BAR1 size 256 MB
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Detected ECC enabled on physical GPU.
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Guest usable FB size is reduced due to ECC.
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): This vGPU type does not support ECC.
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): vGPU supported range: (0x70001, 0x140001)
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Init frame copy engine: syncing…
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): vGPU migration enabled
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): vGPU manager is running in SRIOV mode.
nov 27 09:54:38 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: display_init inst: 0 successful
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_env_log: (0x0): Plugin migration stage change none → resume. QEMU migration state: RESUME
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Start restoring vGPU state …
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: Assertion Failed at 0x85b0fca2:2427
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: 8 frames returned by backtrace
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv009836vgpu+0x35) [0x7e1485ad3195]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv012511vgpu+0x180) [0x7e1485ae2ad0]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv012521vgpu+0x3e2) [0x7e1485b0fca2]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv004801vgpu+0x11f) [0x7e1485ae497f]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libnvidia-vgpu.so(_nv012449vgpu+0x4b) [0x7e1485a70fab]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: vgpu(+0x184c1) [0x607a0b6184c1]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7e1486281134]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_log: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x7e14863017dc]
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: notice: vmiop_log: (0x0): Migration Ended
nov 27 09:54:39 a247 nvidia-vgpu-mgr[31454]: error: vmiop_env_log: (0x0): Failed to write device buffer err: 0x1f

Is there any addinitonal iformation I can provide?
The main question is what is pipelined mode?
Thanx in advance
Bye.

2

The error “nvidia-vgpu-mgr: Error saving page in pipelined mode” could be caused by driver compatibility or configuration issues with NVIDIA vGPU, VFIO, libvirt, and QEMU. Ensure you’re using the correct driver version and check for mismatched settings or conflicts in your configuration files.

3

As for my configuration I use XML config stated in nvidia docs

And use virtual function with profile 598. Also I must say if guest doesn't use nvidia grid driver migration works. Could you please give us correct configuration?