Nvidia-drm: order-3 allocation in nv_drm_atomic_state_alloc() triggers OOM killer under memory fragmentation

Hello,

I experience frequent app killings from the OOM killer despite having free memory. I fed the dmesg log to Claude, which attributed it to memory fragmentation. I tried correcting the issue by creating a large swap file, increasing swappiness, increasing the values of the following settings in /etc/sysctl.d/99-memory.conf:

vm.min_free_kbytes = 262144
vm.watermark_scale_factor = 100
vm.compaction_proactiveness = 50

This did not solve the problem. The OOM killer keeps striking if the system has been running for some time and if I launch more apps than simply a browser and a music player.

This is Claude’s analysis:

Summary

On a desktop with plenty of reclaimable memory and free swap, the OOM killer was invoked by gnome-shell’s KMS thread because nv_drm_atomic_state_alloc() requested a physically contiguous order-3 block (32 KiB) via kmalloc(). Free memory was ~650 MB, but fragmented into 4–16 KiB blocks with no order-3 or larger blocks available in the usable zones, and compaction did not produce one in time.

Since order-3 is not a “costly” order, the page allocator invokes the OOM killer rather than failing the allocation. This means any long-running system with fragmented memory can have a process killed simply because of a routine atomic commit.

System

  • Driver version: 615.78.08
  • Kernel modules: proprietary
  • GPU: GTX 1660
  • Kernel: 7.2.15-200.fc44.x86_64 (Fedora 44)
  • Desktop: GNOME on Wayland (gnome-shell + Xwayland)
  • Motherboard: Gigabyte P55A-UD3R, ~12 GB RAM, 32 GB swap (zram + disk)
  • Uptime at the time: ~9 hours

Call trace

KMS thread invoked oom-killer: gfp_mask=0x40dc0(GFP_KERNEL|__GFP_ZERO|__GFP_COMP), order=3, oom_score_adj=100
CPU: 3 UID: 1000 PID: 26979 Comm: KMS thread Tainted: G           OE       7.2.15-200.fc44.x86_64 #1 PREEMPT(lazy)
Call Trace:
 <TASK>
 dump_stack_lvl+0x5d/0x80
 dump_header+0x43/0x1b3
 oom_kill_process.cold+0xa/0xba
 out_of_memory+0xfd/0x2f0
 __alloc_pages_slowpath.constprop.0+0xa5f/0xca0
 __alloc_frozen_pages_noprof+0x334/0x370
 alloc_pages_mpol+0xb2/0x180
 ___kmalloc_large_node+0xb9/0xf0
 __kmalloc_large_node_noprof+0x1d/0xd0
 __kmalloc_noprof+0x3fe/0x610
 nv_drm_atomic_state_alloc+0x1f/0x60 [nvidia_drm]
 drm_mode_atomic_ioctl+0x14c/0x710
 drm_ioctl_kernel+0xae/0x100
 drm_ioctl+0x2e0/0x560
 __x64_sys_ioctl+0xb9/0x100
 do_syscall_64+0xe2/0x570
 entry_SYSCALL_64_after_hwframe+0x76/0x7e
 </TASK>

Memory state at the time

Not a genuine out-of-memory condition:

free:166542 (~650 MB)   inactive_file:4250676kB   active_file:1183520kB
Free swap = 31433648kB / Total swap = 33554424kB
all_unreclaimable? no

Free lists show no order-3 or larger blocks in DMA32 or Normal:

Node 0 DMA32:  54338*4kB (ME) 35531*8kB (ME) 0*16kB 0*32kB 0*64kB ... = 501600kB
Node 0 Normal: 25030*4kB (UME) 6368*8kB (UME) 41*16kB (UM) 0*32kB 0*64kB ... = 151720kB

The kernel had already raised watermark boost in response to fragmentation (boost:111964kB in Normal, boost:39888kB in DMA32).

Victim: a Firefox content process with ~12 MB anonymous RSS, so the kill freed essentially nothing toward the request:

Out of memory: Killed process 112527 (Web Content) total-vm:2851984kB, anon-rss:12216kB, file-rss:60620kB, shmem-rss:1300kB, UID:1000 pgtables:632kB oom_score_adj:233

Analysis

__kmalloc_noprof → ___kmalloc_large_node indicates the atomic state structure allocated in nv_drm_atomic_state_alloc() is larger than 16 KiB, so it bypasses the slab caches and goes straight to the page allocator as an order-3 request. Because this runs on every atomic commit, the compositor repeatedly needs a contiguous 32 KiB block, which becomes unavailable on a long-running desktop once memory is fragmented.

Suggested fix

If the structure does not need to be physically contiguous, allocating it with kvzalloc() (and freeing with kvfree() in the corresponding atomic_state_free path) would fall back to vmalloc when no contiguous block is available, instead of triggering the OOM killer. Alternatively, reducing the structure’s size below the kmalloc slab limit, or allocating it once and reusing it, would avoid the high-order request entirely.

nvidia-bug-report.log.gz (969.5 KB)