Hello,
I experience frequent app killings from the OOM killer despite having free memory. I fed the dmesg log to Claude, which attributed it to memory fragmentation. I tried correcting the issue by creating a large swap file, increasing swappiness, increasing the values of the following settings in /etc/sysctl.d/99-memory.conf:
vm.min_free_kbytes = 262144
vm.watermark_scale_factor = 100
vm.compaction_proactiveness = 50
This did not solve the problem. The OOM killer keeps striking if the system has been running for some time and if I launch more apps than simply a browser and a music player.
This is Claude’s analysis:
Summary
On a desktop with plenty of reclaimable memory and free swap, the OOM killer was invoked by gnome-shell’s KMS thread because nv_drm_atomic_state_alloc() requested a physically contiguous order-3 block (32 KiB) via kmalloc(). Free memory was ~650 MB, but fragmented into 4–16 KiB blocks with no order-3 or larger blocks available in the usable zones, and compaction did not produce one in time.
Since order-3 is not a “costly” order, the page allocator invokes the OOM killer rather than failing the allocation. This means any long-running system with fragmented memory can have a process killed simply because of a routine atomic commit.
System
- Driver version: 615.78.08
- Kernel modules: proprietary
- GPU: GTX 1660
- Kernel: 7.2.15-200.fc44.x86_64 (Fedora 44)
- Desktop: GNOME on Wayland (gnome-shell + Xwayland)
- Motherboard: Gigabyte P55A-UD3R, ~12 GB RAM, 32 GB swap (zram + disk)
- Uptime at the time: ~9 hours
Call trace
KMS thread invoked oom-killer: gfp_mask=0x40dc0(GFP_KERNEL|__GFP_ZERO|__GFP_COMP), order=3, oom_score_adj=100
CPU: 3 UID: 1000 PID: 26979 Comm: KMS thread Tainted: G OE 7.2.15-200.fc44.x86_64 #1 PREEMPT(lazy)
Call Trace:
<TASK>
dump_stack_lvl+0x5d/0x80
dump_header+0x43/0x1b3
oom_kill_process.cold+0xa/0xba
out_of_memory+0xfd/0x2f0
__alloc_pages_slowpath.constprop.0+0xa5f/0xca0
__alloc_frozen_pages_noprof+0x334/0x370
alloc_pages_mpol+0xb2/0x180
___kmalloc_large_node+0xb9/0xf0
__kmalloc_large_node_noprof+0x1d/0xd0
__kmalloc_noprof+0x3fe/0x610
nv_drm_atomic_state_alloc+0x1f/0x60 [nvidia_drm]
drm_mode_atomic_ioctl+0x14c/0x710
drm_ioctl_kernel+0xae/0x100
drm_ioctl+0x2e0/0x560
__x64_sys_ioctl+0xb9/0x100
do_syscall_64+0xe2/0x570
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Memory state at the time
Not a genuine out-of-memory condition:
free:166542 (~650 MB) inactive_file:4250676kB active_file:1183520kB
Free swap = 31433648kB / Total swap = 33554424kB
all_unreclaimable? no
Free lists show no order-3 or larger blocks in DMA32 or Normal:
Node 0 DMA32: 54338*4kB (ME) 35531*8kB (ME) 0*16kB 0*32kB 0*64kB ... = 501600kB
Node 0 Normal: 25030*4kB (UME) 6368*8kB (UME) 41*16kB (UM) 0*32kB 0*64kB ... = 151720kB
The kernel had already raised watermark boost in response to fragmentation (boost:111964kB in Normal, boost:39888kB in DMA32).
Victim: a Firefox content process with ~12 MB anonymous RSS, so the kill freed essentially nothing toward the request:
Out of memory: Killed process 112527 (Web Content) total-vm:2851984kB, anon-rss:12216kB, file-rss:60620kB, shmem-rss:1300kB, UID:1000 pgtables:632kB oom_score_adj:233
Analysis
__kmalloc_noprof → ___kmalloc_large_node indicates the atomic state structure allocated in nv_drm_atomic_state_alloc() is larger than 16 KiB, so it bypasses the slab caches and goes straight to the page allocator as an order-3 request. Because this runs on every atomic commit, the compositor repeatedly needs a contiguous 32 KiB block, which becomes unavailable on a long-running desktop once memory is fragmented.
Suggested fix
If the structure does not need to be physically contiguous, allocating it with kvzalloc() (and freeing with kvfree() in the corresponding atomic_state_free path) would fall back to vmalloc when no contiguous block is available, instead of triggering the OOM killer. Alternatively, reducing the structure’s size below the kmalloc slab limit, or allocating it once and reusing it, would avoid the high-order request entirely.
nvidia-bug-report.log.gz (969.5 KB)