Problem:
ZONE NORMAL (low memory) is exhausted when VRAM is depleted and further frame updates/textures are used/created. Eventually the out-of-memory (OOM) killer will kick-off and start it’s logic to free up ZONE NORMAL memory by killing applications.
Environment:
HP Z600 Workstations with dual FX580 (512M) video cards driving three monitors with three separate X sessions (one for each) running in-house application. Running 32bit OS kernel version 2.6.32-431.11.2.el6.i686 and drivers tested include 331.20, 337.25, 340.24 all exhibit the same problem.
I do not think the application’s own heap allocations have much of an impact on this issue.
Most of those would be in the individual “depictable” classes that actually load the data inside our source/applications DisplayPanel.{c|h] and DisplayMgr.[c|h] and it simply runs “XCreatePixmap”. If need be the source of the in-house applications can be provided and/or the specific XCreatePixmap calls.
Note that we do not experience this issue on older kernel version 2.6.18-194.17.1.el5 which is running 195.36.24 driver version. Installing driver versions downstream of 256.* results in various “undefined symbols” in the libglx and nvidia_drv shared objects, thus downgrading is not optimal/option as well as the “supported” version for the kernel version and cards we have is 331.20 regardless.
Troubleshooting:
Loading up application and monitoring the slab table, low memory, and VRAM via:
slabtop -n 1 -S c
watch -n 0.25 free -lm
watch -n 0.25 nvidia-smi -q ‘2>&1’ ‘|’ grep -A3 “‘Memory Usage’”
You will notice that when the VRAM for a specific GPU driving the rendering is depleted, the nvidia_pte_t (page table entry structures) appear to start being evicted and calling __get_free_pages with GFP kernel calls inside the driver via nv_alloc_system_pages( ). Eventually the low memory is exhausted and the OOM killer is evoked to free up pages, if it cannot free up enough reclaimable slabs then it will start killing applications based on their oom_adj/score.
We’ve already tuned the VM and kernel to the point where we’ve freed up as much low memory as is possible give the limitations of the 32bit architecture (about 780M free of the allotted max 896M). We still will notice that when this occurs, the nvidia_pte_t slab entries/cache will grow significantly, and was the initial piece of information that lead us to investigating the nvidia driver.
Previous versions appear to not attempt to allocate system pages under the same circumstances, 195.36.24 for example … however I cannot fully attest to that as I don’t have a test environment to run through that on just yet.
Request:
Is there any way to mediate this, backport or build in a way to prevent this situation from occurring that doesn’t introduce additional risk to the system. We could change the nv.c source to mediate the situation, but it doesn’t account for what the application may then do if this occurs under these situations. Unfortunately moving to a 64bit kernel is not possible at this time due to programmatic issues, and if there is a way to remedy the missing symbols perhaps (need to still look into that) then we could use the older driver.
Attached is the bug-report, there are three, one when the system is on the verge but not evoked the OOM killer yet, that’s named firstLoad. The CRASH named report was caught just before the OOM killer was evoked, and I believe the regular named log is after the crash.
Additional Info may be uploaded (slabinfo, kernel ring buffer, etc…) to show the situation but the nvidia-smi will show the issue hopefully, and the references in this post hopefully will job someone?
Thanks for your time and attention!!
[This file was removed because it was flagged as potentially malicious] (98.4 KB)
[This file was removed because it was flagged as potentially malicious] (134 KB)
nvidia-bug-report.log.gz (142 KB)