Jetson platform: Jetson Orin NX 16GB (bug is in generic nvgpu allocator code, not board-specific —
see version-scope section below for confirmation across the whole product line)
L4T / JetPack (originally found on): R36.3.0 / JetPack 6.0 GA
Kernel: 5.15.136-tegra
Confirmed also present, unpatched, in: JetPack 6.2 (L4T R36.4.3), JetPack 7.0 (L4T R38.2.0, Thor/Blackwell
generation), JetPack 7.2 (L4T R39.2.0) — see “Version scope” section below.
Summary
nvgpu_kmem_cache_create() (nvgpu/drivers/gpu/nvgpu/os/linux/kmem.c) creates a dynamically-named
kmem_cache per call using kmem_cache_create(name, size, size, 0, NULL) — flags=0, no constructor. A
cache created this way is mergeable: SLUB silently folds it into an already-existing same-sized cache
and only creates a sysfs alias symlink (/sys/kernel/slab/nvgpu-cache-<ptr>-<size>-<id>) for the
caller’s unique name, rather than a real standalone kmem_cache.
kmem_cache_destroy() on a merged/aliased cache correctly drops the shared cache’s refcount — the
underlying memory really is reused correctly, no error or warning is printed — but it never removes the
sysfs alias symlink it created. This is a long-standing Linux SLUB behavior gap (aliasing via
sysfs_slab_alias() has no corresponding “unalias” cleanup path in shutdown_cache()), not a bug in
nvgpu’s own allocator logic.
Symptom / real-world impact
Every nvgpu_kmem_cache_create() / nvgpu_kmem_cache_destroy() cycle permanently leaks one dangling
sysfs symlink + kobject (kernfs_node/dentry/inode bookkeeping — real, unreclaimable kernel memory).
In practice this is triggered by anything that briefly opens and closes a GPU/CUDA context — we hit this
via the NVIDIA-provided Jetson ffmpeg package (repo.download.nvidia.com/jetson/ffmpeg, which links
libnvbufsurftransform.so/libnvbufsurface.so): every single ffmpeg/ffprobe invocation —
including a bare ffmpeg -version with no encode/decode at all — creates two nvgpu_buddy_allocator
instances (a “gk20a_as_N” / “gk20a_as_N-sys” pair) at GPU-context setup, each backed by an
nvgpu_kmem_cache for struct nvgpu_buddy (128 bytes). Both create and destroy for these allocators
fire correctly on process exit — but the sysfs alias entries are never cleaned up regardless.
On an appliance that invokes ffmpeg/ffprobe frequently (e.g. our case: ~95 invocations/min for video
thumbnail/frame extraction), this accumulates roughly 4 leaked nvgpu-cache-* entries per invocation,
~134 MB of unreclaimable (“SUnreclaim”) memory per day, leading to OOM after roughly 50 days of uptime.
(We were already aware of, and had separately mitigated, this specific ffmpeg package via a userspace
workaround — replacing it with a stock, non-NVIDIA ffmpeg build — before tracing the leak down to this
kernel-side root cause. The kernel-side fix below eliminates the leak regardless of which userspace
component creates/destroys the GPU context, since the bug is generic to any nvgpu_kmem_cache_create()
caller, not specific to ffmpeg.)
Reproduction
# baseline
ls /sys/kernel/slab/ | grep -c '^nvgpu-cache-'
# trigger 100 GPU-context create/destroy cycles (no decode/encode needed)
for i in $(seq 1 100); do ffmpeg -version >/dev/null 2>&1; done
# recount
ls /sys/kernel/slab/ | grep -c '^nvgpu-cache-'
Observed on our system: baseline 51 → after 100 invocations 451 (delta 400, i.e. exactly 4 per
invocation). This is stable/reproducible; also reproduced at 200x and 1000x invocation counts, and with a
real hardware-accelerated video decode workload (not just process startup), all showing the same ~4
leaked entries per GPU-context create/destroy cycle.
Root cause detail
Every nvgpu-cache-* entry we inspected under /sys/kernel/slab/ was a symlink, not a directory,
all pointing at the same underlying SLUB-merged cache:
lrwxrwxrwx ... nvgpu-cache-0x00000000dd65b75c-128-1 -> :0000128
lrwxrwxrwx ... nvgpu-cache-0x00000000dd65b75c-128-10 -> :0000128
Confirmed by instrumenting nvgpu_buddy_allocator_init()/nvgpu_buddy_allocator_destroy() (and the
page/bitmap allocator equivalents) with temporary pr_info() markers: for every ffmpeg invocation, both
_init and _destroy fire correctly for all 4 allocator instances created — the create/destroy pairing
in the allocator layer is not broken. The leak is entirely explained by the SLUB alias-symlink-never-
removed behavior described above.
Verified this mechanism directly against mm/slub.c / mm/slab_common.c in the Jetson Linux 5.15.136-tegra
kernel source (kernel_src.tbz2):
// mm/slub.c
static int sysfs_slab_alias(struct kmem_cache *s, const char *name)
{
if (slab_state == FULL) {
/* If we have a leftover link then remove it. */
sysfs_remove_link(&slab_kset->kobj, name);
return sysfs_create_link(&slab_kset->kobj, &s->kobj, name);
}
...
}
// mm/slab_common.c
void kmem_cache_destroy(struct kmem_cache *s)
{
...
s->refcount--;
if (s->refcount)
goto out_unlock; /* <-- always taken for a widely-shared merged cache */
err = shutdown_cache(s); /* <-- never reached in the merged case */
...
}
sysfs_remove_link(&slab_kset->kobj, name) for our alias is only ever invoked the next time
sysfs_slab_alias() happens to be called again with the exact same name string — which never happens
here, since nvgpu_kmem_cache_create()'s name embeds a monotonically-incrementing atomic counter
(atomic_inc_return(&kmem_cache_id)), so no two calls ever produce the same name. The alias is therefore
permanent. This is a direct, verifiable code-level explanation, not an inference from behavior alone.
Version scope — confirmed present in JetPack 6.2, 7.0, and 7.2 too, unpatched
Pulled the public GPL sources for JetPack 6.2 (L4T R36.4.3), JetPack 7.0 (L4T R38.2.0 — the new Jetson
Thor/Blackwell generation, kernel 6.8, SBSA/Ubuntu 24.04) and JetPack 7.2 (L4T R39.2.0), and diffed both
nvgpu_kmem_cache_create() (os/linux/kmem.c) and nvgpu_page_allocator_destroy()
(common/mm/allocators/page_allocator.c, a second, related cleanup gap noticed along the way — see patch)
against the R36.3.0 version this report is based on.
Both are byte-for-byte unchanged (bug still present) in all three later releases, including the
brand-new Thor-generation source tree (nvgpu was split into its own kernel_nvgpu_src.tbz2 archive there,
but the function bodies are identical apart from a cosmetic cache-name format change unrelated to this
bug). This is not specific to R36.3.0, this AI Key unit, or the Orin NX — it affects the entire nvgpu
allocator subsystem across at least the R36.x and R38.x/R39.x L4T generations, i.e. essentially the whole
current Jetson product line.
Public source URLs used for the comparison (all verified reachable):
- JetPack 6.2:
https://developer.download.nvidia.com/embedded/L4T/r36_Release_v4.3/sources/public_sources.tbz2 - JetPack 7.0:
https://developer.download.nvidia.com/embedded/L4T/r38_Release_v2.0/sources/public_sources.tbz2 - JetPack 7.2:
https://developer.download.nvidia.com/embedded/L4T/r39_Release_v2.0/sources/public_sources.tbz2
Fix
Pass a non-NULL (no-op) constructor to kmem_cache_create(). SLUB’s find_mergeable() unconditionally
excludes any cache with a non-NULL ctor from merging — this is checked before any size/alignment/flags
criteria — so every nvgpu_kmem_cache_create() call now creates a genuine, standalone cache whose sysfs
entry is actually removed by kmem_cache_destroy().
--- a/drivers/gpu/nvgpu/os/linux/kmem.c
+++ b/drivers/gpu/nvgpu/os/linux/kmem.c
@@ -613,6 +613,21 @@
}
#endif /* CONFIG_NVGPU_TRACK_MEM_USAGE */
+/*
+ * A plain kmem_cache_create() with no ctor and no anti-merge flags is
+ * mergeable: SLUB silently folds it into an existing same-sized cache and
+ * only creates a sysfs alias symlink for our unique name. kmem_cache_destroy()
+ * on a merged cache never removes that alias symlink (a long-standing SLUB
+ * limitation), so every create/destroy cycle permanently leaks one
+ * /sys/kernel/slab/nvgpu-cache-* entry even though the underlying memory is
+ * reused correctly. A non-NULL ctor unconditionally disables merging, so the
+ * cache we create here is a real, standalone cache whose sysfs entry is
+ * actually removed on destroy.
+ */
+static void nvgpu_kmem_cache_ctor(void *obj)
+{
+}
+
struct nvgpu_kmem_cache *nvgpu_kmem_cache_create(struct gk20a *g, size_t size)
{
struct nvgpu_kmem_cache *cache =
@@ -627,7 +642,7 @@
"nvgpu-cache-0x%p-%d-%d", g, (int)size,
atomic_inc_return(&kmem_cache_id));
cache->cache = kmem_cache_create(cache->name,
- size, size, 0, NULL);
+ size, size, 0, nvgpu_kmem_cache_ctor);
if (!cache->cache) {
nvgpu_kfree(g, cache);
return NULL;
(Full patch file with commit-message-style header attached below.)
Verification
Built against the R36.3.0 public OOT module sources (kernel_oot_modules_src.tbz2), deployed on-device
by replacing /usr/lib/modules/5.15.136-tegra/updates/nvgpu.ko and rebooting (this module cannot be
safely hot-reloaded live — unrelated pre-existing issue, its .fini/exit handling isn’t designed for
runtime unload/reload — a plain replace-on-disk + reboot is required).
| Test | Before fix | After fix |
|---|---|---|
100x ffmpeg -version |
delta = 400 | delta = 0 |
200x ffmpeg -version |
— | delta = 0 |
1000x ffmpeg -version |
— | delta = 0 |
20x real HW video decode (h264_nvv4l2dec) |
— | delta = 0 |
No functional regressions observed: all GPU-dependent services (CUDA-based inference workload, camera
capture, hardware video decode) continued operating normally after the fix, across multiple reboots and
several hours of production load.
Would appreciate this being picked up for inclusion in a future L4T release across all affected branches
(R36.x, R38.x, R39.x — see version scope above), since the underlying nvgpu_kmem_cache_create()/
_destroy() pattern is used broadly across the nvgpu allocator code (buddy/page/bitmap allocators all use
it) and confirmed unpatched on the current Thor-generation source line too.