Platform
- Module: Jetson Orin NX 16 GB, P3767‑0000 (part 699‑13767‑0000‑301);
- Carrier: Connect Tech Boson (seen on both NGX007 and NGX020 variants with two different series of Jetson Orins);
- BSP: L4T r36.4.3 / JetPack 6.2, kernel 6.12.23, nvgpu built from L4T r36.4.3 kernel_oot_modules_src.tbz2;
- GPU floorsweep (as fused on these modules): gpc_fs_mask=2 → GPC0 disabled, GPC1 active; tpc_fs_mask=0xf0 → TPC0–3 active; fbp_fs_mask=0x1. GPU ECC enabled.
Summary
Intermittent kernel SLUB heap corruption (kmalloc-8 freelist) originating in nvgpu GPU power‑on, specifically the GR / per‑TPC‑ECC init. It is a per‑boot lottery: some boots are fine, others panic with a wild‑pointer Oops (translation fault on a non‑canonical address, ESR=0x96000004). On a corrupted boot the fault may surface immediately in the power‑on path, or later in any unrelated allocator (nvmap, USB audio, tegra_drm) that next uses the poisoned cache. Every module of this SKU is affected (same floorsweep); KASLR decides whether a given boot is fatal.
Corruptor (caught with slub_debug=F)
BUG kmalloc-8: Freechain corrupt (CPU5, Comm: nvpmodel)
kfree ← nvgpu_kfree_impl
← gv100_gr_config_init_sm_id_table+0x580 [nvgpu]
← nvgpu_gr_init_sm_id_early_config [nvgpu]
← nvgpu_gr_init_support ← nvgpu_finalize_poweron
← gk20a_pm_finalize_poweron ← gk20a_pm_runtime_resume
← (sysfs write to /sys/devices/gpu.0/power/control by nvpmodel)
BUG kmalloc-8: Wrong object count. Counter is 149 but counted were 263
Oops: pc = nvgpu_ecc_counter_init_per_tpc+0x218/0x6c0 [nvgpu]
← ga10b_gr_gpc_tpc_ecc_init ← nvgpu_gr_init_support ← nvgpu_finalize_poweron
The 8‑byte (kmalloc-8) overflow happens while nvgpu builds per‑GPC/per‑TPC structures during GR init. Our strong suspicion is a physical‑vs‑logical GPC index / floorsweep handling issue specific to the GPC0‑disabled, GPC1‑active layout (an array sized for the active count but indexed by physical GPC id, or similar), writing one entry past the allocation.
Vanilla “victim” panics (no slub debug) - same fault class, different victims
Oops: __kmalloc_noprof ← nvmap_altalloc ← nvmap_alloc_handle ← nvmap_ioctl_alloc (Comm: argus_thread)
Oops: __kmalloc_noprof ← nvmap_altalloc ← __nvmap_sg_table ← nvmap_dmabuf_map_dma_buf
← tegra_gem_prime_import [tegra_drm] ← drm_prime_fd_to_handle_ioctl
Oops: __kmalloc_cache_noprof ← usb_control_msg ← snd_usb_init_sample_rate
All are ESR=0x96000004 faults dereferencing a corrupted SLUB free pointer — i.e. downstream victims of the same poisoned kmalloc-8 freelist, not independent bugs.
Trigger
The GPU is driven through gk20a_pm_finalize_poweron whenever it powers on:
- at boot, when nvpmodel applies the default power model (MAXN_SUPER) and writes on to /sys/devices/gpu.0/power/control, and
- at runtime, when a GPU client (NVENC encoder, CUDA, nvmedia) wakes the GPU.
Disabling rail‑gating / keeping the GPU powered does not help, the GR/ECC init still runs.
Reproduction
1. Orin NX 16 GB (this floorsweep), L4T r36.4.3, GPU ECC enabled;
2. Boot + exercise the GPU (e.g. start an NVENC encode / camera stream) repeatedly across power cycles;
3. A fraction of boots panic with the above signature. To make it deterministic/attributable, boot with kernel cmdline slub_debug=F, the slab consistency check then trips exactly at gv100_gr_config_init_sm_id_table / nvgpu_ecc_counter_init_per_tpc.
Working workaround (and why we can not keep it)
Booting with slub_debug=FZ fully stops the panics: the Z redzone pads kmalloc-8 so the overflow lands in guard bytes instead of the next object’s free pointer. Verified stable across many power cycles.
However, slub_debug is a debug feature and is too heavy for our product, this is a real‑time remote‑controlled car doing dual‑camera low‑latency WebRTC streaming. Forcing all slab caches onto the debug slow‑path adds allocation latency/jitter and per‑object memory overhead that we cannot ship long‑term. We need a proper driver fix so we can run without slub_debug.
Questions
1. Is this a known nvgpu bug for GPC0‑floorswept Orin NX modules (GPC0 off / GPC1 active) with GPU ECC enabled? Is there a fix in a newer L4T (36.4.4+, 36.5, JetPack 6.x)?
2. Can you confirm the suspected floorsweep/GPC‑index handling issue in gv100_gr_config_init_sm_id_table / nvgpu_ecc_counter_init_per_tpc (per‑TPC kmalloc-8 sizing vs indexing)?
3. Any officially‑recommended interim mitigation lighter than slub_debug?
I will attach the logs with two different BOSONs (NGX007 and NGX020).
NGX020.log (24.5 KB)
NGX007.log (11.9 KB)