Error of pbdma caused related user processes crashed

Hi nvidia team,
We met a pbdma error report in dmesg, which caused many user processes crashed.

We check the jetpack source code, the signature error means the PBDMA’s internal state register no longer holds 0xFACE. The possible causes, in order of likelihood for an Orin platform:

  • Memory corruption in the push buffer The GPU command data in DRAM was corrupted between the time the CPU wrote it and the time the PBDMA read it. This could be caused by an ECC error, a faulty DMA transfer, or a software bug writing invalid commands. Check dmesg for any ECC or memory controller errors near timestamp.

We want to know how could this error happen and how to fix it, thanks.

system info:
Module: Jetson AGX Orin 32G
software: Jetpack-6.0

Hi,
Please use Jetpack 6.2.2 r36.5 or 7.2 r39.2. This looks to be the PSIRT issues we fixed in later releases:
JetPack releases with Security Fixes: CVE-2025-33182 & CVE-2025-33177

Subject: Re: PBDMA Signature Error — Need Reproduction Steps and Verification Path for CVE-2025-33182 /
CVE-2025-33177

Hi,

Thank you for the response. We are running JetPack 6.0 (L4T r36.x, kernel 5.15.136) on Orin and plan to backport the
fix rather than upgrade the full BSP, as we have a production fleet.

We have already diffed the nvgpu source between our current tree and r36.5 (kernel_oot_modules_src.tbz2). The only
security-relevant change is the addition of prof_obj_lock mutex to protect the profiler_objects linked list — across
these 5 files:

  • include/nvgpu/gk20a.h — new struct nvgpu_mutex prof_obj_lock
  • os/linux/driver_common.c — nvgpu_mutex_init(&g->prof_obj_lock)
  • common/profiler/profiler.c — lock around nvgpu_list_add
  • os/linux/ioctl_dbg.c — lock around 7 list traversals/modifications
  • os/linux/ioctl_prof.c — lock around 2 nvgpu_profiler_free calls

All PBDMA code (pbdma_ga10b_fusa.c, fifo_intr_ga10b_fusa.c, hw_pbdma_ga10b.h) is byte-for-byte identical between our
version and r36.5.

We have three questions before we can confidently deploy this fix to production:

  1. How does the profiler race condition cause a PBDMA signature error?

Our error is a hardware-level PBDMA signature validation failure — bit 31 of pbdma_intr_0 (0x80000000), meaning the
PBDMA’s internal signature register no longer matches the expected 0xFACE value. This is detected by the PBDMA
hardware itself, not by software.

The r36.5 fix addresses a software race condition on the profiler_objects linked list (missing mutex). We understand
that a list corruption / use-after-free could cascade into GPU state corruption, but we would like NVIDIA to confirm
the causal chain. Specifically:

  • Does the corrupted profiler object lead to stale or invalid GPU channel/context state being written to PBDMA
    registers?
  • Or does the use-after-free corrupt memory that overlaps with GPU command buffers or GPFIFO entries, which the PBDMA
    then reads as invalid?
  • Or is the connection indirect — the race destabilizes the driver state machine, eventually leading to the PBDMA
    processing garbage commands?

Understanding the mechanism helps us assess whether this fix fully addresses our failure mode, or whether we have a
separate issue that happens to co-occur.

  1. How can we reproduce this issue on demand?

The error occurs intermittently after ~6.9 hours of uptime on our system. Our power monitoring data shows stable
rails (±1 LSB) at the time of failure, ruling out power-related causes. We need a deterministic reproduction path to
validate the fix. Some specific questions:

  • Is there a known workload pattern that triggers the race? For example, does concurrent profiler attach/detach
    (e.g., running nsys or ncu while GPU workloads are active) reliably trigger it?
  • Can we stress-test the profiler path specifically — e.g., rapid NVGPU_DBG_GPU_IOCTL_ALLOC_PROFILER / FREE_PROFILER
    from multiple threads?
  • Is there a known minimum concurrency level (number of threads, sessions) that makes the race window large enough to
    hit reliably?

Without a reproduction method, we cannot distinguish between “the fix resolved the issue” and “the issue is
intermittent and hasn’t appeared yet.”

  1. Is backporting only these 5 files sufficient, or are there dependencies we’re missing?

We confirmed the diff is limited to these 5 files within nvgpu. But we want to verify:

  • Are there any changes outside of kernel_oot_modules_src.tbz2 (e.g., in the kernel itself, device tree, or firmware)
    that are also required for the CVE fix?
  • The CVE bulletin references two CVEs (CVE-2025-33182 and CVE-2025-33177). Are both addressed by this single
    profiler mutex fix, or does one of them require changes in a different component?
  • Are there any runtime dependencies on the mutex (e.g., CONFIG_NVGPU_PROFILER must be enabled in our kernel config)?

We have already applied the 5-file patch and it builds cleanly against our kernel 5.15.136 tree. We would like
confirmation that this is the complete fix before deploying to our fleet.

Thank you for your help.

Hi,
CVE-2025-33177 is a potential memory corruption and it may be related to the pbdma error. Also other issues are found and fixed in later releases. Would suggest upgrade.

If you must stay on Jetpack 6.0 r36.3, suggest also apply the fix:

Making sure you're not a bot!
[NvGPU] slab-out-of-bounds in nvgpu_gr_config_init

Making sure you're not a bot!
[DRIVER] rcu_preempt detected stalls on CPUs/tasks with r36.5

This is specific to Orin 64GB:

Jetson Linux 36.3 | NVIDIA Developer
Additional Files
overlay_libcuda_36.3.tbz2

FYI. It may fail if you apply it to r36.3 because it is for r36.4.4:

Jetson Linux Release 36.4.4 | NVIDIA Developer
Additional Files
cuda_driver_36.4.4.tbz2

Thank you for your reply.

commit af45c11d0507e12f3132ca35b3357945f6bbb30d (tag: rel-36_eng_2025-02-28)
Author: Shashank Singh <shashsingh@nvidia.com>
Date:   Wed Jan 15 08:30:52 2025 +0000

    gpu: nvgpu: make entries check robust for kernel submit

    Check on the number of entries for kernel mode submit needs to be more
    robust to avoid overflows. The overflow can make check useless.

    Bug 5045127
    Bug 5079301

    Signed-off-by: Shashank Singh <shashsingh@nvidia.com>
    Change-Id: I2247718097bad922ac38dd7574b0fc008b1e02e4
    Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvgpu/+/3283910
    (cherry picked from commit 6ed9cc57b4ca77424da58c9a90d7a19ed6deb11c)
    Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvgpu/+/3295258
    GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>
    Reviewed-by: Bibek Basu <bbasu@nvidia.com>
commit b37344e9d7c4006c016c5aef323d7dff8642b569
Author: Chun Ng <chunn@nvidia.com>
Date:   Tue Apr 30 16:58:35 2024 -0700

    gpu: nvgpu: Fix out-of-bound access

    Make sure the index used to access gpc_tpc_physical_id_map does not
    exceed the range.

    Bug 4566775

    Change-Id: I863f4b4d06c8ace3b58b8c3fdc60f9d466846072
    Signed-off-by: Chun Ng <chunn@nvidia.com>
    Reviewed-on: https://git-master.nvidia.com/r/c/linux-nvgpu/+/3341829
    Tested-by: Aasta Lin <aastal@nvidia.com>
    Reviewed-by: Vijayakumar Subbu <vsubbu@nvidia.com>
    Reviewed-by: svcacv <svcacv@nvidia.com>
    GVS: buildbot_gerritrpt <buildbot_gerritrpt@nvidia.com>

We analyzed commits af45c11d0 and b37344e9d from r36.5 against our error (pbdma_intr_0 = 0x80000000, PBDMA unit 1, looping at ~7h uptime).

af45c11d0 fixes a u32 overflow in submit.c:813 that could corrupt RAMFC signature via DMA buffer overflow — but it requires num_entries ≈ 0xFFFFFFFF from userspace, which doesn’t happen in normal CUDA operation.

b37344e9d fixes an OOB write in gr_config.c:688 where the array is sized by logical GPC count but indexed by physical GPC ID. This corrupts kernel heap at boot, not DMA memory where the PBDMA signature lives, and our error appears hours into
runtime.

We will apply both patches, but neither has a clear causal path to our specific symptom. Are there other r36.5 commits targeting PBDMA signature corruption

Hi,
If you cannot upgrade to Jetpack 6.2.2 r36.5 or 7.2 r39.2, please apply the fix for CVE-2025-33177:
Making sure you're not a bot!