Does this is p2p bug?

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

I found a Suspected bug. Please help me analyze it：

// nvdia-p2p.c
int nvidia_p2p_get_pages(u64 vaddr, u64 size,
		struct nvidia_p2p_page_table **page_table,
		void (*free_callback)(void *data), void *data)
{
     .......
     *page_table = kzalloc(sizeof(**page_table), GFP_KERNEL);
    ......
}

but at nvidia-p2p.c，no where show free this struct page_table , At least I didn’t notice。
If this struct(typedef struct nvidia_p2p_page_table) need caller to free this, I think this is a bit unreasonable。
caller don’t know nvidia_p2p_get_pages alloc a memory。
I wonder if there are other places to release, in places I haven’t seen？

5

Hi,

Thanks for reporting this to us.
Could you share which JetPack version you used first?

Thanks.

6

I am using Jetson R36.4.4. I checked the latest Jetson 7 BSP, and its nvidia-p2p.c in P2P is no different from that in R36.4.4.

Now I have found that in the demo for RDMA, GitHub - NVIDIA/jetson-rdma-picoevb: Minimal HW-based demo of GPUDirect RDMA on NVIDIA Jetson AGX Xavier running L4T · GitHub, jetson-rdma-picoevb/kernel-module/picoevb-rdma.c at master · NVIDIA/jetson-rdma-picoevb · GitHub, the following is registered:

static void pevb_p2p_free_callback(void *data)
{
	struct pevb_cuda_surface *cusurf = data;
	struct pevb_file *pevb_file = cusurf->pevb_file;

	mutex_lock(&pevb_file->lock);
	if (cusurf->handle >= 0) {
		idr_remove(&pevb_file->cuda_surfaces, cusurf->handle);
		cusurf->handle = -1;
	}
	mutex_unlock(&pevb_file->lock);

	nvidia_p2p_free_page_table(cusurf->page_table);
#ifndef NV_BUILD_DGPU
	kfree(cusurf->page_table);
#endif
	kfree(cusurf);
}
#endif

Here, the following is freed:

struct pevb_cuda_surface {
	struct pevb_file		*pevb_file;
	u64				va;
	u64				offset;
	u64				len;
	int				handle;
	struct nvidia_p2p_page_table	*page_table;
};
#endif

It is very clear that this frees struct nvidia_p2p_page_table *page_table;. However, in the MMU notifier system:

nvidia_p2p_mn_invl_range_start → nvidia_p2p_mn_invl_range_start_legacy:

static void nvidia_p2p_mn_invl_range_start_legacy(struct mmu_notifier *mn,
	struct mm_struct *mm, unsigned long start, unsigned long end)
{
	struct nvidia_p2p_page_table *page_table = container_of(mn,
						struct  nvidia_p2p_page_table,
						mn);

	u64 vaddr = 0;
	u64 size = 0;

	vaddr = page_table->vaddr;
	size = page_table->size;

	if (vaddr >= start && vaddr <= end) {
#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 0)
		mmu_notifier_put(&page_table->mn);
#else
		mmu_notifier_unregister_no_release(&page_table->mn, page_table->mm);
#endif
		page_table->free_callback(page_table->data);
	}
}

If I register a memory region and, after it has been used, call unmap, there is a gap between mmu_notifier_put(&page_table->mn); and page_table->free_callback(page_table->data);. In other words, page_table->free_callback calls pevb_p2p_free_callback, which frees struct nvidia_p2p_page_table, but in nvidia_p2p_get_pages the following is registered:

	(*page_table)->mn.ops = &nvidia_p2p_mmu_ops;
	(*page_table)->mm = current->mm;
	(*page_table)->free_callback = free_callback;

That is, when the SRCU is later triggered in a kworker, it is very likely to access already-freed memory. Of course, usually on a Jetson AGX Orin, a large amount of kmem-256 will not be allocated, so the historical data in this memory region will be preserved, and thus SRCU can synchronize normally. However, when memory is tight, such a bug will usually manifest as a UAF.
This is bug context:

[  137.525796] [rdma] CMA allocation completed successfully
[  137.552831] [rdma] Starting auto transfer mode: 0
[  209.652256] Unable to handle kernel execute from non-executable memory at virtual address ffff0009a486f660
[  209.652263] Mem abort info:
[  209.652264]   ESR = 0x000000008600000f
[  209.652265]   EC = 0x21: IABT (current EL), IL = 32 bits
[  209.652266]   SET = 0, FnV = 0
[  209.652267]   EA = 0, S1PTW = 0
[  209.652268]   FSC = 0x0f: level 3 permission fault
[  209.652269] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000825ee8000
[  209.652270] [ffff0009a486f660] pgd=180000102cd68003, p4d=180000102cd68003, pud=180000102600e003, pmd=1800001025ee9003, pte=0068000a2486ff07
[  209.652278] Internal error: Oops: 000000008600000f [#1] PREEMPT_RT SMP
[  209.652282] Modules linked in: md5 cifs cifs_arc4 cifs_md4 xt_conntrack xt_MASQUERADE bridge stp llc ip6table_nat ip6table_filter ip6_tables xt_set ip_set iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype iptable_filter nf_tables libcrc32c nfnetlink lzo_rle lzo_compress zram zsmalloc motorcomm(O) ramoops reed_solomon tpm_ftpm_tee spidev tegra_cactmon_mc_all(O) tegra234_aon(O) st_m24(O) at24 mc_hwpm(O) host1x_fence(O) spi_tegra114 xdma(OE) nvidia_p2p(O) ina3221 crct10dif_ce governor_userspace nvidia(O) tegra_drm(O) nvgpu(O) nvethernet(O) tegra234_oc_event(O) nvhost_pva(O) tegra_wmark(O) nvhost_nvdla(O) nvpmodel_clk_cap(O) cec thermal_trip_event(O) tegra_dce(O) nvpps(O) nvhwpm(O) drm_kms_helper tegra_se(O) nvhost_capture(O) tsecriscv(O) host1x_nvhost(O) crypto_engine governor_pod_scaling(O) host1x(O) drm mc_utils(O) nvmap(O) nvsciipc(O) fuse ip_tables x_tables ipv6 pwm_fan pwm_tegra tegra_bpmp_thermal tegra_xudc ucsi_ccg typec_ucsi typec nvme nvme_core
[  209.652346]  phy_tegra194_p2u pcie_tegra194
[  209.652350] CPU: 1 PID: 230 Comm: kworker/1:3 Kdump: loaded Tainted: G           OE     5.15.148-rt-tegra #35
[  209.652353] Hardware name: NVIDIA NVIDIA Jetson AGX Orin Developer Kit/Jetson, BIOS r36.4.4-350c50a6-dirty 03/13/2026
[  209.652355] Workqueue: rcu_gp srcu_invoke_callbacks
[  209.652366] pstate: 40400009 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  209.652368] pc : 0xffff0009a486f660
[  209.652371] lr : srcu_invoke_callbacks+0x118/0x1b0
[  209.652374] sp : ffff80000a72bd30
[  209.652374] x29: ffff80000a72bd30 x28: 0000000000000000 x27: ffffa02048725300
[  209.652377] x26: 0000000000000001 x25: ffffa0204ad809f0 x24: ffffa02048725300
[  209.652378] x23: ffff000fa2bc8580 x22: ffff000fa2bc8540 x21: ffff000fa2bc85a0
[  209.652380] x20: ffff000fa2bc8658 x19: ffff0009a486f658 x18: 0000000000000000
[  209.652382] x17: ffff5fef586e5000 x16: ffff800008008000 x15: 0000000000000000
[  209.652383] x14: 0000000000000000 x13: 0000000000000030 x12: 0101010101010101
[  209.652385] x11: 7f7f7f7f7f7f7f7f x10: fefefefefefefeff x9 : ffffa0204869f784
[  209.652387] x8 : 8080808080808080 x7 : 0000000000008080 x6 : ffff00008717ea74
[  209.652388] x5 : 0000000fffffffe0 x4 : 0000000000000000 x3 : ffff5fef586e5000
[  209.652390] x2 : ffff000088274d80 x1 : ffff0009a486f660 x0 : ffff0009a486f658
[  209.652392] Call trace:
[  209.652393]  0xffff0009a486f660
[  209.652394]  process_one_work+0x200/0x510
[  209.652398]  worker_thread+0x138/0x480
[  209.652401]  kthread+0x194/0x1b0
[  209.652403]  ret_from_fork+0x10/0x20
[  209.652407] Code: 00001a92 00000000 00000000 00000000 (a486f660) 
[  209.652410] ---[ end trace 0000000000000002 ]---
[  210.657296] printk: At kmsg_dump, Line 3370
[  210.657299] printk: At kmsg_dump, Line 3388
[  210.657300] printk: kmsg_dump: calling dumper pstore_dump
[  210.660343] printk: kmsg_dump: dumper pstore_dump returned
[  210.660346] Kernel panic - not syncing:
[  210.660347] Oops: Fatal exception in interrupt
[  211.665243] SMP: stopping secondary CPUs
[  211.665302] Kernel Offset: 0x202040640000 from 0xffff800008000000
[  211.665303] PHYS_OFFSET: 0x80000000
[  211.665304] CPU features: 0x0,000005c1,a3332e5a
[  211.665307] Memory Limit: none
[  211.665309] printk: At kmsg_dump, Line 3370
[  211.665310] printk: At kmsg_dump, Line 3388
[  211.665311] printk: kmsg_dump: calling dumper pstore_dump
[  211.668376] printk: kmsg_dump: dumper pstore_dump returned
[  211.668524] ------------[ cut here ]------------
[  211.668525] WARNING: CPU: 1 PID: 230 at drivers/gpio/gpiolib.c:3340 gpiochip_disable_irq+0x78/0x80

I am only adding my own limited driver; all the official driver code involved is the original code from R36.4.4.

7

0xffff0009a486f660 is the PC pointer. The code I found is:

static void srcu_invoke_callbacks(struct work_struct *work)
{
	long len;
	bool more;
	struct rcu_cblist ready_cbs;
	struct rcu_head *rhp;
	struct srcu_data *sdp;
	struct srcu_struct *ssp;

	sdp = container_of(work, struct srcu_data, work);

	ssp = sdp->ssp;
	......
	spin_unlock_irq_rcu_node(sdp);
	rhp = rcu_cblist_dequeue(&ready_cbs);
	for (; rhp != NULL; rhp = rcu_cblist_dequeue(&ready_cbs)) {
		debug_rcu_head_unqueue(rhp);
		local_bh_disable();
		rhp->func(rhp);
		local_bh_enable();
	}
	......
}

rhp->func(rhp); is the pointer jump. In fact, this pointer has already been freed. The following is the error analysis:

crash> struct nvidia_p2p_page_table ffff0009a486f600
struct nvidia_p2p_page_table {
  version = 2419722664,
  page_size = 4294901760,
  size = 18446638657180601976,
  entries = 3485684386,
  pages = 0xffff80000e78bb88,
  vaddr = 4810,
  mapped = 0,
  mm = 0x7,
  mn = {
    hlist = {
      next = 0x1,
      pprev = 0x1
    },
    ops = 0xffff0000903a0968,
    mm = 0x1a92,
    rcu = {
      next = 0x0,
      func = 0xffff0009a486f660
    },
    users = 2760308320
  },
  lock = {
    rtmutex = {
      wait_lock = {
        raw_lock = {
          {
            val = {
              counter = -851715184
            },
            {
              locked = 144 '\220',
              pending = 219 '\333'
            },
            {
              locked_pending = 56208,
              tail = 52539
            }
          }
        }
      },
      waiters = {
        rb_root = {
          rb_node = 0x0
        },
        rb_leftmost = 0x0
      },
      owner = 0x0
    }
  },
  free_callback = 0xffffa020486be1e0 <delayed_work_timer_fn>,
  data = 0x200001
}

Here, in reality, this memory has already been allocated to another program for use:

text

crash> kmem ffff0009a486f658
 
 CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff000080002300      256      29454     29728    929     8k  kmalloc-256
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  fffffc0026921b80  ffff0009a486e000     0     32         12    20
  FREE / [ALLOCATED]
  [ffff0009a486f600]

      PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
fffffc0026921bc0 a2486f000 dead000000000002        0  0 800000000000000

Specifically, it has been used by:

crash> struct -o host1x_syncpt_fence
struct host1x_syncpt_fence {
    [0] struct dma_fence base;
   [64] atomic_t signaling;
   [72] struct host1x_syncpt *sp;
   [80] u32 threshold;
   [84] bool timeout;
   [88] struct delayed_work timeout_work;
  [176] struct list_head list;
}
SIZE: 192

crash> struct host1x_syncpt_fence 0xffff0009a486f600
struct host1x_syncpt_fence {
  base = {
    lock = 0xffff0000903a09a8,
    ops = 0xffffa01fcd3d0e78 <host1x_syncpt_fence_ops>,
    {
      cb_list = {
        next = 0x30cfc352a2,
        prev = 0xffff80000e78bb88
      },
      timestamp = 209644114594,
      rcu = {
        next = 0x30cfc352a2,
        func = 0xffff80000e78bb88
      }
    },
    context = 4810,
    seqno = 0,
    flags = 7,
    refcount = {
      refcount = {
        refs = {
          counter = 1
        }
      }
    },
    error = 0
  },
  signaling = {
    counter = 1
  },
  sp = 0xffff0000903a0968,
  threshold = 6802,
  timeout = false,
  timeout_work = {
    work = {
      data = {
        counter = 0
      },
      entry = {
        next = 0xffff0009a486f660,
        prev = 0xffff0009a486f660
      },
      func = 0xffffa01fcd3bdb90 <do_fence_timeout>
    },
    timer = {
      entry = {
        next = 0x0,
        pprev = 0x0
      },
      expires = 0,
      function = 0xffffa020486be1e0 <delayed_work_timer_fn>,
      flags = 2097153
    },
    wq = 0x0,
    cpu = 0
  },
  list = {
    next = 0xffff0009a486f6b0,
    prev = 0xffff0009a486f6b0
  }
}
8

R36.4.4

9

Hi,

We need to check this issue with our internal team.
Will update more information with you later.

Thanks.

10

Regarding this issue, you can add a small stress test to the RDMA callback function to increase the reproduction probability:

#ifdef RDMA_P2P_POISON_TEST
#define P2P_POISON_COUNT 32
/* Simulate host1x_syncpt_fence grabbing the just-freed kmalloc-256 slot
 * (for verification only, intentionally not freed) */
static void p2p_poison_slot(void)
{
	int i;

	for (i = 0; i < P2P_POISON_COUNT; i++) {
		void *p = kmalloc(200, GFP_ATOMIC);   /* lands in kmalloc-256 */

		if (p) {
			*(u64 *)((u8 *)p + 0x60) = (u64)p;   /* offset of rcu_head.func, self-reference */
			pr_info("[p2p-poison] poison[%d] = %px, +0x60 = %px\n",
				i, p, (void *)(*(u64 *)((u8 *)p + 0x60)));
		} else {
			pr_info("[p2p-poison] poison[%d] = NULL (kmalloc failed)\n", i);
		}
	}
}
#else
static inline void p2p_poison_slot(void) {}
#endif

Just call p2p_poison_slot() at the end of the RDMA(pevb_p2p_free_callback) free callback.