Clarification on DPA external pointer/window switching when accessing multiple mmap regions

Infrastructure & Networking BlueField
1

(DOCA SDK API, BF3, DPA)
I am trying to clarify the correct programming model for accessing memory regions exported to DPA through DOCA mmap / buf array objects.

In my DPA code, I access host/DPU memory through a helper similar to the following:

static uint8_t *dpa_mmap_ptr(doca_dpa_dev_buf_arr_t buf_arr,
                             uint64_t base_addr,
                             uint32_t buf_size,
                             uint64_t addr,
                             uint64_t size)
{
    doca_dpa_dev_buf_t buf;
    doca_dpa_dev_uintptr_t dev_ptr;

    if (addr < base_addr)
        return 0;

    if ((addr - base_addr) > buf_size ||
        size > (uint64_t)buf_size - (addr - base_addr))
        return 0;

    buf = doca_dpa_dev_buf_array_get_buf(buf_arr, 0);
    dev_ptr = doca_dpa_dev_buf_get_external_ptr(buf);

    return ((uint8_t *)dev_ptr) + (addr - base_addr);
}

I observed that, when the DPA code accesses a specific mmap-backed pointer, repeatedly calling doca_dpa_dev_buf_array_get_buf() and then doca_dpa_dev_buf_get_external_ptr() appears to make the access behave correctly. In contrast, when I cache an external pointer or switch between different mmap regions without re-fetching the corresponding buffer/external pointer, I sometimes observe invalid or stale-looking values.

I would like to understand whether this is an expected requirement of the DPA programming model or whether it indicates a bug in my code.

My questions are:

  1. Is doca_dpa_dev_buf_array_get_buf() purely an accessor that returns a buffer handle, or does it also select/switch some DPA-side memory window or address translation context?

  2. When a DPA thread accesses multiple exported mmap regions, is it required to call doca_dpa_dev_buf_array_get_buf() / doca_dpa_dev_buf_get_external_ptr() for the target region before each access?

  3. If multiple mmap regions are used, is it required to perform an explicit memory-window switch every time the DPA code accesses a region, immediately before dereferencing that region’s external pointer?

The reason I am asking is that the observed behavior suggests that accessing a different mmap region may require selecting the corresponding buffer/window again, but I could not determine whether this is part of the official API contract or only an artifact of my current implementation.

The concrete issue is:

  • I have multiple mmap/buf-array-backed regions.

  • The DPA code computes external pointers using doca_dpa_dev_buf_get_external_ptr().

  • Re-fetching the buffer and external pointer before accessing a region seems to work.

  • Reusing previously cached external pointers while switching across regions sometimes appears to produce invalid or stale data.

Could you clarify the intended usage pattern? In particular, should the recommended pattern be:

buf = doca_dpa_dev_buf_array_get_buf(buf_arr, index);
ptr = doca_dpa_dev_buf_get_external_ptr(buf);
/* access ptr immediately */

every time before accessing a mmap-backed region, or is it valid to initialize these pointers once and reuse them throughout the lifetime of the DPA program?

Thank you.

2

Hello @dbwlgns8314

Thank you for posting your query on our community.

For DPA external memory access, doca_dpa_dev_mmap_get_external_ptr() and doca_dpa_dev_buf_get_external_ptr() initialize thread-local metadata, and each call overwrites the current thread context. Because of that, pointers from different mmap/buffer objects cannot be safely interleaved.

So you do not need to refresh before every access to the same region, but if you switch to another mmap/buffer and later want to access the first one again, you should call doca_dpa_dev_mmap_get_external_ptr() or doca_dpa_dev_buf_get_external_ptr() again for that object.

Also, doca_dpa_dev_buf_array_get_buf() just returns the buffer handle. The external access context is established by doca_dpa_dev_buf_get_external_ptr(), not by get_buf() itself.

Please refer - DOCA DPA - NVIDIA Docs

Thanks,

Bhargavi