Issues porting desktop RDMA app to Tegra: mmap hangs kernel

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
9

Hi,

Thanks for your patience.
Here are some of the suggestions:

1.

The input physical address for io_remap_pfn_range should be address of struct page on Jetson.
Please ensure to compile the sources with the Jetson version of nv-p2p.h.

Something like:

#if (defined(CONFIG_ARM64) && defined(CONFIG_ARCH_TEGRA))
    dmachan->reader_addr[j] = page_to_pfn((struct page*)nvp);
#else
    dmachan->reader_addr[j] = (uint32_t*)(nvp->physical_address + offset);
#endif

It looks like the nv-p2p.h header has different page_table struct fields for desktop and Jetson:

Desktop:
typedef
struct nvidia_p2p_page {
    uint64_t physical_address;
    union nvidia_p2p_request_registers {
        struct {
            uint32_t wreqmb_h;
            uint32_t rreqmb_h;
            uint32_t rreqmb_0;
            uint32_t reserved[3];
        } fermi;
    } registers;
} nvidia_p2p_page_t;

typedef
struct nvidia_p2p_page_table {
    uint32_t version;
    uint32_t page_size; /* enum nvidia_p2p_page_size_type */
    struct nvidia_p2p_page **pages;
    uint32_t entries;
    uint8_t *gpu_uuid;
} nvidia_p2p_page_table_t;
Jetson:
typedef struct nvidia_p2p_page_table {
    u32 version;
    u32 page_size;
    u64 size;
    u32 entries;
    struct page **pages;
  
    u64 vaddr;
    u32 mapped;
  
    struct mm_struct *mm;
    struct mmu_notifier mn;
    struct mutex lock;
    void (*free_callback)(void *data);
    void *data;
} nvidia_p2p_page_table_t;

2.

Ensure to follow “Modification to Kernel API” from https://developer.nvidia.com/blog/gpudirect-rdma-nvidia-jetson-agx-xavier/.
The mapping size should be multiple of 4K, Write combine requirement while remapping.

Thanks.