# Got "local protection error" when doing IBV\_WR\_SEND

**URL:** <https://forums.developer.nvidia.com/t/got-local-protection-error-when-doing-ibv-wr-send/258886>\
**Category:** Mellanox OFED\
**Created:** [July 5, 2023, 11:11am UTC](https://forums.developer.nvidia.com/t/got-local-protection-error-when-doing-ibv-wr-send/258886 "2023-07-05T11:11:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![xeric.w](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@xeric.w](https://forums.developer.nvidia.com/u/xeric.w)\
**Post date:** [July 5, 2023, 11:11am UTC](https://forums.developer.nvidia.com/t/got-local-protection-error-when-doing-ibv-wr-send/258886/1 "2023-07-05T11:11:06Z")

</div>

I’m trying to use RDMA to send data to or from a FPGA PCIe card.  
The test platform is an Intel CPU PC with a cx4121a card and a FPGA PCIe card. OS is Ubuntu 22.04 x86\_64, with MLNX\_OFED\_LINUX-5.8-2.0.3.0.  
I wrote a kernel driver for the FPGA card to implement a peer\_memory\_client, which heavily referenced nv\_peer\_memory.  
It worked fine on IBV\_WR\_RECV, cx4121a could successfully write the data it received to FPGA card’s BAR2 address (which was mapped to DDR on FPGA card). But when doing IBV\_WR\_SEND (using cx4121a to send data from FPGA card’s BAR2 address to remote), I got a local protection error. And there was no warning or error message in kernel log.  
It was the same buffer address/length and same memory region used in both IBV\_WR\_SEND and IBV\_WR\_RECV, so I have no clue why IBV\_WR\_RECV worked but IBV\_WR\_SEND not.

Any help would be appreciated, thanks!

Here are some details:

1. I implement mmap in my driver to map FPGA card’s BAR2 into user space, get an virtual address

```auto
	vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP;
	vma->vm_page_prot = pgprot_noncached(vm_get_page_prot(vma->vm_flags));
	ret = remap_pfn_range(vma, vma->vm_start,
				pfn + vma->vm_pgoff,
				user_size,
				vma->vm_page_prot);

```

1. fill this virtual address to struct ibv\_sge, then pass it to ibv\_post\_send or ibv\_post\_recv

```auto
    ib_res.buf = mmap(NULL, length, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

    struct ibv_sge list = {
            .addr	= (uint64_t)ib_res.buf,
            .length = size,
            .lkey	= ib_res.mr->lkey
    };

    struct ibv_send_wr *bad_wr, wr = {
            .wr_id = SEND_WRID,
            .sg_list = &list,
            .num_sge = 1,
            .opcode = IBV_WR_SEND,
            .send_flags = IBV_SEND_SIGNALED,
            .next = NULL
    };

    ibv_post_send(ib_res.qp, &wr, &bad_wr);

```

1. ib core driver will find out this user virtual address is not normal memory, so it will try to get pages from peers
2. I implement peer\_memory\_client in my driver and register it to ib core. when ib core asks to do a dma\_map, I fill the physical address of FPGA card’s BAR2 into the sg table.

```auto
	for_each_sg(sg_head->sgl, sg, hello_mem_context->npages, i) {
		sg_set_page(sg, NULL, hello_mem_context->page_size, 0);
		sg->dma_address = bar2_phys_addr[hello_mem_context->belongs_to_card] + ((i++) * hello_mem_context->page_size);
		sg->dma_length = hello_mem_context->page_size;
	}

```

If I replaced FPGA card’s BAR2 with a kmalloc buffer in driver, every thing worked fine. So I assume the driver in general is good, just something is wrong with PCIe BAR address mapping in driver?

---

<div class="post-metadata">

**Author:** ![xiaofengl](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xiaofengl/32/367020_2.png) [@xiaofengl](https://forums.developer.nvidia.com/u/xiaofengl)\
**Post date:** [July 6, 2023, 1:23am UTC](https://forums.developer.nvidia.com/t/got-local-protection-error-when-doing-ibv-wr-send/258886/2 "2023-07-06T01:23:25Z")

</div>

SEND is different with regular RDMA read/write, it need mem copy to HCA buffer. If you are developing driver, better way is wotk with kernel community and get IB HCA PRM (program manual) from NIVDIA sales.

---

<div class="post-metadata">

**Author:** ![xeric.w](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@xeric.w](https://forums.developer.nvidia.com/u/xeric.w)\
**Post date:** [July 6, 2023, 11:10am UTC](https://forums.developer.nvidia.com/t/got-local-protection-error-when-doing-ibv-wr-send/258886/3 "2023-07-06T11:10:22Z")

</div>

Thanks for the infomation!

I’ve tested four APIs:

1. ibv\_post\_recv
2. ibv\_post\_send with opcode = IBV\_WR\_SEND
3. ibv\_post\_send with opcode = IBV\_WR\_RDMA\_READ
4. ibv\_post\_send with opcode = IBV\_WR\_RDMA\_WRITE

1 and 3 successed, 2 and 4 failed with local protection error. So it seems HCA can write to FPGA card’ PCIe BAR address, but cann’t read from it… Any thoughts?

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [July 20, 2023, 11:10am UTC](https://forums.developer.nvidia.com/t/got-local-protection-error-when-doing-ibv-wr-send/258886/4 "2023-07-20T11:10:33Z")

</div>

Closed on July 20, 2023, 11:10am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
