Hello, I’m trying to writing a program that reads host kernel memory on BF2 user space program, but getting RDMA Read failure on BF2 side with ‘remote operation error’ in the middle of sequential reading.
On host kernel side, I’ve set 200 pages on scatterlist and register it to a MR.
On BF2 side, it does RDMA Read on address (iova + index * 4KB )with MR’s rkey.
First reads succeed with proper contents, but suddenly it emits error on 86-th read.
When the error occurs on BF2 side, host also emits error message about local protection error.
How can I debug and fix it?
Host kernel version: 5.15.0
Host OFED version: 24.10-1.1.4.0-ubuntu22.04-x86_64
BF2: Set up via bf-bundle-2.9.1-30_24.11_ubuntu-22.04_prod.bfb
Error message on host side dmesg:
[ 66.346121] infiniband mlx5_0: dump_cqe:277:(pid 0): WC error: 4, Message: local protection error
[ 66.346131] cqe_dump: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 66.346136] cqe_dump: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 66.346139] cqe_dump: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 66.346142] cqe_dump: 00000030: 00 00 00 00 02 00 51 04 00 00 00 a6 00 01 e5 e2