We’ve built an FPGA-based RoCEv2 (UC) streaming source that RDMA-WRITEs fixed-size packets into any CX7, but specifically DGX Spark (GB10, ConnectX-7) at ~80 Gb/s per stream. On the host, we’d like the NIC to split each packet at placement time — header bytes to one ring, I/Q payload to a contiguous ring — using an interleaved indirect mkey configured via mlx5dv_wr_mkey_configure / mlx5dv_wr_set_mkey_layout_interleaved (pattern: 32 B header + 65536 B payload, repeated forever).
Questions:
- Is an interleaved mkey with IBV_ACCESS_REMOTE_WRITE supported as the target of incoming RDMA WRITEs on a UC QP in the same PD (with the UMR posted on a separate RC configuration QP)?
- What are the ConnectX-7 limits on num_interleaved entries per pattern and on total mkey span / repeat_count, and which device capability fields report them?
- Is there a known throughput penalty for repeated-block indirect mkeys vs. a direct mkey at this block size (64 KB payload blocks)?
- What base virtual address should a remote writer use for an indirect mkey — zero-based or the address reported at configuration?
We’ve worked from the rdma-core man pages, the mlx5 provider source and tests, and the older public Mellanox Adapters PRM. If the current ConnectX-7 PRM is available to developers (registration, NDA, or partner program), we’d appreciate a pointer to the right access path.
Thanks as always,
Shep Siegel
Atomic Rules LLC