I'm trying to implement Freeflow(https://github.com/microsoft/Freeflow) on RoCE 100G ConnectX-5. So I use libmlx5-1.2.1mlnx instead of libmlx4-1.2.1 in the original code of github link mentioned above. The following error was encountered about UAR.

When I modified libmlx5-1.2.1mlnx according to libmlx4-1.2.1mlnx in Freeflow,

In function mlx5_alloc_context(libmlx5-1.2.1mlnx1/src/mlx5.c): context->uar cannot map any type.

static int mlx5_alloc_context(struct verbs_device *vdev,

struct ibv_context *ctx, int cmd_fd)

{

for (i = 0; i < resp.tot_uuars / MLX5_NUM_UUARS_PER_PAGE; ++i) {

uar_mapped = 0;

/* Don’t map UAR to WC if BF is not used */

if (!context->shut_up_bf) {

context->uar[i].regs = mlx5_uar_mmap(i, MLX5_MMAP_GET_WC_PAGES_CMD, page_size, cmd_fd);

if (context->uar[i].regs != MAP_FAILED) {

context->uar[i].map_type = MLX5_UAR_MAP_WC;

uar_mapped = 1;

}

}

if (!uar_mapped) {

context->uar[i].regs = mlx5_uar_mmap(i, MLX5_MMAP_GET_NC_PAGES_CMD, page_size, cmd_fd);

if (context->uar[i].regs != MAP_FAILED) {

context->uar[i].map_type = MLX5_UAR_MAP_NC;

uar_mapped = 1;

}

}

if (!uar_mapped) {

/* for backward compatibility with old kernel driver */

context->uar[i].regs = mlx5_uar_mmap(i, MLX5_MMAP_GET_REGULAR_PAGES_CMD, page_size, cmd_fd);

if (context->uar[i].regs != MAP_FAILED) {

context->uar[i].map_type = MLX5_UAR_MAP_WC;

legacy_uar_map = 1;

uar_mapped = 1;

}

}

if (!uar_mapped) {

context->uar[i].regs = NULL;

goto err_free_cc;

}

}

}

If i set the resp.tot_uuars to be 0. Then My app will go wrong in the function mlx5_arm_cq

(libraries/libmlx5-1.2.1mlnx1/src/cq.c) because it writes “mlx5_write64(doorbell, ctx->uar[0].regs + MLX5_CQ_DOORBELL, &ctx->lock32);”

In Freeflow libmlx4-1.2.1mlnx1/src/mlx4.c, it ignores some variables in struct variable resp, so in mlx5.c I cannot get the resp.tot_uuars either.

Is there other solution to use RoCE in container in overlay network mode?

Hello Jiawei,

Thank you for your posting on the Mellanox Community.

Unfortunately, Freeflow is not a Mellanox products we develop and maintain. For any issues with Freeflow, please raise an issue on https://github.com/microsoft/Freeflow or reach out to the contacts mentioned on the homepage.

Thank you,

~Mellanox Technical Support