Maximum message size supported for multicast?

1

I was testing out the example multicast application provided by mellanox in section 7.3 of their document: http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf . I’m able to compile and run the application and do multicast. I run using the following commands:

Receiver side:

example -M 224.0.0.1 -b 192.168.201.101 -c 1 -C1024 -S 1024

…

Sender’s side

example -M 224.0.0.1 -b 192.168.201.30 -c 1 -C1024 -S 1024 -s

I believe this is the same as mckey utility.

We have a requirement to transfer or copy 10-20 GB of data from server to multiple clients. I thought using multicast will be efficient. I have couple of questions about the given example (mckey):

  1. I’m able to only increase the message size to 4096 bytes. If I increase message size beyong that, there is a check for message_size as below:

if (message_count && message_size > (1<< (port_attr.active_mtu + 7)))

which kicks in and gives an error.

The port attributes in our network are as follows:

{state =IBV_PORT_ACTIVE, max_mtu = IBV_MTU_4096, active_mtu = IBV_MTU_4096, gid_tbl_len= 8, port_cap_flags = 642902090, max_msg_sz = 1073741824, bad_pkey_cntr = 0, qkey_viol_cntr =0, pkey_tbl_len = 128, lid = 6, sm_lid = 9, lmc = 0 ‘\000’, ax_vl_num = 3’\003’, sm_sl = 0 ‘\000’, subnet_timeout = 18 ‘\022’, init_type_reply = 0’\000’, active_width = 2’\002’, active_speed = 32 ’ ', phys_state = 5 ‘\005’, link_layer = 1 ‘\001’,reserved = 0 ‘\000’}

I can see that max_msg_sz supported by the port, from above, is 1GB and why am I not allowed to specify more than 4096 as message size? In fact we have another application where QP type is RC and we are able to transfer messages up to 1GB there between the same nodes and HCA and port. Is there a limitation while doing multicast?

  1. What is the maximum message count? Anythingbeyond 16K, leads to error during rdma_create_qp.

Thanks

Sri

2

Hello

There is a screenshot of section 7.3 (pages 151) in the provided reference document as follows:

From above, the type of QP is set to IBV_QPT_UD, this is unreliable datagram transmission. Its max message size is limited to Active MTU.

3

IB multicast is UD and as such max message sfize is MTU of MC group. Unlike RC and some other transports, UD does not include segmentation/reassembly (in hardware) and would need to be implemented in software above the pure UD sends. Note also that UD is not reliable (hence the U in UD).

– Hal