Mutil-device cudaMemcyPeer error-out of memory

I come to an error when i am practicing mutil device P2P memcpy.
I am using the 8 cards Titan XP, while Device Rank 5 and 7 was totally occupied by others, so I hired device 0 1 2 3 4 and 6.
But the Device Rank 6 will always come to wrong for the cudamemcpypeer call, errcode is 2, err string is “out of memory”.

and there is another interesting phenomemon, when i only hire 4 device (ranks 0 1 2 3, 2 3 4 6 or any other conbination), 2 device(ranks 4, 6 or oher conbination), it will be fine.

So i want t know why? I would be so appraciate for your advice.

here is the device information and P2P memcoy code.