Hi wenbinjlumj,
I did not investigate this issue any further as it seems that is the performance you can expect with the current software stack.
To use the 10Gbit network interface for communication between Xavier A and B, I simply connected the two 10GBit RJ45 connectors with a cable. The system is already set up with fixed IP addresses on this connection (192.168.0.10 and 192.168.0.11 on Xavier A and B, respectively) so no configuration is needed on the software side.