What is the host here? Is it another Xavier system (apart from the two Xaviers which are operating as EPs behind the switch) ? or an x86 system?
If the host is not Xavier, then, whether Xavier has P2P support or not doesn’t matter. Xavier’s P2P capability is immaterial when it comes to communicating with another Xavier operating as EP.
Although you may need P2P support for the host (be it x86 or Xavier), it is not mandatory either if the PCIe switch has support for P2P.
Bottom line: Check for P2P support in the PCIe switch and you are good to go.
Regarding the data transmission efficiency, there are multiple factors that can affect this. One major factor being the SW which is operating the DMA engines to do the data transfers. From Xavier point of view, if the device driver for its EP mode is using the DMA engines properly, it can satisfy your requirement.