This is a design experience: IDT’s PCIE Switch is BUGGY in term of its DMA function
-
the DMA speed is insanely slow.
-
the TLPs emitted by DMA function maybe silently discarded, and no AER gets triggered.
As a result, a DMA based memory copy operation maybe incomplete.
I tried to copy a source buffer to the same dest buffer for quite a number of times,
the outcome is like this:
for example, src buffer is, src[1024] = {0, 1, 2, …, 15, 0, 1, 2, …, 15 …, 0, 1, 2, …, 15 } ;
for the 1st time of dma copy, you will find that the dst buffer holds nothing:
dst[1024] = {0, 0, 0, …, 0 };
but, as the times of dma copies increases, you will find that dst buffer will gradually
hold everything expected, until finally, dst buffer is fully loaded:
dst[1024] = {0, 1, 2, …, 15, 0, 1, 2, …, 15 …, 0, 1, 2, …, 15 } ;
IDT’s PCIE Switch functions well only with its NTB port.
But, its DMA engine is so buggy, totally shit !
Anyone want’s to connect 2 or more GPU(s), like, TX2(i) and Xavier, Please NEVER NEVER consider
the choice of IDT’s PCIE Switch.
A good alternative maybe PLX product (already acquired by Broadcom), like PEX**** series.