Hi,
There are 6 NVMe SSD cards + 1 Mellanox Ethernet card connecting to the PCIe interface. Concurrent access through PCIe switch looks to be the cause of performance drop. Are you able to try 6 NVMe SSD cards in one PCIe interface and 1 Mellanox Ethernet card in the other?
And a known issue in PCIe switch is
[35.3.1] Intel I210 throughput issue on Jetson Orin Nano Devkit - #12 by sgidel
Please try the patch:
Boot stuck while enumerating NVMe via PFX switch, seems to be PCIe driver issue - #8 by WayneWWW