my excuses in case this discussion is misplaced. I’m a new member of this community, but I expect o be here more often in the future.
I’m currently working on a concept and realization for a high-data-rate-acquisition system based on a cluster and Infiniband as an interconnect. Nothing new up to this point. The big thing with our system will be the data rate for data acquisition. Here at University, we’re going for a large research project for signal processing topics at THz spectrum. Our concept includes two clusters (Infiniband, MPI, GPU) as well as input/output components for high-data-rate-aquisition (FPGA based). Each compute-node will be used as an input/output node for data with data rates up to 5 GBytes/s per node. The whole system will scale up with parallel instances of cluster-node/FPGA entities ending in system data rates of more than 80 GBytes/s. Lot of data to be transmited and stored in short time. Data will be processed offline.
I have some concerns regarding data input/output processing on node base. How to get the data into the cluster-nodes? Big question. Up to now, there is the solution of using PCIe bus for connecting compute-node with FPGA board. Surely, this will work, but still an ‘old-fashioned’ way without using RDMA, Infiniband etc. I’ve looked around to find a suitable solution for integrating Infiniband and/or Converged Ethernet as a technology to couple with FPGA data stream. To note the result: COTS devices are not available.
Currently, we’re a little bit lost. On FPGA (UltraScale+) we could make use of a 100 GbE Xilinx transceiver. But how to get this transceiver working with Mellanox Connectx? I’m not really sure that Connectx and Xilinx 100 GbE transceiver can communicate with each other- though both explain comformity for 100 GbE. In addition, support by Xilinx for upper transport layers are missing. Not to mention RDMA or RoCE.
Does anybody have experience with such an interconnect based on 100 GbE Ethernet? The 100 GbE transceiver from Xilinx FPGA (UltraScale+) comes out-of-the-box with less or no support. What about hardware offloading? Xilinx seems to support this …
Second concept is to use Infiniband as an interconnect for compute-node/FPGA coupling. Then it would be very helpful to get an Infiniband IP core for FPGA for at least FDR (5 GBytes/s transfer rate!). Could someone provide me some buisiness contacts for this?
Third concept could be to develop an integrated solution, FPGA plus Connectx silicon on PCIe interface card. This seems to be the most expensive solution wrt time and effort.
Would be very helpful to get some supporting answers from the community.