We are about to setup a network file system (Ceph) with Mellanox NICs and switches. Typically, multiple networks need to be configured (storage / access / admin …). This can either be achieved via multiple VLANs each running over a dedicated NIC port or by using QoS for multiple VLANs on a single port.
However, we are wondering how much latency QoS adds on Mellanox switches / NICs since most guides on “cluster setups” recommend using dedicated ports over Quality of Service.
Example: If you compare 4 dedicated 25GbE ports to one 100GbE port with evenly distributed QoS enabled for 4 VLANs, how much impact is to be expected on latency? Are there any benchmarks / comparisons?