Good day,
I have a problem with RDMA and CPU utilization. My setup consists of 2x Dell AX-750 servers with 4x 25GB Mellanox Connect-X5 cards each. The servers are wired together with 4x 25GB DAC cables.
I get a speed of 80-95 Gbps with ntttcp.I tested the whole thing with RDMA and without RDMA but I see almost no change in the CPU load. The core aspect of RDMA is supposed to be the reduction of CPU load but I see nothing of it. I see in the Windows Admin Center that RDMA activity is present but why is the CPU load not reduced?
Driver information Windows 2019 core :3.0.25668.0
RDMA is enabled on the HYPER V switch. In the BIOS RDMA is enabled am I missing something?
While RDMA can significantly reduce CPU utilization in certain workloads, it is not a silver bullet solution and there can be several factors that impact its effectiveness. Here are some things you can check to help identify the cause of your high CPU utilization:
Make sure that your application is using RDMA for data transfer. Depending on how the application is implemented, it may not be automatically taking advantage of RDMA even if it is enabled at the system level. Check the application documentation or consult with the application vendor to verify RDMA support and configuration.
Ensure that your RDMA drivers and firmware are up-to-date. Check with the vendor for the latest driver and firmware versions and update them if necessary.
Check the RDMA settings on your network adapters. There may be advanced settings in the RDMA drivers that can impact performance, such as the number of outstanding RDMA operations or the size of the send and receive queues. Consult the vendor documentation for guidance on these settings and experiment with different values to see if it improves performance.
Verify that your workload is suited for RDMA. While RDMA can be very effective for certain types of workloads (e.g., large file transfers), it may not provide significant benefits for other workloads. Make sure that your application is well-suited for RDMA and that you are running a workload that can take advantage of its benefits.
Monitor system resources during your workload to identify potential bottlenecks. Use performance monitoring tools to track CPU, memory, network, and disk utilization during your workload to identify potential bottlenecks that could be impacting performance.
Check the RDMA settings on your Hyper-V virtual switch. Make sure that RDMA is enabled on the virtual switch and that the appropriate RDMA settings are configured for your virtual machines.
Overall, the effectiveness of RDMA will depend on the specific workload and system configuration, and it may not always lead to a significant reduction in CPU utilization. However, by verifying that RDMA is properly configured and experimenting with different settings, you may be able to improve performance and reduce CPU utilization.