100g connectX 5 slow speed between Mac to Proxmox TrueNAS VM

[yariv1]

Hi I’m experiencing slow speed using100g connectX 5 between Mac to Proxmox TrueNAS VM.
My setup is as follows:

• MAC PRO 2019 computer with Mellanox MCX516A-CCAT installed on a x6 lane PCIe gen3
• Supermicro server with Mellanox MCX516A-CDAT installed on a x16 lane PCIe gen3
• MIkrotik CRS504-4XQ-IN 100G switch connected to both Mellanox adaptors using DAC MCP1600 Mellanox cables.

on the Supermicro server i installed proxmox 7 hypervisor and TrueNAS VM. both mellanox firmware is on the latest version to date and on the same network (no VLANS). both adaptors show 100G link however when i use a iperf & iperf3 test i get only 9G speed between the MAC PRO to the TrueNAS and 27G between VM to VM. i also tried MTU 9000 and PTP bypassing the mikrotik switch but no change.

it feels like my speed is capped somewhere.

i’m out of ideas hope you can help

Thank you
Yariv

[yariv1]

Hi All,

Following to my OP i did more research looking for a macOS driver on NVIDIA, it doesn’t show it supported, is that correct?

however i found a post on twitter that Mellanox Connect -x5 has a driver on the new macOS Ventura see link for the post
https://twitter.com/khronokernel/status/1539658235761655809

[yariv1]

macOS Ventura location

[yariv1]

Network

[yariv1]

after upgrading to macOS Ventura i get bit better speed 13G Mac > VM when using iperf2 however i’m unable to setup jambo frames, only max frame size available is 2034! even through terminal get an error that 2034 is the max i can go.

i called apple they told me to ask NVIDIE although its an apple driver.

is it a bug or settings, any idea how can i change / fix it?

Thank you

[ipavis]

Hi Yariv,

Thank you for contacting us!
At this moment NVIDIA ConnectX Adapters are not supported on the Mac OS platform.
This hardware has not been tested on a Mac OS system and driver it self is an Apple driver.
As we have no knowledge of how this driver has been written and what features or test it has gone through we cannot offer any support on this issue.

Thanks and have a great day!
Ilan.

[nikon331]

I getting around the same speeds with almost the same hardware but a 4 port QNAP 100gbe switch and I’m running Windows 11 via Bootcamp on the 2019 Max Pro 16core 96GB ram. Max I’ve ever seen was 17gbps also using Jumbo Frames nothing seems to fix the speed issue. The card is showing 100gbe and running at x16 3.0. Don’t get it wonder if it’s a machine limitation tried all of the x16 PCIE slots no changes.