100g connectX 5 slow speed between Mac to Proxmox TrueNAS VM

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi I’m experiencing slow speed using100g connectX 5 between Mac to Proxmox TrueNAS VM.
My setup is as follows:

  • MAC PRO 2019 computer with Mellanox MCX516A-CCAT installed on a x6 lane PCIe gen3
  • Supermicro server with Mellanox MCX516A-CDAT installed on a x16 lane PCIe gen3
  • MIkrotik CRS504-4XQ-IN 100G switch connected to both Mellanox adaptors using DAC MCP1600 Mellanox cables.

on the Supermicro server i installed proxmox 7 hypervisor and TrueNAS VM. both mellanox firmware is on the latest version to date and on the same network (no VLANS). both adaptors show 100G link however when i use a iperf & iperf3 test i get only 9G speed between the MAC PRO to the TrueNAS and 27G between VM to VM. i also tried MTU 9000 and PTP bypassing the mikrotik switch but no change.

it feels like my speed is capped somewhere.

i’m out of ideas hope you can help

Thank you
Yariv

2

Hi All,

Following to my OP i did more research looking for a macOS driver on NVIDIA, it doesn’t show it supported, is that correct?

however i found a post on twitter that Mellanox Connect -x5 has a driver on the new macOS Ventura see link for the post
https://twitter.com/khronokernel/status/1539658235761655809

3

macOS Ventura location

Screenshot 2023-02-28 at 13.30.22
Screenshot 2023-02-28 at 13.30.221587×475 115 KB

4

Network

Screenshot 2023-02-28 at 13.32.56
Screenshot 2023-02-28 at 13.32.56708×625 57.6 KB

5

after upgrading to macOS Ventura i get bit better speed 13G Mac > VM when using iperf2 however i’m unable to setup jambo frames, only max frame size available is 2034! even through terminal get an error that 2034 is the max i can go.

Screenshot 2023-02-28 at 13.37.16
Screenshot 2023-02-28 at 13.37.16714×622 60.2 KB

i called apple they told me to ask NVIDIE although its an apple driver.

is it a bug or settings, any idea how can i change / fix it?

Thank you

6

Hi Yariv,

Thank you for contacting us!
At this moment NVIDIA ConnectX Adapters are not supported on the Mac OS platform.
This hardware has not been tested on a Mac OS system and driver it self is an Apple driver.
As we have no knowledge of how this driver has been written and what features or test it has gone through we cannot offer any support on this issue.

Thanks and have a great day!
Ilan.