Issues with real-time sample streaming on DGX Spark RTL8127 NIC

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

Hi,

I am trying to debug an issue with real time streaming over ethernet using Realtek NIC on DGX Spark when using it for RAN workload (no other AI workload running) with a USRP based radio.

System:

  • Nvidia DGX Spark GB10
  • OS/Kernel: Ubuntu 24.04.4 LTS, DGX OS 7.5.0 / 6.17.0-1029-nvidia
  • NIC: Realtek RTL8127 r8127 - enP7s7
  • Radio: USRP X310 10GbE
  • RAN Workload: Duranta-OAI develop and Nvidia sionna-rk (SRK v1.3.2)

The link between the radio and spark is detected as 10gig

$ sudo ethtool enP7s7 | grep -E 'Speed|Link'	
	Link partner advertised link modes:  10baseT/Full
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Link detected: yes

Around June/july, I could run my RAN workload at 20MHz and 40MHz bandwidth configs reliably. These configs roughly correspond to streaming raw IQ (sc16) payload of 1.5 and 3 Gbps respectively.

But after applying latest system/driver updates for DGX spark, this setup is now unable to sustain the rates required for 20MHz & 40MHz - the RAN (OAI) just reports a continuous stream of L late packet issues and crashes after failing to recover (RAN logs - gnb_20mhz_nos1.log, gnb_40mhz_nos1_txthread.log). At this moment, 10MHz config (<1Gbps) in SRK is the only one that’s working reliably with current Ethernet interface based setup.

Switching the radio to USRP B210 which uses USB3 interface does not show the late packet behavior, with these configs. So I strongly suspect the issue lies with the NIC / driver updates. Responses from OAI mailing list suggested trying out real-time kernel and that they were able to run same radio setup with different interface. So waiting on QSFP to SFP+ adapter order to test the same on ConnectX NIC, but in the mean time

Questions:

  • If this is indeed a limitation on consumer grade NIC - is there a way to to test/debug by rolling back updates for specific driver or spark as whole to get my system back to previous working state
  • Has anyone tested any similar setups with Spark’s Realtek NIC and noticed similar issues in performance/ realtime streaming - would be interested to know your experience
  • Also, DGX spark already comes with optimized ubuntu kernel but as suggested in OAI mailing list and previously raised in forum - how can I go about trying out real-time kernels, are there any instructions /resources where I can know more on this?

Thanks

2

Can you provide the full ethtool output during the test? I want to make sure the link isn’t silently negotiating down to 1G/2.5G