Iperf3 PCIe NTB Non-crosslink bandwidth measurement

Autonomous Vehicles DRIVE AGX Xavier DRIVE AGX Xavier General
1

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.6.0.8170
other

Host Machine Version
native Ubuntu 18.04
other

Hello, I followed the steps in Non-Transparent Bridging and PCIe Interface Communication and I have two Xavier devices communicating through each other using a PCIe ntb non-crosslink communication.
When trying to measure the bandwidth using iperf3, I get the following figures:

nvidia@xavier1a:~/workspace$ iperf3 -u -c 192.168.1.11 -b 100G
Connecting to host 192.168.1.11, port 5201
[  4] local 192.168.1.12 port 60717 connected to 192.168.1.11 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   210 MBytes  1.76 Gbits/sec  26837  
[  4]   1.00-2.00   sec   210 MBytes  1.76 Gbits/sec  26913  
[  4]   2.00-3.00   sec   211 MBytes  1.77 Gbits/sec  26996  
[  4]   3.00-4.00   sec   211 MBytes  1.77 Gbits/sec  27036  
[  4]   4.00-5.00   sec   211 MBytes  1.77 Gbits/sec  27027  
[  4]   5.00-6.00   sec   212 MBytes  1.77 Gbits/sec  27072  
[  4]   6.00-7.00   sec   211 MBytes  1.77 Gbits/sec  27055  
[  4]   7.00-8.00   sec   211 MBytes  1.77 Gbits/sec  27001  
[  4]   8.00-9.00   sec   211 MBytes  1.77 Gbits/sec  27032  
[  4]   9.00-10.00  sec   211 MBytes  1.77 Gbits/sec  27010  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  2.06 GBytes  1.77 Gbits/sec  0.011 ms  9011/269979 (3.3%)  
[  4] Sent 269979 datagrams

iperf Done.

The bandwidth of 1.77 Gbits/sec is not in line with the theoretical limit of a PCIe connection of 4 lanes.
How can we get a better performance out of the connection in order to get as close to the theoretical limit as possible?

Thanks in advance for your reply

4

Dear @erick.vieyra,
We will check internally and update you. Could you check the same with our latest release DRIVE OS 5.2.6? Also, check with iperf instead of iperf3 and share the results.

5

Thanks for your reply @SivaRamaKrishnaNV !
I have redone the tests using iperf and iperf3. There does not seem to be much difference. The results are shown below:

iperf XavierB Server, XavierA Client

nvidia@xavier1a:~$ iperf -c 192.168.1.12
------------------------------------------------------------
Client connecting to 192.168.1.12, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.11 port 54554 connected with 192.168.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth     
[  3]  0.0-10.0 sec  2.70 GBytes  2.32 Gbits/sec

iperf3 XavierB Server, XavierA Client

Connecting to host 192.168.1.12, port 5201
[  4] local 192.168.1.11 port 57544 connected to 192.168.1.12 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   274 MBytes  2.29 Gbits/sec    0    639 KBytes       
[  4]   1.00-2.00   sec   279 MBytes  2.34 Gbits/sec    0    639 KBytes       
[  4]   2.00-3.00   sec   279 MBytes  2.34 Gbits/sec    0    639 KBytes       
[  4]   3.00-4.00   sec   280 MBytes  2.34 Gbits/sec    0    639 KBytes       
[  4]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec    0    639 KBytes       
[  4]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec    0    639 KBytes       
[  4]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0    639 KBytes       
[  4]   7.00-8.00   sec   279 MBytes  2.34 Gbits/sec    0    639 KBytes       
[  4]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0    639 KBytes       
[  4]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0    639 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.72 GBytes  2.34 Gbits/sec    0             sender  
[  4]   0.00-10.00  sec  2.72 GBytes  2.34 Gbits/sec                  receiver

iperf Done.