Hi
I have connected 10gbps XT540 pcie card to xavier .
I have used iperf utility to check the average bandwidth.
With local host connection (server&client) on xavier I get only ~3.5 gbps.
This is same if I connect Xavier pcie to a desktop pcie (10gbps supported ,ubuntu version same as xavier) back to back.
Is there something to be enabled on xavier to get expected throughput ?
Jetpack v4.1 in use on Xavier.
This is same if I connect Xavier pcie to a desktop pcie (10gbps supported ,ubuntu version same as xavier) back to back.
Do you mean this card also only have 3.5Gbps on your desktop?
No.I meant even with back to back connection with desktop & xavier, iperf test with loop back (local host) on xavier gives same ~3.5 gbps.
Do you enable Xavier as end point mode?
No.
Do I need to enable it.?
Any link/info to enable it would be helpful.
I will try that and get back.
I just don’t understand how you connect desktop with xavier through PCIe and try the loopback test.
As for the PCIe performance, I will check internal team for help and may take some time.
Have any update ?
Sorry that we still don’t get any feedback yet.
Could you share what did you mean “back to back connection with desktop & xavier”? How did you connect it?
In back to back connection ,I have connected the PCIe of xavier to desktop PCIe (supports 10gbps) with Ethernet cable cat-type 7.
But there is no cat7 port on devkit, do you use some adapter?
I am using PCIe card Intel XT540 with 10gbps support.
Ok, got it. Our team is still checking. Thanks for you patience.
Hi,
Our team tried some test but does not have that low performance.
The test is between two Xavier.
We have tried using 2 Xaiver boards
Device 1 : Connect the Startech 10G Pcie card .
Device 2 : Connect the Intel x540 10G Pcie card .
Connect both the PCIe card using ethernet cable .
Configure private ip address to the ethernet interfaces .
The result of iperf is
[ 3] local 192.168.1.2 port 35746 connected with 192.168.1.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 8.11 GBytes 6.96 Gbits/sec
nvidia@jetson-0423818046617:~$
Are you also using iperf?
Sorry for delay in reply .
Yes, I am using iperf .
But I am trying the loopback ,with which I get ~3.5 gbps speed.
Can you let me know is there any setting to be enabled or configured.?
The bandwidth speed your team got has made any changes to network related configuration ?
Attachment has the screenshot of iperf output with loopback
iperf_nvidia.tar.gz (190 KB)
Hi ,
Have any updated ?
No, I don’t think our team has any special configuration.
Could you try to enhance cpu frequency with jetson_clock.sh and nvpmodel?
Hi ,
I tried with setting clock to max limit with jetson_clock.sh and nvpmodel mode 0.
Max speed with loopback I got is 3.35Gbps.
Do I need anything more to configure ?
L4T vesrion:
sudo sh jetson_clocks.sh --show
SOC family:tegra194 Machine:jetson-xavier
Online CPUs: 0-7
CPU Cluster Switching: Disabled
cpu0: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu1: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu2: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu3: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu4: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu5: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu6: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu7: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1
Fan: speed=255
nvidia@jetson:~$ iperf -c 127.0.0.1 -u -b 10G -i 1
------------------------------------------------------------
Client connecting to 127.0.0.1, UDP port 5001
Sending 1470 byte datagrams, IPG target: 1.10 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 127.0.0.1 port 60133 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 355 MBytes 2.98 Gbits/sec
[ 3] 1.0- 2.0 sec 360 MBytes 3.02 Gbits/sec
[ 3] 2.0- 3.0 sec 365 MBytes 3.06 Gbits/sec
[ 3] 3.0- 4.0 sec 323 MBytes 2.71 Gbits/sec
[ 3] 4.0- 5.0 sec 322 MBytes 2.70 Gbits/sec
[ 3] 5.0- 6.0 sec 325 MBytes 2.72 Gbits/sec
[ 3] 6.0- 7.0 sec 335 MBytes 2.81 Gbits/sec
[ 3] 7.0- 8.0 sec 356 MBytes 2.99 Gbits/sec
[ 3] 8.0- 9.0 sec 356 MBytes 2.99 Gbits/sec
[ 3] 0.0-10.0 sec 3.38 GBytes 2.90 Gbits/sec
[ 3] Sent 2467847 datagrams
[ 3] Server Report:
[ 3] 0.0-1260.8 sec <b>3.35 GBytes</b> 22.8 Mbits/sec 0.000 ms 18739/2467847 (0%)
nvidia@jetson:~$ sudo nvpmodel -q
[b]NV Power Mode: MAXN
0[/b]
nvidia@jetson:~$
How about changing some parameters for iperf? Need to clarify if this is due to iperf.
iperf3 -c <server> -u -b 0 -l 16K (default length is 8K) -t 120 -i 1
Hi WayneWWW ,
I tried with shared command on fresh setup with Jetpack 4.1.1.
Sending data from xavier(has 10gbps pcie card) to desktop PC (has 10gbps pcie card) .
With nvpmodel 2 and default clock setting of freq 1190400 on all cpu cores.
1.iperf -c 192.168.0.4 -u -b 10G -l 8k -i 1
[ 3] 0.0-10.0 sec 1.98 GBytes 1.70 Gbits/sec
[ 3] Sent 265615 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 372 MBytes 312 Mbits/sec 0.000 ms 60/265615 (0%)
2.iperf -c 192.168.0.4 -u -b 10G -l 4k -i 1
[ 3] 0.0-10.0 sec 1.92 GBytes 1.65 Gbits/sec
[ 3] Sent 514432 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 721 MBytes 605 Mbits/sec 0.000 ms 200/514432 (0%)
3.iperf -c 192.168.0.4 -u -b 10G -i 1
[ 3] 0.0-10.0 sec 2.43 GBytes 2.08 Gbits/sec
[ 3] Sent 1772046 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 2.09 GBytes 1.79 Gbits/sec 0.000 ms 246576/1772046 (0%)
With nvpmodel 0 and default clock setting of freq 2265600 on all cpu cores (with jetson_clocks.sh)
1.iperf -c 192.168.0.4 -u -b 10G -l 8k -i 1
[ 3] 0.0-10.0 sec 2.86 GBytes 2.45 Gbits/sec
[ 3] Sent 383271 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 533 MBytes 447 Mbits/sec 0.000 ms 3285/383271 (0%)
2.iperf -c 192.168.0.4 -u -b 10G -l 4k -i 1
[ 3] 0.0-10.0 sec 2.84 GBytes 2.44 Gbits/sec
[ 3] Sent 761910 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 1.02 GBytes 874 Mbits/sec 0.000 ms 18775/761910 (0%)
3.iperf -c 192.168.0.4 -u -b 10G -i 1
[ 3] 0.0-10.0 sec 3.17 GBytes 2.72 Gbits/sec
[ 3] Sent 2315981 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 2.35 GBytes 2.02 Gbits/sec 0.000 ms 600166/2315981 (0%)
With all the above tryouts I could get max of 2.73Gbps (average 2.02Gbps) of bandwidth with nvpmodel 0 and max freq of cpu’s.
Complete log-
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 10G -l 8k -i 1
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 8000 byte datagrams, IPG target: 5.96 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 34702 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 199 MBytes 1.67 Gbits/sec
[ 3] 1.0- 2.0 sec 197 MBytes 1.66 Gbits/sec
[ 3] 2.0- 3.0 sec 226 MBytes 1.90 Gbits/sec
[ 3] 3.0- 4.0 sec 195 MBytes 1.64 Gbits/sec
[ 3] 4.0- 5.0 sec 194 MBytes 1.63 Gbits/sec
[ 3] 5.0- 6.0 sec 200 MBytes 1.68 Gbits/sec
[ 3] 6.0- 7.0 sec 193 MBytes 1.62 Gbits/sec
[ 3] 7.0- 8.0 sec 202 MBytes 1.69 Gbits/sec
[ 3] 8.0- 9.0 sec 209 MBytes 1.75 Gbits/sec
[ 3] 0.0-10.0 sec 1.98 GBytes 1.70 Gbits/sec
[ 3] Sent 265615 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 372 MBytes 312 Mbits/sec 0.000 ms 60/265615 (0%)
nvidia@jetson-0423518032154:~$
nvidia@jetson-0423518032154:~$
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 10G -l 4k -i 1
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 4000 byte datagrams, IPG target: 2.98 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 45628 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 197 MBytes 1.65 Gbits/sec
[ 3] 1.0- 2.0 sec 195 MBytes 1.63 Gbits/sec
[ 3] 2.0- 3.0 sec 186 MBytes 1.56 Gbits/sec
[ 3] 3.0- 4.0 sec 203 MBytes 1.70 Gbits/sec
[ 3] 4.0- 5.0 sec 190 MBytes 1.60 Gbits/sec
[ 3] 5.0- 6.0 sec 193 MBytes 1.62 Gbits/sec
[ 3] 6.0- 7.0 sec 200 MBytes 1.67 Gbits/sec
[ 3] 7.0- 8.0 sec 198 MBytes 1.66 Gbits/sec
[ 3] 8.0- 9.0 sec 201 MBytes 1.68 Gbits/sec
[ 3] 9.0-10.0 sec 201 MBytes 1.68 Gbits/sec
[ 3] 0.0-10.0 sec 1.92 GBytes 1.65 Gbits/sec
[ 3] Sent 514432 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 721 MBytes 605 Mbits/sec 0.000 ms 200/514432 (0%)
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 10G -i 1
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 1470 byte datagrams, IPG target: 1.10 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 58985 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 245 MBytes 2.05 Gbits/sec
[ 3] 1.0- 2.0 sec 245 MBytes 2.06 Gbits/sec
[ 3] 2.0- 3.0 sec 253 MBytes 2.12 Gbits/sec
[ 3] 3.0- 4.0 sec 249 MBytes 2.09 Gbits/sec
[ 3] 4.0- 5.0 sec 245 MBytes 2.05 Gbits/sec
[ 3] 5.0- 6.0 sec 252 MBytes 2.11 Gbits/sec
[ 3] 6.0- 7.0 sec 252 MBytes 2.11 Gbits/sec
[ 3] 7.0- 8.0 sec 248 MBytes 2.08 Gbits/sec
[ 3] 8.0- 9.0 sec 250 MBytes 2.10 Gbits/sec
[ 3] 0.0-10.0 sec 2.43 GBytes 2.08 Gbits/sec
[ 3] Sent 1772046 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 2.09 GBytes 1.79 Gbits/sec 0.000 ms 246576/1772046 (0%)
nvidia@jetson-0423518032154:~$ sudo sh jetson_clocks.sh
jetson_clocks.sh: 410: jetson_clocks.sh: [[: not found
jetson_clocks.sh: 412: jetson_clocks.sh: [[: not found
jetson_clocks.sh: 414: jetson_clocks.sh: [[: not found
tegra194 416******
nvidia@jetson-0423518032154:~$ sudo nvpmodel -m 0
nvidia@jetson-0423518032154:~$ sudo sh jetson_clocks.sh
jetson_clocks.sh: 410: jetson_clocks.sh: [[: not found
jetson_clocks.sh: 412: jetson_clocks.sh: [[: not found
jetson_clocks.sh: 414: jetson_clocks.sh: [[: not found
tegra194 416******
nvidia@jetson-0423518032154:~$ sudo sh jetson_clocks.sh --show
jetson_clocks.sh: 410: jetson_clocks.sh: [[: not found
jetson_clocks.sh: 412: jetson_clocks.sh: [[: not found
jetson_clocks.sh: 414: jetson_clocks.sh: [[: not found
tegra194 416******
SOC family: Machine:jetson-xavier
Online CPUs: 0-7
CPU Cluster Switching: Disabled
cpu0: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu1: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu2: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu3: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu4: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu5: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu6: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
cpu7: Gonvernor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600
GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1
Fan: speed=255
nvidia@jetson-0423518032154:~$
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 10G -l 4k -i 1
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 4000 byte datagrams, IPG target: 2.98 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 55734 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 279 MBytes 2.34 Gbits/sec
[ 3] 1.0- 2.0 sec 288 MBytes 2.41 Gbits/sec
[ 3] 2.0- 3.0 sec 286 MBytes 2.40 Gbits/sec
[ 3] 3.0- 4.0 sec 293 MBytes 2.46 Gbits/sec
[ 3] 4.0- 5.0 sec 286 MBytes 2.40 Gbits/sec
[ 3] 5.0- 6.0 sec 282 MBytes 2.36 Gbits/sec
[ 3] 6.0- 7.0 sec 295 MBytes 2.47 Gbits/sec
[ 3] 7.0- 8.0 sec 294 MBytes 2.47 Gbits/sec
[ 3] 8.0- 9.0 sec 299 MBytes 2.51 Gbits/sec
[ 3] 0.0-10.0 sec 2.84 GBytes 2.44 Gbits/sec
[ 3] Sent 761910 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 1.02 GBytes 874 Mbits/sec 0.000 ms 18775/761910 (0%)
nvidia@jetson-0423518032154:~$
nvidia@jetson-0423518032154:~$
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 10G -l 8k -i 1
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 8000 byte datagrams, IPG target: 5.96 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 36749 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 284 MBytes 2.39 Gbits/sec
[ 3] 1.0- 2.0 sec 296 MBytes 2.48 Gbits/sec
[ 3] 2.0- 3.0 sec 293 MBytes 2.46 Gbits/sec
[ 3] 3.0- 4.0 sec 300 MBytes 2.51 Gbits/sec
[ 3] 4.0- 5.0 sec 293 MBytes 2.45 Gbits/sec
[ 3] 5.0- 6.0 sec 286 MBytes 2.40 Gbits/sec
[ 3] 6.0- 7.0 sec 283 MBytes 2.38 Gbits/sec
[ 3] 7.0- 8.0 sec 288 MBytes 2.42 Gbits/sec
[ 3] 8.0- 9.0 sec 296 MBytes 2.48 Gbits/sec
[ 3] 0.0-10.0 sec 2.86 GBytes 2.45 Gbits/sec
[ 3] Sent 383271 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 533 MBytes 447 Mbits/sec 0.000 ms 3285/383271 (0%)
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 10G -i 1
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 1470 byte datagrams, IPG target: 1.10 us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 38920 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 324 MBytes 2.72 Gbits/sec
[ 3] 1.0- 2.0 sec 326 MBytes 2.73 Gbits/sec
[ 3] 2.0- 3.0 sec 324 MBytes 2.72 Gbits/sec
[ 3] 3.0- 4.0 sec 324 MBytes 2.72 Gbits/sec
[ 3] 4.0- 5.0 sec 325 MBytes 2.73 Gbits/sec
[ 3] 5.0- 6.0 sec 325 MBytes 2.72 Gbits/sec
[ 3] 6.0- 7.0 sec 324 MBytes 2.72 Gbits/sec
[ 3] 7.0- 8.0 sec 324 MBytes 2.72 Gbits/sec
[ 3] 8.0- 9.0 sec 325 MBytes 2.72 Gbits/sec
[ 3] 0.0-10.0 sec 3.17 GBytes 2.72 Gbits/sec
[ 3] Sent 2315981 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 2.35 GBytes 2.02 Gbits/sec 0.000 ms 600166/2315981 (0%)
nvidia@jetson-0423518032154:~$ iperf -c 192.168.0.4 -u -b 0 -i 1
WARNING: delay too large, reducing from inf to 1.0 seconds.
------------------------------------------------------------
Client connecting to 192.168.0.4, UDP port 5001
Sending 1470 byte datagrams, IPG target: inf us (kalman adjust)
UDP buffer size: 224 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.5 port 56667 connected with 192.168.0.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 2.87 KBytes 23.5 Kbits/sec
[ 3] 1.0- 2.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 2.0- 3.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 3.0- 4.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 4.0- 5.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 5.0- 6.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 6.0- 7.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 7.0- 8.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 8.0- 9.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 9.0-10.0 sec 1.44 KBytes 11.8 Kbits/sec
[ 3] 0.0-11.0 sec 15.8 KBytes 11.8 Kbits/sec
[ 3] Sent 11 datagrams
[ 3] Server Report:
[ 3] 0.0-11.0 sec 15.8 KBytes 11.8 Kbits/sec 0.000 ms 0/ 11 (0%)
nvidia@jetson-0423518032154:~$
can you share me the settings(nvpmodel ,clock Freq .etc) on Xavier your team used to get ~7 Gbps bandwidth. ?