Hello,
I have tried to give 10 Gb Ethernet functionality to my TX2, which has the latest Jetpack/L4T installed.
I used https://www.startech.com/Networking-IO/Adapter-Cards/pci-express-10g-sfp-network-adapter-card~PEX10000SFP that I saw in a forum post (https://devtalk.nvidia.com/default/topic/965204/?comment=5111350) about TX1 but then this came with all sorts of issues I read in the forums about speed being below 1Gbps and occasionally the driver crashing, or the interface going up and down.
I compile the latest driver for that card (3.6.13), by downloading the kernel sources through http://www.jetsonhacks.com/2017/03/25/build-kernel-and-modules-nvidia-jetson-tx2/ and then placing the sources among the other drivers, changing the .config and Kconfig files to build my driver as a module. I also tried with the driver patch described in the same forum post above.
Ethtool correctly advertises the speed as 10000 for the interface and lspci -vvv shows the correct driver loaded (tn40xx).
I also change the UDP buffers as below
sysctl -w net.core.rmem_default=33554432
sysctl -w net.core.rmem_max=33554432
sysctl -w net.core.wmem_default=33554432
sysctl -w net.core.wmem_max=33554432
The driver for some reason does not allow me to change the autoneg. options with ethtool.
Furthermore, adding the boot options
vmalloc=512M cma=64M coherent_pool=32M"
to /boot/extlinux/extlinux.conf
from https://devtalk.nvidia.com/default/topic/770634/large-coherent-dma-blocks/ seems to have fixed the interface going up and down problem and I stopped having crashes, but still, running an iperf -u session shows results that are below 1gb ethernet performance, with or without mtu 9000.
It is worth noting here that my test involves a TX2 and a TX1, both having the same Startech card, the same fix in the driver, boot options, their respective latest L4T from JetPack 3.0 (27.1 and 24.2.1 respectively), communicating directly. They both run with max clocks. In addition the TX1 has the fixes proposed for the kernel in the forum post about it.
When TX2 is the iperf server the iperf results are:
nvidia@tegra-ubuntu:~$ iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 32.0 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 43743
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 7.9 sec 355 MBytes 377 Mbits/sec 0.020 ms 66787/320017 (21%)
[ 3] 0.0- 7.9 sec 1 datagrams received out-of-order
^[[A
[ 4] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 40457
[ 4] 0.0-10.0 sec 413 MBytes 346 Mbits/sec 0.024 ms 0/294464 (0%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 50592
[ 3] 0.0-10.0 sec 466 MBytes 391 Mbits/sec 0.024 ms 0/332070 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 35259
[ 4] 0.0-20.9 sec 462 MBytes 185 Mbits/sec 0.018 ms 0/329284 (0%)
[ 4] 0.0-20.9 sec 1 datagrams received out-of-order
[ 5] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 35945
[ 5] 0.0-10.0 sec 452 MBytes 379 Mbits/sec 0.018 ms 30733/353388 (8.7%)
[ 5] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 46128
[ 3] 0.0-10.0 sec 456 MBytes 383 Mbits/sec 0.022 ms 0/325464 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 56242
[ 4] 0.0-10.0 sec 509 MBytes 427 Mbits/sec 0.033 ms 0/363063 (0%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 44037
[ 3] 0.0-10.0 sec 499 MBytes 419 Mbits/sec 0.029 ms 0/356065 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 39089
[ 4] 0.0-10.0 sec 519 MBytes 435 Mbits/sec 0.032 ms 0/369901 (0%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 41855
[ 3] 0.0-10.0 sec 482 MBytes 405 Mbits/sec 0.032 ms 0/344083 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.201 port 5001 connected with 192.168.2.200 port 54896
[ 4] 0.0-10.0 sec 515 MBytes 432 Mbits/sec 0.029 ms 0/367555 (0%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
//TX1 getting results like the below
ubuntu@tegra-ubuntu:~$ iperf -u -c 192.168.2.201 -b10G
------------------------------------------------------------
Client connecting to 192.168.2.201, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 32.0 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.200 port 54896 connected with 192.168.2.201 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 515 MBytes 432 Mbits/sec
[ 3] Sent 367556 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 515 MBytes 432 Mbits/sec 0.028 ms 0/367555 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
When the TX1 is the server:
ubuntu@tegra-ubuntu:~$ iperf -u -s
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 32.0 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 33335
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.2 sec 370 MBytes 304 Mbits/sec 12.603 ms 345203/609432 (57%)
[ 3] 0.0-10.2 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 46709
[ 4] 0.0-10.2 sec 372 MBytes 305 Mbits/sec 12.453 ms 348630/613950 (57%)
[ 4] 0.0-10.2 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 50262
[ 3] 0.0-10.0 sec 373 MBytes 313 Mbits/sec 0.016 ms 348231/614310 (57%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 60731
[ 4] 0.0-10.0 sec 374 MBytes 313 Mbits/sec 0.019 ms 358177/624641 (57%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 37318
[ 3] 0.0-10.0 sec 372 MBytes 314 Mbits/sec 0.021 ms 360227/625916 (58%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 37712
[ 4] 0.0-10.0 sec 374 MBytes 313 Mbits/sec 0.086 ms 354789/621515 (57%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 48871
[ 3] 0.0-10.0 sec 374 MBytes 314 Mbits/sec 0.081 ms 351278/618021 (57%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 49666
[ 4] 0.0-10.0 sec 565 MBytes 473 Mbits/sec 0.071 ms 218889/622130 (35%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 50083
[ 3] 0.0-10.0 sec 568 MBytes 475 Mbits/sec 0.023 ms 225166/630612 (36%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 37971
[ 4] 0.0-10.0 sec 574 MBytes 480 Mbits/sec 0.065 ms 226453/636046 (36%)
[ 4] 0.0-10.0 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 47321
[ 3] 0.0-10.3 sec 568 MBytes 465 Mbits/sec 13.574 ms 229590/634697 (36%)
[ 3] 0.0-10.3 sec 1 datagrams received out-of-order
[ 4] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 56819
[ 4] 0.0-10.3 sec 565 MBytes 463 Mbits/sec 12.971 ms 229560/632746 (36%)
[ 4] 0.0-10.3 sec 1 datagrams received out-of-order
[ 3] local 192.168.2.200 port 5001 connected with 192.168.2.201 port 36239
[ 3] 0.0-10.2 sec 373 MBytes 306 Mbits/sec 12.484 ms 367257/633472 (58%)
[ 3] 0.0-10.2 sec 1 datagrams received out-of-order
//TX2 getting results like the below
nvidia@tegra-ubuntu:~$ iperf -u -c 192.168.2.201 -b10G
[ 3] local 192.168.2.201 port 37318 connected with 192.168.2.200 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 877 MBytes 736 Mbits/sec
[ 3] Sent 625917 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 372 MBytes 314 Mbits/sec 0.021 ms 360227/625916 (58%)
[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
Mtu set to 9000 had slightly better results but not dramatically different.
I noticed that inserting a device in both USB slots of the TX1, as seen in both experiments above after a few executions, increased the speed, which was also observed in other forum posts. I am curious though as to why since I use a pciE card rather than the Jetson Ethernet interface. In addition, when TX2 is the client, it shows a transfer speed of 877 MBytes/736 Mbits/sec, whereas the TX1 has half of that.
Has anybody managed to make that card work for TX2/TX1 with the correct speed? Some members noted that it was working fine till L4T 24.1 but then broke for L4T 24.2.1.
Is there any other card, tried and tested, that can be used out of the box for 10Gb Ethernet performance instead of the Startech one?
Thanks