Poor performance of the 10Gb network of Drive AGX Orin

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
[*] DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
[*] Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
[*] DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
[*] other

Host Machine Version
[*] native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello,
I want to use NAS as a storage expansion device for Drive AGX Orin to transfer data over a 10Gb network (mgbe1-0) (RJ45)

Connection method: Use a network cable (cat7a) to connect the 10Gb network ports of the NAS and Orin, and mount the NAS to Orin through the NFS protocol

The speed of using dd to test IO is only about 5Gbps

Using the same method to connect NAS and PC, the IO speed tested by dd can reach 10Gbps

Why is the transmission performance of Orin’s 10Gb network only about half? Is it a phy issue or a firmware issue?

Regards

Please investigate the outputs of running the ‘ethtool’ tool on both network interfaces of the devkit and the PC. This should provide insight into any discrepancies in performance between the two setups.

Hello @VickNV ,
Thank you for your reply.
For more testing information:
Prepare using the same NAS device connected to PC and Drive AGX Orin.
Set all the eth MTU to 9000.
The speed of network card negotiation seems to be correct, but there is a significant difference in the results of testing with dd:

1. NAS + Drive_AGX_Orin_10Gb_mgbe1_0:
1.1 dd 4k:


1.2 dd 8k:

1.3 ethtool:

2.NAS + PC_10Gb
2.1 dd 4k:


2.2 dd 8k:

2.3 ethtool:

I’ll check this with our team and get back to you.

Hello @VickNV ,
Did you get any new updates from your team?

Thanks

If you have access to the performance reports through NVOnline, the network performance (throughput) on the10G interface meets the expectation, indicating that the network stack on Orin Linux is not the bottleneck. Additionaly, the performance of the dd command on USB 3.1 type-C matches the specified performance.

To confirm or measure the throughput on the 10 Gbps interface, could you measure the throughput between the Orin device and a PC with a 10G interface?

Also, could you provide information on which NAS software is installed on Orin Linux to enable the NAS functionality?

Hello @VickNV ,
No special software is installed on Orin Linux except for the NFS v4.
We only need to mount the NAS device to a directory in Orin linux using the mount command, for example: sudo mount -t nfs 192.168.1.10:/volume /media/nas/

So we will make some other testing to see if we can find the bottleneck.

Could you please provide details on how you installed the NFS v4 software?

Hello @VickNV ,
I remember that DriveOS already has NFS installed by default, so we don’t need to install any software.
We only need to mount the NAS device to a specific directory in DriveOrin,then we can use the dd command to test the read and write speed.

You shared the result of the dd copy to the NAS. Have you tried dd copying from the NAS? Are you still experiencing worse performance compared to your PC?

Orin reads slightly faster than it writes, but overall it is still slower than PC, with a read and write speed of up to 10Gbps

Orin Read NAS Testing:
(Read to /dev/null We will get a very high rate, I think this should be unreliable. )

PC cannot provide specific test data at the moment because there are other tasks running

Thank you for confirming.

Please try this when the PC is available.

1.About the throughput between the Orin device and a PC with a 10G interface,
I have tested it before, which is around 8-9.4Gbps (different versions of driveos will have different throughput), and it doesn’t seem like a big problem.

  1. About the USB3.1, I also tested the write speed with dd and found that the performance on orin (type-c) was also very poor, about 16-45MB/s (I have tried three orins, DriveOS 6.0.3/8/9), and the same USB-SSD on PC was about 450MB/s~2.4GB/s
    Orin:

    PC:

Can you share the testing results of your USB 3.1?

Summarize the testing information above:


NAS
NFS-Write/Read
cpu load of dd
(orin/PC)
Throughput 10Gb
(nas to orin/pc)
USB3.1 SSD
to Orin W: 450MB/s
R: 540MB/s
75% about: 8~9.4Gbps
DriveOS 6.0.6/8/9
16.4~45.2MB/s
to PC W: 1GB/s
R: 1GB/s
40% about: 8~9.4Gbps
DriveOS 6.0.6/8/9
450~2.4GB/s

These test results were all run in the orin idle state without any other processes affecting IO performance (and different orin devices and d

riveos versions were tried). Currently, it seems that a 10G interface may not be the direct cause of poor NAS writing performance, but it may be due to some system configurations causing IO performance issues in the orin.

Yes, our performance report (you may have access because it seems that you have access to 6.0.9) also indicates that Ethernet performance meets the expectations.

For the USB performance topic, could you create another topic linked to this one?

@xiaolei.qin Please inform me once you create a new topic regarding the issue with USB storage performance.

Hello @VickNV ,
The USB3.1 performance issue please refer to USB3.1 storage performance issue - DRIVE AGX Orin / DRIVE AGX Orin General - NVIDIA Developer Forums

Are we still discussing this issue here?

1 Like

Seems got improved at USB3.1 storage performance issue - DRIVE AGX Orin / DRIVE AGX Orin General - NVIDIA Developer Forums

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.