I’ve been benchmarking NVMe and USB3.1 disk performance on the Xavier NX. Our application involves capturing video data and storing it on disk.
The tested hardware setup included:
- Samsung 980 SSD (model MZ-V8V1T0)
- Sabrent EC-SNVE enclosure
- Jetson Xavier NX
- Mac Mini M1 (as a reference point / for comparison)
We’ve tested three hardware configurations:
- The SSD installed directly in the NX’s NVMe slot
- The SSD plugged in to the USB3.1 port on the NX, using the enclosure
- The SSD plugged in to the Thunderbolt 3 port on the Mac Mini, using the same enclosure
- Xavier NX was in the 20W 6-core mode
- The disk/partition was correctly identified by the OS at /dev/nvme0n1p1 when plugged directly into the NVMe slot
- The system was stripped of most inessential software, unnecessary services were disabled (including the X11 GUI), the test was performed over an SSH session, to minimise interference
The setup on the software side involved the following:
- A new tmpfs (ram-disk) file system was created using: mount -t tmpfs tmpfs /tmp/tmp
- A test file sized 4071313408 bytes (approx 4.1GB) was created under /tmp/tmp/test, using data from /dev/urandom
- The SSD was formatted using ext4 filesystem and mounted under /mnt
- The file was copied using nocache ionice -c1 dd bs=1M if=/tmp/tmp/test of=/mnt/test (skip page cache, use realtime IO priority, use 1M block size with dd)
- The test was repeated three times, discarding the best and worst results
On the Mac side, we’ve used a similar setup, but formatted the drive using APFS, and used the built-in storage instead of ram-disk as a data source (as the internal disk could easily keep up).
Since our target application is video capture and recording, we consider a sequential write (as with dd) to be an accurate approximation of a production IO workload.
The results for the three hardware configurations, as reported by dd (note the output on macOS is reported in a slightly different format, due to the system’s BSD-based userland):
- NX + NVMe: 7,43719 s, 547 MB/s
- NX + USB3.1: 7,19526 s, 566 MB/s
- Mac Mini: 4.256440 secs (956506724 bytes/sec)
Observations: the performance gap between NVMe and USB3.1 on the NX seems extremely small, and the recorded difference could be due to an error in the method, small data set, other activity on the system, etc.
Question: is this the maximum of what we are to expect from NVMe write performance on the Xavier NX? It seems like both the disk and the enclosure are capable of higher transfer rates. Also noting the similar performance between NVMe and USB3.1, we suspect, that there’s a shared bottleneck between these physical paths. Or is there something wrong about our method? What else could we do to try to maximise write performance?