NVMe SSD write performance degradation between TX1 and TX2

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

I have a J120 (rev11) base board with a 1TB EVO 970 PRO installed. I’ve measured the write performance using

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync

Write performance using a TX1 (Jetpack 4.3.1) is around 430 MB/s.

Running the same command on a TX2 - same board, same SSD I only get 230 MB/s.

I’ve used the https://auvidea.eu/download/firmware/TX2/v2.2/J90_v2.2_4.2.2.tar.bz2 from AUVIDEA to get the USB on the board working.

lspci -vv reports “Width x4” on both setups. I’ve also tried running jetsonclocks.

I am running a bit out of ideas. One thing I did notice is that the plugin-manager seems to have changed on 4.3.1, and that it doesn’t seem to apply any of the fragments - nothing is reported on dmesg, as I believe it used to. I also couldn’t find any documentation on what changed there. E.g. I believe ‘pcie0_lane2_mux’ is disabled.

2

Hi jschwendner,

Could you put this test on nvidia devkit and share the result again?

pcie0_lane2_mux should not matter here. If this issue is related to this mux, then the pcie device would just be disabled.

3

What would be the best way to connect the NVMe to the devkit? The M.2 slot on the board does not seem to fit a full size M.2 I believe.

5

I have connected the ssd to the dev-board now, using a pcie-nvme adapter. The result is the same. I’ve also noticed that when using O_DIRECT as parameter in the dd, the write speeds are around 700 MB/s. The values that I would actually expect from the SSD.
I’ve tested on both tx1 and tx2 using the same 4.9 kernel, and could still see the difference as described above.

8

This is still an issue for us, and even more so because TX1 has been phased out.
I found out that the CPU frequency does have an impact. With power mode set to 2:

# nvpmodel -m 2
# dd if=/dev/zero of=nvme0n1 bs=2M count=500
500+0 records in
500+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.43424 s, 163 MB/s

and now with power mode 0

# nvpmodel -m 0
# dd if=/dev/zero of=nvme0n1 bs=2M count=500
500+0 records in
500+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 4.50262 s, 233 MB/s