Installed HighPoint SSD7505 PCie 4.0 x16 On Xavier AGX get less than x8 lanes performance

mhwang1 · May 31, 2021, 9:48am

Hi,

Sorry for replying so late,but I have confirmed my operation so many times, replacing dtb and kernel Image files and refreshing the device, and It has prompted that the kernel and the kernel-dtb have been updated successfully, but the iommus is still not disabled.

Thanks
Highpoint Technologies, Inc.

JerryChang · May 31, 2021, 10:35am

hello mhwang1,

could you please examine sysnode for the settings.
please check all your iommus configuration has applied correctly.
thanks

mhwang1 · June 2, 2021, 8:00am

Hi,

I updated the kernel and DTB according to the following steps,

I downloaded the source package from https://developer.nvidia.com/embedded/l4t/r32_release_v5.1/r32_release_v5.1/sources/t186/public_sources.tbz2 , unzipped the source package in the /Linux_for_Tegra folder and unzipped the kernel_src.tbz2 to extract the kernel source code
/Linux_for_Tegra/source/public/kernel/kernel-4.9/
After patching to modify the arm-smmu-t19x.c and tegra194-soc-pcie.dtsi files and setting the compilation output path,
make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
After success,
make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j8
After the compilation, the Image file and dts folder are generated,
I replaced the Image file and dtb files to the burn directory，and
make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=/Linux_for_Tegra/rootfs
to install kernel modules
Then Xavier enters forced recovery mode to connect to the host machine
to flash the kernel:
sudo ./flash.sh -k kernel jetson-xavier mmcblk0p1 refresh the kernel
to flash the dtbs:
sudo ./flash.sh -k kernel-dtb jetson-xavier mmcblk0p1 refresh dtb
after success it shows on the screen that
The [kernel] has been updated successfully.
The [kernel-dtb] has been updated successfully.

Please check if there is any problem with my operation steps, can I get the result of updating the kernel and dtbs?

Thanks
Highpoint Technologies, Inc.

JerryChang · June 2, 2021, 8:58am

hello mhwang1,

those looks correct steps to flash kernel and dtb partition.
however, it’s CBoot functionality includes a default booting scan sequence.
please see /boot/extlinux/extlinux.conf, it’ll loads the binary file from filesystem if the entry has specified.
for example, it loads the kernel binary file from the LINUX entry, and kernel-dtb binary file from FDT entry
thanks

mhwang1 · June 7, 2021, 6:19am

Hi,

I have successfully updated the kernel and DTB, and checked that iommu has been disabled. Then I installed the SSD7505 which has four Samsung 980 PRO on it to AGX Xavier to test the read and write performance. What is puzzling is that the test results show that the 2m-seq-read performance of a single disk is very low. Parallel read and write performance of 4 disks is also very low.

Single NVMe
2m-seq-read 1276MB/s
2m-seq-write 4915MB/s
Four NVMe Parallel
2m-seq-read 1277MB/s
2m-seq-write 3506MB/s

So from the results,I don’t think that disabling iommu has resulted in performance improvements.

Thanks
Highpoint Technologies, Inc.

Manikanta · June 7, 2021, 6:55am

Hi,

We verified all the parameters which can impact performance. The only difference now is CPU architecture. Is it possible to verify on any other arm platform?

Thanks,
Manikanta

mhwang1 · June 7, 2021, 8:45am

Hi,

We do not have pcie4.0 arm platform for us to verify and test at the moment, is there any arm platform with pcie4.0 slot that can be recommended?

Thanks
Highpoint Technologies, Inc.

Manikanta · June 7, 2021, 9:21am

I don’t have any suggestions on gen4 supported arm platforms.

BTW, have you tried maxing out CPU and memory clocks?

mhwang1 · June 8, 2021, 2:15am

Hi,

No,I have not, so what can I do to max out CPU and memory clocks for xavier agx, could u please provide some steps on how to implement it？

Thanks
Highpoint Technologies, Inc.

JerryChang · June 8, 2021, 5:39am

hello mhwang1,

please refer to Maximizing Jetson Xavier Performance. you can execute jetson_clocks.sh script to maximize a Jetson AGX Xavier device’s performance.
please also refer to Supported Modes and Power Efficiency for the supported NVPModel clock configurations. you may display current power modes as following, $ sudo /usr/sbin/nvpmodel -q.
thanks

mhwang1 · June 8, 2021, 6:54am

Hi,

I check the information of jetson_clocks --show, which is

SOC family:tegra194  Machine:Jetson-AGX
Online CPUs: 0-7
cpu0: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu1: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu2: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu3: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu4: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu5: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu6: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
cpu7: Online=1 Governor=schedutil MinFreq=1190400 MaxFreq=1190400 CurrentFreq=1190400 IdleStates: C1=1 c6=1 
GPU MinFreq=318750000 MaxFreq=905250000 CurrentFreq=318750000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=0
Fan: PWM=200
NV Power Mode: MODE_30W_ALL

Thanks
Highpoint Technologies, Inc.

Manikanta · June 10, 2021, 3:25pm

Hi,

First select MAXN mode in nvpmodel, it will increase the maxFreq from 1.2GHz to 2.2 GHz. Then use jetson_clocks to boost the CPU frequency.

Thanks,
Manikanta

mhwang1 · June 11, 2021, 6:42am

Hi,

I re-burned the system without modifying the kernel and dtb files, then selected MAXN mode in nvpmodel, used jetson_clocks to boost the CPU frequency.

SOC family:tegra194  Machine:Jetson-AGX
Online CPUs: 0-7
cpu0: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu1: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu2: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu3: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu4: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu5: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu6: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
cpu7: Online=1 Governor=schedutil MinFreq=2265600 MaxFreq=2265600 CurrentFreq=2265600 IdleStates: C1=0 c6=0 
GPU MinFreq=1377000000 MaxFreq=1377000000 CurrentFreq=1377000000
EMC MinFreq=204000000 MaxFreq=2133000000 CurrentFreq=2133000000 FreqOverride=1
Fan: PWM=200
NV Power Mode: MAXN

After the preparation was completed I did a fio performance test.
HighPoint SSD7505 connected with 4 Samsung 980PRO NVMe
2m-seq-read 11.1GB/s,
2m-seq-write 12.2GB/s

Thanks
Highpoint Technologies, Inc.

Manikanta · June 11, 2021, 6:54am

These numbers are now matching with numbers on AMD, reported in Installed HighPoint SSD7505 PCie 4.0 x16 On Xavier AGX get less than x8 lanes performance - #5 by mhwang1

Marking MAXN as the solution.