Bad memory performance on JetPack 5.0.2 (5.10.104-tegra)

Memory performance has declined severely with the new kernel.

Could you please advice to kernel settings that restore performance to the JetPack 4.4.1 level (4.9.140-tegra)?

Performance comparison:
sudo mount tmpfs -ttmpfs -osize=4G /mnt/
sudo dd if=/dev/zero of=/mnt/zero.bin bs=1M count=4000

  • JP441: 4194304000 bytes (4.2 GB, 3.9 GiB) copied, 1.55166 s, 2.7 GB/s
  • JP502: 4194304000 bytes (4,2 GB, 3,9 GiB) copied, 3,9952 s, 1,0 GB/s

sudo dd if=/mnt/zero.bin of=/dev/null bs=1M

  • JP441: 4194304000 bytes (4.2 GB, 3.9 GiB) copied, 0.626467 s, 6.7 GB/s
  • JP502: 4194304000 bytes (4,2 GB, 3,9 GiB) copied, 0,76411 s, 5,5 GB/s

for OPER in write read; do sysbench --threads=$(nproc) --memory-oper=${OPER} memory run; done

  • JP441: 2271.58 MiB/sec / 20623.23 MiB/sec
  • JP502: 1869.11 MiB/sec / 12256.02 MiB/sec

Could you boost clocks to try.

sudo nvpmodel -m 0
sudo jetson_clocks
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

No change at all unfortunatly.

Could you check if the emc clock with J4.x

cat /sys/kernel/debug/bpmp/debug/clk/emc/rate

It’s lower than for JP5.

JP411

$ sudo cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate
1600000000
$ sudo cat /sys/kernel/debug/bpmp/debug/clk/emc/rate
1600000000

Could you check the /etc/nvpmode.conf to confirm the select the Max power mode like

POWER_MODEL ID=8 NAME=MODE_20W_6CORE

Yes, this is already verified to not be the culprit.
I’m running the same mode on both 4.1.1 and 5.0.2.

The 5.0.2 default configuration does not however contain all modes, so needed to add them manually.

Do you have any other suggestions?

I’ve tried running both kernels on the same rootfs, still get the poor performance on the jp5 kernel.
Everything points to a kernel problem, not a configuration problem.

Do you have any kernel settings to improve performance?