Emulation Flash Configurations not working with Jetpack 6.0

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
1

Hi,

we tried flashing our Orin-Devkit with the following command:

sudo ./flash.sh jetson-agx-orin-devkit-as-jao-32gb internal

but when checking e.g. with tegrastats, we see that 12 cores are active (which should be only 8 in the emulation mode):

$ tegrastats
11-21-2024 17:16:39 RAM 2059/30697MB (lfb 5x4MB) SWAP 0/15349MB (cached 0MB) CPU [0%@729,0%@729,0%@729,0%@729,0%@729,0%@729,0%@729,0%@729,0%@729,0%@729,0%@729,0%@729] GR3D_FREQ 0% cpu@44.5C tboard@31.5C soc2@41.687C tdiode@36.25C soc0@41.75C tj@44.593C soc1@41.562C VDD_GPU_SOC 2379mW/2379mW VDD_CPU_CV 396mW/396mW VIN_SYS_5V0 3824mW/3824mW VDDQ_VDD2_1V8AO 402mW/402mW

also other tools, like jtop are showing 12 active cores.

/boot/extlinux/extlinux.conf shows the correct configuration though:

TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} root=PARTUUID=d8da58d9-f573-4d70-916a-f2a869e5f66e rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb video=efifb:off console=tty0 nr_cpus=8

also, /etc/nv_boot_control.conf shows, that the emulation mode should be active:

$ cat /etc/nv_boot_control.conf
TNSPEC 3701-500-0000-J.0-1-1-jetson-agx-orin-devkit-as-jao-32gb-
COMPATIBLE_SPEC 3701-300-0000--1--jetson-agx-orin-devkit-as-jao-32gb-
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0`

Do you have any advice what the problem might be here? We need to configure the Orin-devkit to have the same hardware capabilities as the 32GB Module (regarding GPU and CPU).

We used the Orin devkit in the past with Jetpack 5 for example, to build engines and deployed them successfully on the 32GB module, and we would like to do this with Jetpack 6.0 as well.

2

Hi,
Here are some suggestions for the common issues:

1. Performance

Please run the below command before benchmarking deep learning use case:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

2. Installation

Installation guide of deep learning frameworks on Jetson:

3. Tutorial

Startup deep learning tutorial:

4. Report issue

If these suggestions don’t help and you want to report an issue to us, please attach the model, command/step, and the customized app (if any) with us to reproduce locally.

Thanks!

3

Thanks for the quick response.

I think there was a misunderstanding of my post above, let me try to clarify:

  • We have a Jetson Orin DevKit .
  • We want to use it to emulate a Jetson AGX Orin module 32GB, so we flashed it with Jetpack 6.0 following Emulation Flash Configurations — NVIDIA Jetson Linux Developer Guide 1 documentation with the jetson-agx-orin-devkit-as-jao-32gb parameter (see my post above)
  • This did not seem to work the same way as it did with Jetpack 5.0; after flashing with above command, we see that the DevKit still has 12 CPU cores active instead of 8 as before.

So, we basically want to emulate the 32GB variant with our DevKit using Jetpack6.0.

Are we missing a step? Or does it work differently now?

5

Hi user84666,

Do you mean that you have only AGX Orin 64GB module but you want to know the status for AGX Orin 32GB so that you are using jetson-agx-orin-devkit-as-jao-32gb board config?

Have you also tried to configure them in nvpmodel configuration(/etc/nvpmodel.conf)?
Please share the result of # nvpmodel -q --verbose on your board.

6

Sorry, I was mistaken on the hardware model we have (I’ll edit it in my posts above as well). We have an older AGX Orin DevKit, which in fact has only 32GB.

We want to use that to emulate a Jetson AGX Orin 32GB module, as described in Emulation Flash Configurations — NVIDIA Jetson Linux Developer Guide 1 documentation.

Here the output of nvpmodel -q --verbose. This is after we flashed it with sudo ./flash.sh jetson-agx-orin-devkit-as-jao-32gb internal as per the documentation linked in my previous paragraph:

$ nvpmodel -q --verbose
NVPM VERB: Config file: /etc/nvpmodel.conf
NVPM VERB: parsing done for /etc/nvpmodel.conf
NVPM VERB: Current mode: NV Power Mode: MAXN
0
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_0: PATH /sys/devices/system/cpu/cpu0/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_1: PATH /sys/devices/system/cpu/cpu1/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_2: PATH /sys/devices/system/cpu/cpu2/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_3: PATH /sys/devices/system/cpu/cpu3/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_4: PATH /sys/devices/system/cpu/cpu4/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_5: PATH /sys/devices/system/cpu/cpu5/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_6: PATH /sys/devices/system/cpu/cpu6/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM CPU_ONLINE: ARG CORE_7: PATH /sys/devices/system/cpu/cpu7/online: REAL_VAL: 1 CONF_VAL: 1
NVPM VERB: PARAM TPC_POWER_GATING: ARG TPC_PG_MASK: PATH /sys/devices/platform/gpu.0/tpc_pg_mask: REAL_VAL: 128 CONF_VAL: 128
NVPM VERB: PARAM GPU_POWER_CONTROL_ENABLE: ARG GPU_PWR_CNTL_EN: PATH /sys/devices/platform/gpu.0/power/control: REAL_VAL: auto CONF_VAL: on
NVPM VERB: PARAM CPU_A78_0: ARG MIN_FREQ: PATH /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq: REAL_VAL: 729600 CONF_VAL: 729600
NVPM VERB: PARAM CPU_A78_0: ARG MAX_FREQ: PATH /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: REAL_VAL: 2188800 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM CPU_A78_1: ARG MIN_FREQ: PATH /sys/devices/system/cpu/cpu4/cpufreq/scaling_min_freq: REAL_VAL: 729600 CONF_VAL: 729600
NVPM VERB: PARAM CPU_A78_1: ARG MAX_FREQ: PATH /sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq: REAL_VAL: 2188800 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM GPU: ARG MIN_FREQ: PATH /sys/devices/platform/17000000.gpu/devfreq_dev/min_freq: REAL_VAL: 306000000 CONF_VAL: 0
NVPM VERB: PARAM GPU: ARG MAX_FREQ: PATH /sys/devices/platform/17000000.gpu/devfreq_dev/max_freq: REAL_VAL: 930750000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM GPU_POWER_CONTROL_DISABLE: ARG GPU_PWR_CNTL_DIS: PATH /sys/devices/platform/gpu.0/power/control: REAL_VAL: auto CONF_VAL: auto
NVPM VERB: PARAM EMC: ARG MAX_FREQ: PATH /sys/kernel/nvpmodel_clk_cap/emc: REAL_VAL: 3199000000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM DLA0_CORE: ARG MAX_FREQ: PATH /sys/devices/platform/bus@0/13e00000.host1x/15880000.nvdla0/clk_cap/dla0_core: REAL_VAL: 1408000000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM DLA1_CORE: ARG MAX_FREQ: PATH /sys/devices/platform/bus@0/13e00000.host1x/158c0000.nvdla1/clk_cap/dla1_core: REAL_VAL: 1408000000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM DLA0_FALCON: ARG MAX_FREQ: PATH /sys/devices/platform/bus@0/13e00000.host1x/15880000.nvdla0/clk_cap/dla0_falcon: REAL_VAL: 742400000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM DLA1_FALCON: ARG MAX_FREQ: PATH /sys/devices/platform/bus@0/13e00000.host1x/158c0000.nvdla1/clk_cap/dla1_falcon: REAL_VAL: 742400000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM PVA0_VPS: ARG MAX_FREQ: PATH /sys/devices/platform/bus@0/13e00000.host1x/16000000.pva0/clk_cap/pva0_vps: REAL_VAL: 1190400000 CONF_VAL: 9223372036854775807
NVPM VERB: PARAM PVA0_AXI: ARG MAX_FREQ: PATH /sys/devices/platform/bus@0/13e00000.host1x/16000000.pva0/clk_cap/pva0_cpu_axi: REAL_VAL: 857600000 CONF_VAL: 9223372036854775807

There are indeed only 8 cores listed, however jtop, tegrastats (see my post above) htop report 12 active cores.

Does this mean the emulation mode is already active? Or is some additional step needed?
When we did the emulation flashing with JetPack 5, it showed us only 8 cores in jtop, without having to reconfigure the nvpmodel config.

7

Are we the only ones that have a problem with the emulation mode on Jetpack6?
It seems like a pretty important feature to have for the DevKits.

8

Sorry that I’m not clear about this.
If your AGX Orin devkit has AGX Orin 32GB module, why you want to use jetson-agx-orin-devkit-as-jao-32gb as board config.

Please share the result of cat /etc/nv_boot_control.conf for further check.

Not too many users do this and we don’t receive similar issue like it so that we would like to also verify it locally.

10

Hi,

Actually the emulation config is seldom in use as all the modules are released already.

We will check this on our side.

12

Hi,

It seems we are not able to reproduce this issue on our side with “jetson-agx-orin-devkit-as-jao-32gb”.

What is the result of cat /etc/nv_boot_control.conf for your board if you flash the common use board config?

13

Hi @KevinFFF, hi @WayneWWW,

thanks for the replies! The AGX Orin devkit with 32 GB memory has the same hardware capabilities (despite of the memory) as the 64GB module. The AGX Orin devkit has e.g., 12 CPU cores (instead of 8), 2048 tensor cores (instead of 1792), higher GPU frequencies, and so on. See NVIDIA Jetson AGX Orin

We want to have the same hardware emulated as we are building into our robots. We use the devkit for instance to evaluate the performance of our models in simulation. If the DevKit is faster than the modules on the robots, we might get better results as we will have in reality afterwards. That is why is is so important to us to have the same hardware capabilities emulated on the devkit.

Regarding the results of cat /etc/nv_boot_control.conf: I already shared them in the initial post, here once again the results:

$ cat /etc/nv_boot_control.conf
TNSPEC 3701-500-0000-J.0-1-1-jetson-agx-orin-devkit-as-jao-32gb-
COMPATIBLE_SPEC 3701-300-0000--1--jetson-agx-orin-devkit-as-jao-32gb-
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0
14

Hi,

No, I mean what is the result if you don’t flash with “jetson-agx-orin-devkit-as-jao-32gb-”.

I didn’t ask you to share a result based on jetson-agx-orin-devkit-as-jao-32gb. I want the original board config.

15

And once you shared out the result, please also share me the flash log of “jetson-agx-orin-devkit-as-jao-32gb” for this module. Attach the log as whole text file and put it here.

16

Hi @WayneWWW,

sorry for the confusion, here is the output with the original board config:

$ cat /etc/nv_boot_control.conf
TNSPEC 3701-500-0000-J.0-1-1-jetson-agx-orin-devkit-
COMPATIBLE_SPEC 3701-300-0000--1--jetson-agx-orin-devkit-
TEGRA_BOOT_STORAGE mmcblk0
TEGRA_CHIPID 0x23
TEGRA_OTA_BOOT_DEVICE /dev/mtdblock0
TEGRA_OTA_GPT_DEVICE /dev/mtdblock0

And here is the full log when flashing the devkit as jao-32gb:
flash_log_jetson-agx-orin-devkit-as-jao-32gb.txt (99.8 KB)

18

Do you have the dmesg to share?

19

Sure:
dmesg.log (63.2 KB)

20

there should be a kernel cmdline of 8 cores cpu. But didn’t show up in your log.

 33         OVERLAY_DTB_FILE+=",tegra234-p3737-0000+p3701-0000-as-p3701-0004.dtbo";
 34         # Select the Base DTB based on SKU
 35         if [ "${board_sku}" = "0000" ]; then
 36                 BPFDTB_FILE="tegra234-bpmp-3701-0000-as-3701-0004-3737-0000.dtb";
 37                 CMDLINE_ADD+=" nr_cpus=8";
21

It seems to be at least successfully added to /boot/extlinux/extlinux.conf:
(here, I added the .txt appendix to the file, because I was not allowed to upload a .conf file, but the file ist the original /boot/extlinux/extlinux.conf and shows the nr_cpus=8)
extlinux.conf.txt (985 Bytes)

Any idea what the issue might be?