Bad gpu performance on some nano boards (jetpack-4.6)

Here you are :

[0000.240] [L4T TegraBoot] (version 00.00.2018.01-l4t-8728f3cb)
[0000.245] Processing in cold boot mode Bootloader 2
[0000.250] A02 Bootrom Patch rev = 1023
[0000.253] Power-up reason: pmc por
[0000.257] No Battery Present
[0000.259] pmic max77620 reset reason
[0000.263] pmic max77620 NVERC : 0x50
[0000.266] RamCode = 0
[0000.268] Platform has DDR4 type RAM
[0000.272] max77620 disabling SD1 Remote Sense
[0000.276] Setting DDR voltage to 1125mv
[0000.280] Serial Number of Pmic Max77663: 0x1e37ed
[0000.287] Entering ramdump check
[0000.290] Get RamDumpCarveOut = 0x0
[0000.294] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.299] Last reboot was clean, booting normally!
[0000.303] Sdram initialization is successful
[0000.308] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.314] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.319] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.325] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.331] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.337] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.343] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.349] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.365] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.371] Platform-DebugCarveout: 0
[0000.374] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.380] Non secure mode, and RB not enabled.
[0000.384] *** Booting BFS0.
[0000.387] Initialize FailControl
[0000.390] Read PT from (0:3)
[0000.405] Csd NumOfBlocks=0
[0000.444] PT crc32 and magic check passed.
[0000.448] Using BFS PT to query partitions
[0000.452] PT: Partition LNX NOT found !
[0000.456] *** Booting KFS0.
[0000.458] NvTbootFailControlDoFailover: No failover; Continuing ...
[0000.464] BoardID = 3448, SKU = 0x2
[0000.468] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0000.473] Read GPT from (0:3)
[0000.482] Using GPT Primary to query partitions
[0000.486] NvTbootFailControlDoFailover: No failover; Continuing ...
[0000.492] Loading Tboot-CPU binary
[0000.498] Verifying TBC in OdmNonSecureSBK mode
[0000.508] Bootloader load address is 0xa0000000, entry address is 0xa0000258
[0000.515] Bootloader downloaded successfully.
[0000.519] Downloaded Tboot-CPU binary to 0xa0000258
[0000.525] MAX77620_GPIO5 configured
[0000.528] CPU power rail is up
[0000.531] CPU clock enabled
[0000.534] Performing RAM repair
[0000.537] Updating A64 Warmreset Address to 0xa00002e9
[0000.542] BoardID = 3448, SKU = 0x2
[0000.546] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0000.552] NvTbootFailControlDoFailover: No failover; Continuing ...
[0000.558] Loading NvTbootBootloaderDTB
[0000.575] Verifying NvTbootBootloaderDTB in OdmNonSecureSBK mode
[0000.629] Bootloader DTB Load Address: 0x83000000
[0000.633] BoardID = 3448, SKU = 0x2
[0000.636] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0000.642] Loading NvTbootKernelDTB
[0000.659] Verifying NvTbootKernelDTB in OdmNonSecureSBK mode
[0000.726] Kernel DTB Load Address: 0x83100000
[0000.731] BoardID = 3448, SKU = 0x2
[0000.734] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0000.740] NvTbootFailControlDoFailover: No failover; Continuing ...
[0000.747] Loading cboot binary
[0000.758] Verifying EBT in OdmNonSecureSBK mode
[0000.800] Bootloader load address is 0x92c00000, entry address is 0x92c00258
[0000.807] Bootloader downloaded successfully.
[0000.811] BoardID = 3448, SKU = 0x2
[0000.814] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0000.820] NvTbootFailControlDoFailover: No failover; Continuing ...
[0000.826] PT: Partition NCT NOT found !
[0000.830] Warning: Find Partition via PT Failed
[0000.834] Next binary entry address: 0x92c00258
[0000.839] BoardId: 3448
[0000.843] Overriding pmu board id with proc board id
[0000.848] Display board id is not available
[0000.852] BoardID = 3448, SKU = 0x2
[0000.856] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0000.862] NvTbootFailControlDoFailover: No failover; Continuing ...
[0000.875] Verifying SC7EntryFw in OdmNonSecureSBK mode
[0000.928] /bpmp deleted
[0000.930] SC7EntryFw header found loaded at 0xff700000
[0001.126] OVR2 PMIC
[0001.128] Bpmp FW successfully loaded
[0001.132] BoardID = 3448, SKU = 0x2
[0001.135] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0001.141] NvTbootFailControlDoFailover: No failover; Continuing ...
[0001.148] WB0 init successfully at 0xff780000
[0001.153] Verifying NvTbootWb0 in OdmNonSecureSBK mode
[0001.158] Set NvDecSticky Bits
[0001.161] GSC2 address ff53fffc value c0edbbcc
[0001.167] GSC MC Settings done
[0001.170] BoardID = 3448, SKU = 0x2
[0001.174] Not Nano-SD or !QSPI-ONLY, check GPT table first ...
[0001.180] TOS Image length 53680
[0001.183]  Monitor size 53680
[0001.186]  OS size 0
[0001.192] Secure Os AES-CMAC Verification Success!
[0001.196] TOS image cipher info: plaintext
[0001.200] Loading and Validation of Secure OS Successful
[0001.216] SC7 Entry Firmware - 0xff700000, 0x4000
[0001.221] NvTbootPackSdramParams: start.
[0001.226] NvTbootPackSdramParams: done.
[0001.229] Tegraboot started after 169706 us
[0001.233] Basic modules init took 719390 us
[0001.237] Sec Bootdevice Read Time = 54 ms, Read Size = 3682 KB
[0001.243] Sec Bootdevice Write Time = 0 ms, Write Size = 0 KB
[0001.249] Next stage binary read took 7392 us
[0001.253] Carveout took -34785 us
[0001.256] CPU initialization took 397600 us
[0001.260] Total time taken by TegraBoot 1089597 us

[0001.265] Starting CPU & Halting co-processor

64NOTICE:  BL31: v1.3(release):b5eeb33f7
NOTICE:  BL31: Built : 12:09:37, Jul 26 2021
ERROR:   Error initializing runtime service trusty_fast
[0001.387] get_emc_table_offset: failed to find EMC node
[0001.392] LPDDR4 Training: Can't find emc-table node
[0001.434]
[0001.435] Debug Init done
[0001.438] Marked DTB cacheable
[0001.441] Bootloader DTB loaded at 0x83000000
[0001.446] Marked DTB cacheable
[0001.448] Kernel DTB loaded at 0x83100000
[0001.452] DeviceTree Init done
[0001.465] Pinmux applied successfully
[0001.470] gicd_base: 0x50041000
[0001.473] gicc_base: 0x50042000
[0001.476] Interrupts Init done
[0001.480] Using base:0x60005090 & irq:208 for tick-timer
[0001.485] Using base:0x60005098 for delay-timer
[0001.490] platform_init_timer: DONE
[0001.493] Timer(tick) Init done
[0001.497] osc freq = 38400 khz
[0001.501]
[0001.502] Welcome to L4T Cboot
[0001.505]
[0001.506] Cboot Version: 00.00.2018.01-t210-c952b4e6
[0001.511] calling constructors
[0001.514] initializing heap
[0001.517] initializing threads
[0001.520] initializing timers
[0001.522] creating bootstrap completion thread
[0001.527] top of bootstrap2()
[0001.530] CPU: ARM Cortex A57
[0001.533] CPU: MIDR: 0x411FD071, MPIDR: 0x80000000
[0001.537] initializing platform
[0001.591] Config for emmc ddr50 mode completed
[0001.595] sdmmc bdev is already initialized
[0001.600] Enable APE clock
[0001.602] Un-powergate APE partition
[0001.606] of_register: registering tegra_udc to of_hal
[0001.611] of_register: registering inv20628-driver to of_hal
[0001.616] of_register: registering ads1015-driver to of_hal
[0001.622] of_register: registering lp8557-bl-driver to of_hal
[0001.628] of_register: registering bq2419x_charger to of_hal
[0001.633] of_register: registering bq27441_fuel_gauge to of_hal
[0001.645] gpio framework initialized
[0001.648] of_register: registering tca9539_gpio to of_hal
[0001.654] of_register: registering tca9539_gpio to of_hal
[0001.659] of_register: registering i2c_bus_driver to of_hal
[0001.665] of_register: registering i2c_bus_driver to of_hal
[0001.670] of_register: registering i2c_bus_driver to of_hal
[0001.676] pmic framework initialized
[0001.679] of_register: registering max77620_pmic to of_hal
[0001.685] regulator framework initialized
[0001.689] of_register: registering tps65132_bl_driver to of_hal
[0001.694] initializing target
[0001.700] gpio_driver_register: register 'tegra_gpio_driver' driver
[0001.708] board ID = D78, board SKU = 2
[0001.714] fixed regulator driver initialized
[0001.732] initializing OF layer
[0001.734] NCK carveout not present
[0001.738] Skipping dts_overrides
[0001.742] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.758] I2C Bus Init done
[0001.760] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.770] I2C Bus Init done
[0001.773] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.783] I2C Bus Init done
[0001.786] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.796] I2C Bus Init done
[0001.799] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.809] I2C Bus Init done
[0001.812] of_children_init: Ops found for compatible string maxim,max77620
[0001.822] max77620_init using irq 118
[0001.827] register 'maxim,max77620' pmic
[0001.831] gpio_driver_register: register 'max77620-gpio' driver
[0001.838] of_children_init: Ops found for compatible string nvidia,tegra210-i2c
[0001.848] I2C Bus Init done
[0001.852] NCK carveout not present
[0001.861] Find /i2c@7000c000's alias i2c0
[0001.865] get eeprom at 1-a0, size 256, type 0
[0001.874] Find /i2c@7000c500's alias i2c2
[0001.878] get eeprom at 3-a0, size 256, type 0
[0001.882] get eeprom at 3-ae, size 256, type 0
[0001.886] pm_ids_update: Updating 1,a0, size 256, type 0
[0001.892] I2C slave not started
[0001.895] I2C write failed
[0001.897] Writing offset failed
[0001.900] eeprom_init: EEPROM read failed
[0001.904] pm_ids_update: eeprom init failed
[0001.909] pm_ids_update: Updating 3,a0, size 256, type 0
[0001.939] pm_ids_update: The pm board id is 3448-0002-400
[0001.945] Adding plugin-manager/ids/3448-0002-400=/i2c@7000c500:module@0x50
[0001.953] pm_ids_update: pm id update successful
[0001.957] pm_ids_update: Updating 3,ae, size 256, type 0
[0001.963] I2C slave not started
[0001.966] I2C write failed
[0001.968] Writing offset failed
[0001.971] eeprom_init: EEPROM read failed
[0001.975] pm_ids_update: eeprom init failed
[0002.005] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.011] shim_eeprom_update_mac:267: Failed to update 0 MAC address in DTB
[0002.019] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.025] shim_eeprom_update_mac:267: Failed to update 1 MAC address in DTB
[0002.033] updating /chosen/nvidia,ethernet-mac node 00:04:4b:e9:d2:bd
[0002.039] Plugin Manager: Parse ODM data 0x000a4000
[0002.049] shim_cmdline_install: /chosen/bootargs: kmemleak=on earlycon=uart8250,mmio32,0x70006000
[0002.065] Find /i2c@7000c000's alias i2c0
[0002.069] get eeprom at 1-a0, size 256, type 0
[0002.078] Find /i2c@7000c500's alias i2c2
[0002.081] get eeprom at 3-a0, size 256, type 0
[0002.086] get eeprom at 3-ae, size 256, type 0
[0002.090] pm_ids_update: Updating 1,a0, size 256, type 0
[0002.096] I2C slave not started
[0002.099] I2C write failed
[0002.101] Writing offset failed
[0002.104] eeprom_init: EEPROM read failed
[0002.108] pm_ids_update: eeprom init failed
[0002.112] pm_ids_update: Updating 3,a0, size 256, type 0
[0002.142] pm_ids_update: The pm board id is 3448-0002-400
[0002.149] Adding plugin-manager/ids/3448-0002-400=/i2c@7000c500:module@0x50
[0002.156] pm_ids_update: pm id update successful
[0002.160] pm_ids_update: Updating 3,ae, size 256, type 0
[0002.166] I2C slave not started
[0002.169] I2C write failed
[0002.171] Writing offset failed
[0002.174] eeprom_init: EEPROM read failed
[0002.178] pm_ids_update: eeprom init failed
[0002.208] Add serial number:1423019115000 as DT property
[0002.215] Applying platform configs
[0002.221] platform-init is not present. Skipping
[0002.226] calling apps_init()
[0002.247] Found 17 GPT partitions in "sdmmc3_user"
[0002.252] Proceeding to Cold Boot
[0002.255] starting app android_boot_app
[0002.259] Device state: unlocked
[0002.262] display console init
[0002.267] status of sor1 node is not okay
[0002.271] sor0 is not supported
[0002.274] display_console_init: no valid display out_type
[0002.282] subnode volume_up is not found !
[0002.286] subnode back is not found !
[0002.290] subnode volume_down is not found !
[0002.294] subnode menu is not found !
[0002.297] Gpio keyboard init success
[0002.305] DT entry for leds-pwm not found
[0002.351] found decompressor handler: lz4-legacy
[0002.365] decompressing blob (type 1)...
[0002.431] display_resolution: No display init
[0002.435] Failed to retrieve display resolution
[0002.440] Could not load/initialize BMP blob...ignoring
[0002.490] decompressor handler not found
[0002.493] load_firmware_blob: Firmware blob loaded, entries=2
[0002.499] XUSB blob version 0 size 126464 @ 0x92cb328c
[0002.505] -------> se_aes_verify_sbk_clear: 747
[0002.509] se_aes_verify_sbk_clear: Error
[0002.513] SE operation failed
[0002.516] bl_battery_charging: connected to external power supply
[0002.525] display_console_ioctl: No display init
[0002.530] switch_backlight failed
[0002.539] device_query_partition_size: failed to open partition sdmmc3_user:MSC !
[0002.547] MSC Partition not found
[0002.556] device_query_partition_size: failed to open partition sdmmc3_user:USP !
[0002.564] USP partition read failed!
[0002.567] blob_init: blob-partition USP header read failed
[0002.573] android_boot Unable to update recovery partition
[0002.578] kfs_getpartname: name = LNX
[0002.582] Loading kernel from LNX
[0002.675] load kernel from storage
[0002.685] decompressor handler not found
[0002.719] Successfully loaded kernel and ramdisk images
[0002.724] board ID = D78, board SKU = 2
[0002.728] display_resolution: No display init
[0002.733] Failed to retrieve display resolution
[0002.737] bmp blob is not loaded and initialized
[0002.742] Failed to display boot-logo
[0002.745] NCK carveout not present
[0002.749] Skipping dts_overrides
[0002.752] NCK carveout not present
[0002.762] Find /i2c@7000c000's alias i2c0
[0002.765] get eeprom at 1-a0, size 256, type 0
[0002.774] Find /i2c@7000c500's alias i2c2
[0002.778] get eeprom at 3-a0, size 256, type 0
[0002.783] get eeprom at 3-ae, size 256, type 0
[0002.787] pm_ids_update: Updating 1,a0, size 256, type 0
[0002.792] I2C slave not started
[0002.795] I2C write failed
[0002.798] Writing offset failed
[0002.801] eeprom_init: EEPROM read failed
[0002.805] pm_ids_update: eeprom init failed
[0002.809] pm_ids_update: Updating 3,a0, size 256, type 0
[0002.839] pm_ids_update: The pm board id is 3448-0002-400
[0002.846] Adding plugin-manager/ids/3448-0002-400=/i2c@7000c500:module@0x50
[0002.854] pm_ids_update: pm id update successful
[0002.859] pm_ids_update: Updating 3,ae, size 256, type 0
[0002.864] I2C slave not started
[0002.867] I2C write failed
[0002.870] Writing offset failed
[0002.873] eeprom_init: EEPROM read failed
[0002.877] pm_ids_update: eeprom init failed
[0002.907] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.912] shim_eeprom_update_mac:267: Failed to update 0 MAC address in DTB
[0002.920] eeprom_get_mac: EEPROM invalid MAC address (all 0xff)
[0002.926] shim_eeprom_update_mac:267: Failed to update 1 MAC address in DTB
[0002.934] updating /chosen/nvidia,ethernet-mac node 00:04:4b:e9:d2:bd
[0002.941] Plugin Manager: Parse ODM data 0x000a4000
[0002.952] shim_cmdline_install: /chosen/bootargs: kmemleak=on earlycon=uart8250,mmio32,0x70006000
[0002.962] Add serial number:1423019115000 as DT property
[0002.970] "bpmp" doesn't exist, creating
[0002.976] Updated bpmp info to DTB
[0002.981] Updated initrd info to DTB
[0002.984] "proc-board" doesn't exist, creating
[0002.990] Updated board info to DTB
[0002.993] "pmu-board" doesn't exist, creating
[0002.999] Updated board info to DTB
[0003.003] "display-board" doesn't exist, creating
[0003.008] Updated board info to DTB
[0003.012] "reset" doesn't exist, creating
[0003.016] Updated reset info to DTB
[0003.020] display_console_ioctl: No display init
[0003.024] display_console_ioctl: No display init
[0003.029] display_console_ioctl: No display init
[0003.033] Cmdline: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 gpt
[0003.068] DTB cmdline: kmemleak=on earlycon=uart8250,mmio32,0x70006000
[0003.075] boot image cmdline: root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1
[0003.091] Updated bootarg info to DTB
[0003.095] Adding uuid 0000000164449587240000000e018300 to DT
[0003.100] Adding eks info 0 to DT
[0003.106] WARNING: Failed to pass NS DRAM ranges to TOS, err: -7
[0003.112] Updated memory info to DTB
[0003.120] set vdd_core voltage to 1075 mv
[0003.124] setting 'vdd-core' regulator to 1075000 micro volts
[0003.129] Found secure-pmc; disable BPMP


U-Boot 2020.04-g46e4604c78 (Jul 26 2021 - 12:09:42 -0700)

SoC: tegra210
Model: NVIDIA Jetson Nano Developer Kit
Board: NVIDIA P3450-0000
DRAM:  4 GiB
MMC:   sdhci@700b0000: 1, sdhci@700b0600: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
582 bytes read in 38 ms (14.6 KiB/s)
p2771-0000 eMMC boot options
1:      primary kernel
2:      recovery kernel
Enter choice:
  1. Please put this module back to devkit and see if same line gets printed .

  2. Please dump below info

sudo i2cdump -y 2 0x50

Sorry WayneWWW, I do not have a devkit at hand.

nvidia@nano-jp46:~$ sudo i2cdump -y 2 0x50
[sudo] password for nvidia:
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 02 00 04 46 00 00 00 00 00 00    ?.?.x??.?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 32 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    02-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff bd d2 e9 4b 04 00 31 34 32 33 30 31    ....???K?.142301
50: 39 31 31 35 30 30 30 00 00 00 00 00 00 00 00 00    9115000.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff bd d2 e9 4b    ............???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 95    ...............?
nvidia@nano-jp46:~$

Hello WayneWW,

best wishes for this new year.

Could my problem be related to the following terse instruction I found in https://developer.download.nvidia.com/embedded/L4T/r32-2_Release_v1.0/Tegra_Linux_Driver_Package_Nano_Adaptation_Guide.pdf?RAf5NHWjgLJHGdWmk0MXQ66qpNKI2mDxPJ0xwBGDRoe1prsf73mo-NuhFjwhx7CbIPJYOecH2V-anI4sCU6fMC7OtZK-o4kBxBPE-aLMxoZnxWGmNMMl4Wy7bkmdrfR5Yu57YvtVUcZtD9UP50QkQpahga_B591Va8ahomzrWZzQRCxzzi69r01bXM6pRn699sNo&t=eyJscyI6ImdzZW8iLCJsc2QiOiJodHRwczpcL1wvd3d3Lmdvb2dsZS5jb21cLyJ9

If you are changing DDR in the CVM P3448, be sure to get the DDR memory training
parameters generated and update them in the emc_memory section of the kernel
DTB.

I do not find any instruction on ‘generate DDR memory training parameters’ and I do not find any ‘emc_memory’ section in any dts or dtsi file. Is there a better/newer version of that pdf file ?

Hi,

Sorry for late reply. We will check your sku with our internal team tomorrow since it was holiday season.

Those “DDR memory change” is not conducted by you. It means different kind of modules are getting involved.

No problem. It was holiday season here too.

Hi,

Could you share the i2c dump from the both the good and NG modules that you think have bad gpu performance?

The result you shared here was from the bad module, right?

I just checked the whole thread again. And this looks more like a hardware problem.

In the first post you said you have a devkit, but now you don’t have a devkit. Which comment is correct?

Also, how many jetson nano modules do you have there? Do you see lots of modules have these “failed to find EMC node” error? or only one of the module?

Hello WayneWWW,

We, as a company, have only one devkit. I currently can only access it through its network interface, so I cannot exchange the jetson nano on it. We have several jetson nano’s. I have currently access to only one jetson nano by serial line. Is there a way to retrieve the early boot messages appeared on the serial line from within linux ?

The i2c dump I gave is from a slow board. Here are the results from another slow board :

nvidia@qcam3:~$ sudo i2cdump -y 2 0x50
[sudo] password for nvidia:
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 02 00 04 46 00 00 00 00 00 00    ?.?.x??.?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 32 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    02-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff e1 da e9 4b 04 00 31 34 32 33 31 31    ....???K?.142311
50: 39 30 35 30 33 30 31 00 00 00 00 00 00 00 00 00    9050301.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff e1 da e9 4b    ............???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 dd    ...............?
nvidia@qcam3:~$ sudo tegrastats
RAM 960/3964MB (lfb 397x4MB) IRAM 0/252kB(lfb 252kB) CPU [43%@1479,43%@1479,33%@1479,38%@1479] EMC_FREQ 70%@204 GR3D_FREQ 0%@76 NVJPG 627 APE 25 PLL@38.5C CPU@41.5C PMIC@50C GPU@40.5C AO@49C thermal@41.25C POM_5V_GPU 3293/3293 POM_5V_IN 0/0 POM_5V_CPU 872/872
RAM 960/3964MB (lfb 397x4MB) IRAM 0/252kB(lfb 252kB) CPU [50%@1479,42%@1479,38%@1479,38%@1479] EMC_FREQ 70%@204 GR3D_FREQ 0%@76 NVJPG 627 APE 25 PLL@38.5C CPU@42C PMIC@50C GPU@40.5C AO@49C thermal@41.25C POM_5V_GPU 3412/3352 POM_5V_IN 0/0 POM_5V_CPU 1111/991
^C
nvidia@qcam3:~$ sudo cat /sys/kernel/debug/clk/emc/clk_rate
204000000
nvidia@qcam3:~$

Here are the results on a fast board; strangely, tegrastats does not give the EMC_FREQ.

nvidia@qcam3:~$ sudo i2cdump -y 2 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 02 00 04 46 00 00 00 00 00 00    ?.?.x??.?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 32 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    02-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff c5 da e9 4b 04 00 31 34 32 33 31 31    ....???K?.142311
50: 39 30 35 31 37 31 34 00 00 00 00 00 00 00 00 00    9051714.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff c5 da e9 4b    ............???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02    ...............?
nvidia@qcam3:~$ sudo tegrastats
RAM 1053/3964MB (lfb 46x4MB) IRAM 0/252kB(lfb 252kB) CPU [22%@1479,9%@1479,7%@1479,9%@1479] PLL@18.5C CPU@23C PMIC@50C GPU@22C AO@26.5C thermal@21.75C POM_5V_GPU 4093/4093 POM_5V_IN 0/0 POM_5V_CPU 2222/2222
RAM 1043/3964MB (lfb 44x4MB) IRAM 0/252kB(lfb 252kB) CPU [60%@102,39%@102,32%@102,19%@102] PLL@17.5C CPU@20.5C PMIC@50C GPU@21.5C AO@25.5C thermal@21.75C POM_5V_GPU 2070/3081 POM_5V_IN 0/0 POM_5V_CPU 199/1210
^C
nvidia@qcam3:~$ sudo cat /sys/kernel/debug/clk/emc/clk_rate
1600000000
nvidia@qcam3:~$

And here are the results on the devkit

nvidia@tegra-nano:~$ sudo i2cdump -y 2 0x50
[sudo] password for nvidia:
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 00 00 04 46 00 00 00 00 00 00    ?.?.x?..?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 30 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    00-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff 41 3d ec 4b 04 00 31 34 32 34 37 32    ....A=?K?.142472
50: 30 30 34 35 32 31 30 00 00 00 00 00 00 00 00 00    0045210.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff 41 3d ec 4b    ............A=?K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d9    ...............?
nvidia@tegra-nano:~$ sudo tegrastats
RAM 486/3964MB (lfb 213x4MB) IRAM 0/252kB(lfb 252kB) CPU [13%@102,3%@102,2%@204,1%@204] EMC_FREQ 0%@204 GR3D_FREQ 0%@921 APE 25 PLL@21.5C CPU@26.5C PMIC@50C GPU@25.5C AO@29.5C thermal@26C POM_5V_IN 872/872 POM_5V_GPU 0/0 POM_5V_CPU 158/158
RAM 486/3964MB (lfb 213x4MB) IRAM 0/252kB(lfb 252kB) CPU [13%@102,2%@102,0%@102,1%@102] EMC_FREQ 0%@204 GR3D_FREQ 0%@921 APE 25 PLL@21.5C CPU@26.5C PMIC@50C GPU@25.5C AO@29.5C thermal@26C POM_5V_IN 872/872 POM_5V_GPU 0/0 POM_5V_CPU 158/158
RAM 486/3964MB (lfb 213x4MB) IRAM 0/252kB(lfb 252kB) CPU [12%@204,3%@204,0%@204,0%@204] EMC_FREQ 0%@204 GR3D_FREQ 0%@921 APE 25 PLL@21C CPU@26.5C PMIC@50C GPU@25C AO@29.5C thermal@26C POM_5V_IN 872/872 POM_5V_GPU 0/0 POM_5V_CPU 158/158
^C
nvidia@tegra-nano:~$ sudo cat /sys/kernel/debug/clk/emc/clk_rate
1600000000
nvidia@tegra-nano:~$

There is no method to get bootloader log unless using uart.

Please check how many jetson nano modules have such “slow performance”. I believe there is only one module.

At least two.

Please share the exact boot log and eeprom content of these two modules.

Also share the S/N of the module.

OK,

I’ll collect those informations.

How can I easily retrieve the S/N of the modules ?

To dump the serial #, you can use

/proc/device-tree# cat serial-number

Let me clarify what we need here.

  1. Please check if all the “emmc” based module. How many of them have such problem.

  2. If every of them which suffers this error has the “failed to find EMC node” error in the bootloader log.

Yes, all the slow boards (I have tested 3 slow boards) say

64NOTICE:  BL31: v1.3(release):b5eeb33f7
NOTICE:  BL31: Built : 12:09:37, Jul 26 2021
ERROR:   Error initializing runtime service trusty_fast
get_emc_table_offset: failed to find EMC node
LPDDR4 Training: Can't find emc-table node

while the devkit says

64NOTICE:  BL31: v1.3(release):b5eeb33f7
NOTICE:  BL31: Built : 12:09:37, Jul 26 2021
ERROR:   Error initializing runtime service trusty_fast
RamCode = 0
LPDDR4 Training: Read DT: Number of tables = 2
EMC Training (SRC-freq: 204000; DST-freq: 1600000)
EMC Training Successful
408000 not found in DVFS table
RamCode = 0
DT Write: emc-table@204000 succeeded
DT Write: emc-table@1600000 succeeded
LPDDR4 Training: Write DT: Number of tables = 2

I currently do not have access to the uart of the only board where EMC_FREQ can reach 1600 MHz.

Serial numbers and i2cdump’s :

Devkit

nvidia@nano-devkit:~$ tr "\0" "\n" < /proc/device-tree/serial-number
1424720045210
nvidia@nano-devkit:~$ sudo i2cdump -y 2 0x50
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 00 00 04 46 00 00 00 00 00 00    ?.?.x?..?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 30 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    00-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff 41 3d ec 4b 04 00 31 34 32 34 37 32    ....A=?K?.142472
50: 30 30 34 35 32 31 30 00 00 00 00 00 00 00 00 00    0045210.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff 41 3d ec 4b    ............A=?K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d9    ...............?
nvidia@nano-devkit:~$

‘Fast’ board

nvidia@qcam3:~$ tr "\0" "\n" < /proc/device-tree/serial-number
1423119051714
nvidia@qcam3:~$ sudo i2cdump -y 2 0x50
[sudo] password for nvidia:
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 02 00 04 46 00 00 00 00 00 00    ?.?.x??.?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 32 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    02-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff c5 da e9 4b 04 00 31 34 32 33 31 31    ....???K?.142311
50: 39 30 35 31 37 31 34 00 00 00 00 00 00 00 00 00    9051714.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff c5 da e9 4b    ............???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02    ...............?
nvidia@qcam3:~$

‘Slow’ boards :

nvidia@qcam3-4:~$ tr "\0" "\n" < /proc/device-tree/serial-number
1423119050301
nvidia@qcam3-4:~$ sudo i2cdump -y 2 0x50
[sudo] password for nvidia:
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 02 00 04 46 00 00 00 00 00 00    ?.?.x??.?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 32 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    02-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff e1 da e9 4b 04 00 31 34 32 33 31 31    ....???K?.142311
50: 39 30 35 30 33 30 31 00 00 00 00 00 00 00 00 00    9050301.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff e1 da e9 4b    ............???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 dd    ...............?
nvidia@qcam3-4:~$

and

nvidia@nano-jp46:~$ tr "\0" "\n" < /proc/device-tree/serial-number
1423019115000
nvidia@nano-jp46:~$ sudo i2cdump -y 2 0x50
[sudo] password for nvidia:
No size specified (using byte-data access)
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 fc 00 78 0d 02 00 04 46 00 00 00 00 00 00    ?.?.x??.?F......
10: 00 00 00 00 36 39 39 2d 31 33 34 34 38 2d 30 30    ....699-13448-00
20: 30 32 2d 34 30 30 20 46 2e 30 00 00 00 00 00 00    02-400 F.0......
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff    ................
40: ff ff ff ff bd d2 e9 4b 04 00 31 34 32 33 30 31    ....???K?.142301
50: 39 31 31 35 30 30 30 00 00 00 00 00 00 00 00 00    9115000.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: ff ff ff ff ff ff ff ff ff ff ff ff bd d2 e9 4b    ............???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 95    ...............?
nvidia@nano-jp46:~$

I have also a third ‘slow’ one with serial number 1423119050700, but unreachable at the moment,
and we have about 500 naked jetson-nano’s in boxes waiting to be mounted on boards.

Where and how does ‘BL31’ look for the emc-table’s ? There are ‘emc-table’ nodes in the DTB and DTB-1 partitions of both the fast and the slow boards, and those DTB and DTB-1 partitions are identical on the slow and fast boards ,like every other pair of partitions except of course ‘APP’, and also ‘UDA’.

Hi,

You don’t need to worry about those bootloader logs at this moment. There is nothing you can do.

The real thing you can do that could help me is please try to move those slow modules to devkit and test if possible.

Also, if your “devkit” is a sdcard based module, then no need to add to comparison.

Just want to know, could you list out how many “emmc” modules on your side that are slow and how many of them are normal? 1x module is fast and 3x modules are slow? Are there any more?

Hello Wayne,

we are in the prototyping phase, so we do not have many boards now. I am aware of another one at our hardware developer’s site; I try to get more info’s about it, but it’s still holiday season here for some people.

At least 3 boards are slow, and 1 is fast. IIRC, the fast jetson module has been put once on the devkit for testing and then put back on our board, and it seems that it became fast after that. If I understand correctly, you would like me to repeat that operation with another jetson nano ?

Yes, our devkit is sdcard based.

Hi,

There is no need to put it to devkit first and put it back to your board. It won’t have any improvement with such test.

I just want to know if those slow modules would also hit issue on devkit.

Hi,

We have checked the serial number of these 2 slow and 1 fast modules. They are all built Aug 2019 and the DRAM and eMMC are all the same.

Thus, these modules are same. I can only suggest you to move them to devkit and test.

  1. Please also check if you can manually modify the dram frequency with the sysfs node.

  2. Please run jetson clocks and share me the dmesg on the slow board.

Hello Wayne,

Sorry, I have not been able to use the devkit to put the jetson Nano in it, but I did other tests.

I took from our stock a sealed jetson nano coming directly from Nvidia, and put it our custom board.
Without flashing it, it started and put messages on the serial line indicating that it had found the ‘emc-tables’

 [0001.268] LPDDR4 Training: Read DT: Number of tables = 2
 [0001.273] EMC Training (SRC-freq: 204000; DST-freq: 1600000)
 [0001.285] EMC Training Successful
 [0001.288] 408000 not found in DVFS table
 [0001.295] RamCode = 0
 [0001.298] DT Write: emc-table@204000 succeeded
 [0001.304] DT Write: emc-table@1600000 succeeded
 [0001.308] LPDDR4 Training: Write DT: Number of tables = 2

, and then booting into a terse linux, where IIRC the emc_freq could reach 1600 MHz, but it is impossible to run the jetson_benchmarks or even tegrastats, by lack of ethernet/git/dpkg support.

I flashed it with the vanilla jetpack-4.6 for jetson-nano-emmc and everything worked well, except that I had to redo it after commenting the line starting with DISPLAY_PLUGGED in nv-oem-config.sh script to get the oem-config.sh script on the virtual serial line on my desktop pc. I’ll describe that more in detail in another post. With that installation (thus with the dtb of a devkit), emc_freq can also reach 1600 MHz.

Looking at our dts file, I noticed that the “porg-platforms/tegra210-porg-p3448-emc-a00.dtsi” file was not included, and included it. IMHO, if that file must be included for every jetson-nano platform, it should be included in “tegra210-porg-p3448-common.dtsi”, and not be included separately by each platform-specific top-level dts file.

Thereafter I did many tests and have some conclusions :

  • The dtb used when flashing must include the “porg-platforms/tegra210-porg-p3448-emc-a00.dtsi” file

  • At flash time that dtb is copied to the DTB and DTB-1 partitions, but something else is stored also outside of the partitions. If I overwrite the DTB and DTB-1 partitions of a ‘slow’ board with the matching partitions of a ‘fast’ board without reflashing, my board is stil slow

  • If the extlinux.conf FDT entry points to a copy of the dtb including the emc* nodes stored in the DTB partition that what used at flashing time, the card runs ‘fast’

  • If the extlinux.conf FDT entry points to a slightly different dtb, one cannot predict the result :
    I have tested there dtb’s without emc-table nodes giving still fast board, and other yielding slow boards. U-boot seems to add the external-memory-controller@7001b000 node from the DTB partition to the dtb file pointed by FDT, but this does not always give a fast board. The same happens with FDT-dtb’s containing the emc-table nodes : the result is unpredictable.

The fact that we cannot at the moment trust a new FDT-selected dtb to maintain the fast speed of the EMC is worrying, because we must sometimes add or change entries in that dtb without the ability to reflash boards that are at remote locations.

What does the flashing procedure do with the emc info in the dtb, that is not done by merely overwriting the DTB and DTB-1 partitions ? How could we get the same result without the USB-flash procedure by simply rewriting some emmc block or other internal memory of the jetson nano ?

Do you recommend to include the “porg-platforms/tegra210-porg-p3448-emc-a00.dtsi” file in the dtb file pointed by FDT, or rather rely on U-boot to merge the one stored in the DTB and DTB-1 partition ?

Thanks for reading