Pstore tool doesn’t work on Thor Devkit R38.2.1

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Dear nvidia team:

Thor Devkit R38.2.1 release

Use echo c > /proc/sysrq-trigger to trigger oops.

The directories /sys/fs/pstore/ and /var/lib/systemd/pstore/ are empty.

Can you reproduce this problem?

4

Hi,
The issue is known and we are working on it. As of now please check UART log to get information.

5

Thank you for your reply.

Could you briefly explain the reason for the problem?

In which version are you expecting to resolve this?

Thank you.

6

Is this a software bug or an parameter configuration issue?

7

Hi,
The issue is under debugging. It seems to be a software issue but is not rootcaused yet.

8

By the way, has this bug been fixed yet?

9

Hi,
It is under debugging. On Jetpack 7.1 r38.4, it is known the tool does not work.

10

when can we use this tools?

11

On Jetpack 7.2 R39.2 device-tree is good for ramoops. I added, or Jetpack 7.2 may have added some of the entries to extlinux.conf noted below.

sudo dtc -I fs -O dts -o ~/live.dts /proc/device-tree
grep -A7 ramoops live.dts     
                ramoops_reserved = "/reserved-memory/ramoops_carveout";
		ramoops_carveout {
			alignment = <0x00 0x10000>;
			record-size = <0x10000>;
			alloc-ranges = <0x00 0x00 0x01 0x00>;
			compatible = "ramoops";
			size = <0x00 0x200000>;
			console-size = <0x80000>;
			status = "okay";
			reg = <0x1f 0xbe280000 0x00 0x200000>;
			phandle = <0x2f2>;
			no-map;
		};

sudo dmesg | grep -Ei 'pstore|ramoops|efi_pstore'
sudo ls -al /sys/fs/pstore


# I had two oom events a few days ago. Just checked and
sudo ls -al /var/lib/systemd/pstore
total 224
drwxr-xr-x  2 root root   4096 Mar 24 06:46 .
drwxr-xr-x 12 root root   4096 Feb 17 14:07 ..
-rw-------  1 root root 109199 Jun  3 20:16 dmesg-ramoops-0
-rw-------  1 root root 109191 Jun  3 20:16 dmesg-ramoops-1


/boot/extlinux/extlinux.conf
# Have these parameters to the APPEND line. 
printk.always_kmsg_dump=1 panic=5 efi_pstore.pstore_disable=1

  - printk.always_kmsg_dump=1 — write kmsg to pstore on oops, panic, and shutdown
  - panic=5 — auto-reboot 5 seconds after kernel panic (so the system comes back up and you can retrieve pstore records)
  - efi_pstore.pstore_disable=1 — suppress the EFI pstore backend; without this, systemd will try to load efi_pstore and it may conflict.
  
sudo reboot
14

Hi @whitesscott
If you run

# echo c > /proc/sysrq-trigger

Do you see the file be generated? We are still working on this and not sure why you see it working. Would like to know if the files are generated only when hitting OOM.

15

Hi @DaneLLL

Files were only generated when hitting OOM.

# This causes reboot but does not create files.
echo c > /proc/sysrq-trigger


# This takes a while to run but both times it ran did create files:
-rw-------  1 root root 109199 Jun  3 20:16 dmesg-ramoops-0
-rw-------  1 root root 109191 Jun  3 20:16 dmesg-ramoops-1

cd vllm
python -m benchmarks.kernels.benchmark_selective_state_update --ngroups 8 --headdim 64 \
  --dstate 128 --nheads 8 16 32 64 128 256 --mamba-ssm-cache-dtype float32 --compare \
  --validate --save-configs
16

Hello @DaneLLL ,

Is it true that ramoops is still not working on L4T 39.2.1?

We would like to use the feature. Maybe we could try to debug and fix it ourself. Do you know what could be the reason or what is already ruled out?

17

Hi,
It is true that ramoops is no working on Jetpack 7.2.1 r39.2.1.