Jetson AGX Orin SSD malfunctioning

I am using a Jetson AGX Orin running JP5.1.4. While the system is running, the SSD frequently encounters errors, rendering even basic commands unusable, and the kernel generates a large number of log entries like the following:

Sep  7 14:41:44 galbot-echo kernel: [15093.078599] blk_update_request: critical medium error, dev nvme0n1, sector 87195648 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.121077] blk_update_request: critical medium error, dev nvme0n1, sector 87195776 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.153046] blk_update_request: critical medium error, dev nvme0n1, sector 87201792 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.195166] blk_update_request: critical medium error, dev nvme0n1, sector 87201944 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.235868] blk_update_request: critical medium error, dev nvme0n1, sector 87201944 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.253457] blk_update_request: critical medium error, dev nvme0n1, sector 87214080 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.295810] blk_update_request: critical medium error, dev nvme0n1, sector 87214208 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.339233] blk_update_request: critical medium error, dev nvme0n1, sector 87189504 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.373146] blk_update_request: critical medium error, dev nvme0n1, sector 87189648 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Sep  7 14:41:44 galbot-echo kernel: [15093.386082] blk_update_request: critical medium error, dev nvme0n1, sector 87180288 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Sep  7 14:41:53 galbot-echo kernel: [15102.379988] print_req_error: 15 callbacks suppressed

It returns to normal after a restart, but reverts to this state after running for a while. What could be the cause, and how can this be prevented?

Full log:

kern.log (30.6 KB)

Could you first provide the NVMe model, firmware version, and health information?

Please run these commands after restarting the system, while the SSD is accessible:

sudo nvme list
sudo nvme id-ctrl /dev/nvme0 | grep -E 'mn|sn|fr'
sudo nvme smart-log /dev/nvme0
sudo nvme error-log /dev/nvme0 --log-entries=16

If nvme is not installed:

sudo apt update
sudo apt install nvme-cli

Please post the complete outputs. In particular, we want to check the SSD temperature, critical_warning, media_errors, num_err_log_entries, available spare capacity, and percentage used.

If the data is important, please back it up before performing additional tests.

—————————————————————————-
Tobias Fonseca

Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com