AGX Xavier eMMC storage capacity issue

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
, ,
1

Hello,

Environment

  • AGX Xavier(32GB)+Custom Baord
  • Jetpack 4.6

I am currently experiencing an issue with the root partition storage capacity on the eMMC of the customer’s AGX Xavier 32GB devices.
There are a total of 11 AGX Xavier devices with a custom board in use. Among them, 9 devices (“normal_Jetson”) are functioning normally, while 2 devices (“abnormal_Jetson”) are showing signs of storage capacity issues.
All 11 devices were cloned from the same image and share identical usage patterns and software environments.
However, the root partition on the abnormal_Jetson devices is showing a higher used storage capacity compared to the normal_Jetson devices.

<abnormal_Jetson>/ <normal_Jetson>
image
image

According to the results of the $ df -h command result:

  • abnormal_Jetson: 25G used
  • normal_Jetson: 20G used

However, $ sudo du -xh --max-depth=1 -x / | sort -hr command result:

  • abnormal_Jetson: 16G used
  • normal_Jetson: 20G used

→ While all normal_Jetson devices consistently report 20G of used space,
the abnormal_Jetson devices show a discrepancy—indicating 25G used according to the $ df -h command, despite the $ du command indicating only 16G is actually used.

I have conducted inspections and tests regarding the issue described above, but have not identified any significant findings.

TEST

  1. fsck: File system integrity check → Result: clean
  2. lsof, truncate: Checked for and cleared deleted files that remained open → Freed a few megabytes, but the issue persists
  3. tune2fs, dumpe2fs: Checked metadata such as ext4 reserved blocks, inodes, and journal → No differences found between normal_Jetson and abnormal_Jetson
  4. Inspected Large directories: logs, package caches, Docker data, and build outputs → No anomalies detected
  5. Checked Hidden mount points and files → No anomalies detected

For your review regarding the issue, I am providing the df/du command outputs and the dmesg and syslog logs from both the normal_Jetson and abnormal_Jetson.
jetson_storage_issue.zip (122.1 KB)

Attachment
jetson_storage_issue
├─ normal : normal_Jetson Log folder
│ ├ normal_df_du_result.txt : df/du Command output
│ ├ masked_normal_dmesg.txt : dmesg
│ └ masked_normal_syslog.txt : syslog
└─ abnormal : abnormal_Jetson Log folder
… ├ abnormal_df_du_result.txt : df/du Command output
… ├ masked_abnormal_dmesg.txt : dmesg
… └ masked_abnormal_syslog.txt : syslog

I would appreciate it if you could review and confirm the storage capacity issue.
If there are any further diagnostic methods, hardware-level tools or guidelines we could use to investigate the issue in more detail, Please provide me with the information.
Additionally, please advise on whether this issue may potentially be caused by a hardware defect.

3

Hi,

Could you provide the commands for us to review?

Thanks

4

Hello,

We checked df and du commands when we got a message about insufficient memory.
And we’ve reviewed several check points as above post after we realized it has an issue.
If what you’re asking is about options for the commands we’ve checked, we’ll share them with you.

But I don’t know well If you ask us the command how to reproduce this issue.
We don’t know what makes it and how does it makes also.
It is not a phenomenon that occurs temporarily when performing a specific action.
Once it occurs, it continues to remain in that state.
We are using 11ea systems with same environment and same software.
The symptom occurred only in two of them.

7

I’ve noticed that you have NVMe SSD connected and the usage for both cases are so much different.
normal: /dev/nvme0n1 916G 522G 348G 60% /luceon_output
abnormal: /dev/nvme0n1 916G 13G 857G 2% /luceon_output
Please also share the result of lsblk in both cases or you can try to remove NVMe SSD.

Please try to put it on the devkit and use SDK manager to flash.
It could help you identifying if there’s any HW defect.

8

Hello,
We’ve found the reason with following command.

$ du -shx /

We were used the directory it has some datas and then SSD is mounted on this path.
$ du command doesn’t show used data before mounting.
So, It makes problem.
Thank you for the support.