Inconsistent disk usage results

[marmikshah]

Hello,
I am using a Jetson Xavier (with JP 4.6.1).
When I use du -sh /* and df -h, it shows different disk usage.

Results with du -sh /* | sort -h

0	proc
0	sys
4.0K	dev
4.0K	mnt
4.0K	README.txt
4.0K	srv
4.0K	tegra194-mb1-bct-ratchet-p2888-0000-p2822-0000.cfg
8.0K	snap
16K	lost+found
84K	tmp
832K	root
11M	bin
11M	sbin
13M	etc
23M	run
100M	boot
175M	home
450M	lib
457M	opt
3.9G	var
9.9G	usr
1.6T	media

With ncdu

ncdu 1.12 ~ Use the arrow keys to navigate, press ? for help                                                                            
--- / ----------------------------------------------------------------------------------------------------------------------------------
    9.8 GiB [##########] /usr                                                                                                           
    3.8 GiB [###       ] /var
  456.4 MiB [          ] /opt
  449.1 MiB [          ] /lib
  174.9 MiB [          ] /home
   99.5 MiB [          ] /boot
   12.5 MiB [          ] /etc
   11.0 MiB [          ] /sbin
   10.9 MiB [          ] /bin
  832.0 KiB [          ] /root
   84.0 KiB [          ] /tmp
e  16.0 KiB [          ] /lost+found
    8.0 KiB [          ] /snap
e   4.0 KiB [          ] /srv
e   4.0 KiB [          ] /mnt
    4.0 KiB [          ] /media
    4.0 KiB [          ]  tegra194-mb1-bct-ratchet-p2888-0000-p2822-0000.cfg
    4.0 KiB [          ]  README.txt
>   0.0   B [          ] /sys
>   0.0   B [          ] /run
>   0.0   B [          ] /proc
>   0.0   B [          ] /dev

And df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p1   28G   26G  599M  98% /
none            7.4G     0  7.4G   0% /dev
tmpfs           7.8G   52K  7.8G   1% /dev/shm
tmpfs           7.8G   23M  7.8G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1    1.8T  1.6T  182G  90% /media/data
tmpfs           1.6G   12K  1.6G   1% /run/user/120
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/739c2d67689ce60f4b2123c6e04305bf2e469f278a90b3d0624897608098a9db/merged
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/a4210e4346bd0020f70676ebbe2a5300bab7bd2ebd8861333fcb43794a062cbd/merged
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/3934ffc67e3c3ba26b3c787cc1a0efc495e6a03b9bbeb1d340903397d9041e9e/merged
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/2abc96574d1d9adca704903d8982012b23da86c66f6e4b92f5de1a3d5eb633db/merged
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/88533dd5735f63ed25775ed02bb66c20492c4c66eccdfd3e88840e9c209a9bf0/merged
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/f086bd3a73ddfabedc695ecaba1537c3ca3c8a6810dff3a30429dc26f306306e/merged
tmpfs           1.6G     0  1.6G   0% /run/user/1000
overlay          28G   26G  599M  98% /var/lib/docker/overlay2/d483e9e66d6c84abe06f2c09829c76636e8a6dd5c7952b222abdda4c9c96d77a/merged

Is there any way to tell where the extra 10+ Gb is coming from?

Thanks.

[JerryChang]

hello marmikshah,

they’re two difference commands,
du is used to estimate file space usage.
df to report the amount of space used and available on the file system.

so,
could you please point-out what’s your concerns?

[marmikshah]

Hey,
Thanks for your reply
My concern is that df seems to show about 28gb used while du @ / totals about 16gb only.
I want to be able to find where the extra 12 Gb is coming from.

I have about 15 other Xaviers running with exact same system configuration and docker services, but this seems to happen on just one of them. So wondering if there is any other way I can find the route cause?

[JerryChang]

hello marmikshah,

I’m not expertise in docker service.
but… could you please check docker volume for comparison,
please see-also, Understanding Docker Volumes - Earthly Blog

[marmikshah]

Hi,
I have already checked the docker services. They are not consuming that much memory.
Here is the result of docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          7         6         3.354GB   106.1MB (3%)
Containers      7         7         4.063MB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

This corresponds to the space used in /var/ folder.
So the extra 10 Gb is not related to docker.

[marmikshah]

My issue seems very similar to this thread

10G of disk space disappeared of xavier

Basically, as you drill down by clicking on subdirectories, it will show only the content used in that new subdirectory or its child subdirectories. You’re going through a forest, and searching for smaller and smaller patches until you see something out of place. In the first filelight image I see “/usr” contains the bulk of used space, although “/home” is significant. Probably, if you drill down on “/usr”, then you can go to its largest subdirectory, and see what is there. If you get to a subd…

But seems the solution is not mentioned here.

[marmikshah]

Incase this helps anyone.

This had happened in a rare case when at boot the NVME did not mount and docker daemon created a folder (which should’ve been on the NVME) on the eMMC. So videos were being written into eMMC until the next reboot (few hours later).
After that the NVME has always mounted correctly, so the /media/data/ of the eMMC was never visible because NVME’s data was shown there.

I manged to find it by mounting the root filesystem to another directory and checking/media/data all the videos that were written to eMMC were found there.

sudo su
mount -o bind /  /mnt

The extra files could be found at /mnt/media/data then.

So df -h was indeed correct all the time.