Permission denied emmc memory space

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hello all,

Using xdiskimage to check on disk space, I am seeing a “permission denied” space of about 44Mi in the “/” directory. This is on a Jetson TX2 right after flashing.

I am wondering what this allocated disk space is for? And why does it say permission denied?

The 44Mi size is pretty harmless but I have another TX2 that shows 12Gb being used! This one has been used for about a year now. The issue being that it keeps growing and eventually memory runs out on the emmc. “df” reports disk is 99% or 100% full.

I am using Jetson TX2 with aetina ace-n622 carrier board. Running L4T_28_2_1 with Jetpack 3.3.

Thanks for the help

#2

I am not familiar with “xdiskimage”. However, if a non-root user is attempting to query disk locations for their specific content, then security will deny reading some of this.

If you run “df -H -t ext4”, then you will see the totals for any ext4 formatted content. Exclude “-t ext4” and you will see everything. Subdirectory content is not detailed in this.

If instead you want to see from a certain location and recursively into it the space of each subdirectory, then on command line:
du -h /where/ever/to/start
…which will give you security errors if you are not root and a subdirectory is something you don’t have permission to see. To see without error prefix with “sudo”:
sudo du -h /where/ever/to/start

It takes a lot of disk space to install, but if you want to see a graphical pie chart of all content from a certain location and in, with the ability to drill down, then “filelight” is a lot of fun. If you don’t mind how large the program is, then:
sudo apt-get install filelight

Similar to the other case, if you just run “filelight” without “sudo”, don’t expect to see details of someone else’s content. However, “sudo filelight” will show everything.

Note: It doesn’t really count since it isn’t a real filesystem, but there is a “gvfs” (Gnome virtual file system) which looks like inaccessible disk space, and in some ways acts like disk space, but it will not be readable for anyone with the usual tools.

#3

Thanks linuxdev! I am only now getting back to this

Typo from my original post, I meant “xdiskusage” and not “xdiskimage”.

I tried “sudo filelight” but it is not showing me the permission denied folder. xdiskusage tells me it is located in “/” with a size of 44Mib (for this particular Jetson). The smallest I can find with filelight is the 214Mib /opt folder.

How would I go about zeroing in on the right folder?

Also I tried unmounting the gvfs filesystem but that did not solve the issue. Memory is still being allocated to that permission denied folder

#4

“gvfs” is not a real filesystem, it exists only in RAM and should not be umounted, and will not consume actual disk space on eMMC. Just to emphasize, gvfs is working as expected when it denies access. In no way does gvfs consume any disk space even if it has a size (that size is in RAM).

For filelight to show content which root owns you would need to start it with “sudo filelight”. Then it should see everything which is a real disk (you could start from where you expect the disk is consumed, e.g., “sudo /home” or “sudo /” or “sudo /usr/local”). In the case of gvfs this is part of secure interprocess communications and it is intended that very little will be able to access this, not even root under normal circumstances.

If you really want to see what is going on with the eMMC, and not be confused by pseudo files, then you would probably want to clone, and then loopback mount the raw clone on your local PC. You could run “filelight” against the loopback mounted clone, and you would be guaranteed to see everything which was actually part of the eMMC.

Clone information has changed with time. The newer L4T/JetPack/SDKM instructions differ from R28.2.1. The R28.2.1 URL (including documentation) is here:
https://developer.nvidia.com/embedded/linux-tegra-r2821
(you may need to go there, log in, and then go there again).

A clone will create both a smaller (probably several GB in size if the filesystem is near empty, but approaching 30GB if the filesystem is full) “sparse” clone, plus a larger (the exact size of the entire partition, perhaps about 30GB) “raw” clone. I throw away the sparse clone and only keep the raw clone. If you check the docs for your release for cloning, and want to know how to work with the clone from your PC, then just ask.

Note that the combination of clone, fix, then flash by clone, is a fairly common use of a clone, but it takes a lot of time. One clone would take well over an hour. Copying a 30GB file on a hard drive might take half an hour as well. However, your PC can work with a clone with maxed out content without caring, whereas the Jetson requires temp files to be written for a number off procedures, often including login…and so a full disk can prevent login.

The serial console is very simple, and has very few login requirements, and so you might be able to log in via this even on a system which has disk filled, and delete content. Using serial console could (in less than a minute) get around the whole long and painful process of cloning and examining clones and flashing with a fixed clone.

If I were to suspect a normal install of using up disk space, then here are some key places to check:

  • /var/log (see “sudo du -h -s /var/log”)
  • /opt (see “sudo du -h -s /opt”)
  • /usr/local(see “sudo du -h -s /usr/local”)
  • /home(see “sudo du -h -s /home”)
  • /home/...each user...(see “sudo du -h -s /home/...user name...”)

The big question: Can you log in via serial console? Assuming you can, are you able to drop into a root shell with “sudo -s”? If you can, then you don’t need to clone. You can solve the issue from there. If you cannot, then you need to clone and work on the clone instead of the Jetson.

#5

Got it for gvfs! I’ll leave that alone as it does not seem to be an issue

“sudo filelight” in / is not showing me the used disk space I am expecting to find there. So i guess this used space is not a “real” disk. To give you more visibility, attached capture shows the output of xdiskusage and the 1.58Gb permission denied space.

Capture_xdiskusage
Capture_xdiskusage1696×964 46.2 KB

For the time being I am able to log in with no issues, space usage is not big enough to prevent malfunction yet but I am slowly watching it grow…

Checking for size of the key places you mentionned this is what I get:
/var/log is 11M big
/opt is 187M big
/usr/local is 1.7G big
/home is 3.8G big
only one user for this jetson

As reported by xdiskusage, the used disk space seems to be located at / and I do not think it would be in any of these 4 places as only the “permission denied” space grows while those folders keep the same sizes.

I’ll keep digging

#6

Note: this is on a different Jetson TX2 than the original post is about. On this particular one unwanted disk space usage grew from 45Mi (fresh install) to 1.6Gb in about a month and a half

#7

For filelight you can use the mouse to “drill down”. If you look at a part of the “pie chart” which looks large, and click on this, then it will navigate to that subdirectory. It peels back like the layers of an onion. If you want to use this, then you would start with the largest slice, and then drill down with the mouse until reaching an actual file which is large and useful to delete.

Am I correct in my assumption that you ran xdiskusage with sudo?

I do not see a name with the “permission denied” content. I do not normally use xdiskusage, but I see that it includes:

(inodes)
1.447Gi

…which is a node count underlying how ext4 works. Basically, this is metadata about how the files are stored. The fact that the “permission denied” is shown in a similar way makes me thing this is not regular storage. I can’t guarantee it, and am far from sure about it, but perhaps this is journal space. Or perhaps it is a swap file…do you have approximately 1.5GB of swap? What do you see from “sudo swapon -s”? How much swap shows up when running “htop” ("sudo apt-get install htop" if you don’t have it)?

Basically, I am thinking this “permission denied” space is not “content” within the filesystem. Most likely this has something to do overhead or metadata, but I couldn’t guarantee it. Running with “sudo” implies the only “denied” space is space not supporting browsing it. Metadata and overhead won’t have names any more than the inode content won’t have a name within the filesystem.

However, if you really want to know what this is, and if this is not normal filesystem content, but is in some way otherwise related to the filesystem, or even if it were malicious content, then I would say the only method of being certain is to clone that partition and examine it from your PC.

#8

As you said, looks like this used space does not support browsing it… After looking around I believe sudo filelight does not report that space. It does not show up at all and is not being added up to the total used disk space reported either.

Yes correct

“sudo swapon-s” does not return anything and htop shows 0K out of 0K swap is being used

#9

turning the Jetson on this morning, denied space is at 4.2Gb! It was 1.5Gb yesterday, this is puzzling

upon restart it will randomly be between 4Gb and 5Gb

#10

I think if you really want to know more about it you will have to clone the rootfs partition and examine it on another computer. A clone (the raw version, not the sparse version) can be covered by loopback and the loopback device can be treated as if it were an actual hard drive or partition. With another system doing the work you will know that the operating system itself is not interacting.

I am not sure if shutting down would release this space or not, but there are ways you can freeze the disk and set it to read-only when this is going on, and then shut down. It is likely the partition could be examined this way on the other Linux computer and tools there would be fairly “authoritative” and the operating system itself would not interfere since it is just a spare drive which is not even mounted so far as the PC is concerned.

Note that although you would need your specific release of R28.2.1 if you were to use this clone for flashing, that any release will work to create a clone (cloning has virtually no release version dependency, whereas restoring is very specific to release version). Even the newer R32.x could be used to clone.

NOTE: For all I know this could be part of the space used for “leveling”, which is normal for solid state memory. Or it could be something completely different. Asking the developers of the tool you are using may be the best way to find out…they would be able to tell you right away if this is something “normal”, like wear leveling reserved space. Going directly at this is certainly thorough, and even interesting to learn from, but it may not be worth your time compared to just asking the tool developers.

#11

i am used to making clones so that’s no problem, I am not that familiar with mounting the image as a loopback device though

looking online I found this command “sudo losetup -Pf disk_image.raw” which seems to do the job (image.raw being the raw clone)

Is that how you would go about mounting the image?

then I ran “sudo filelight” and “sudo xdiskusage” on the mounted drive with below results

filelight
filelight1920×1080 205 KB

xdiskusage
xdiskusage1920×1080 115 KB

basically same results as before, denied space is not seen by filelight (you don’t see it on the xdiskusage capture image either but it’s there, 256MB on this image)

#12

That is correct. You might also consider (after covering with loopback, but before mounting) running “gparted” on the loopback device and see what it says. Additionally run “sudo gdisk -l” on the loopback device.

Keep in mind that the “(inodes)” is just a different way of looking at how the filesystem is arranged, and is not a file or directory…this is the tree structure metadata which ext4 uses. I still don’t know what that 256MB is, but I keep thinking that this is probably just something to do with either wear leveling or metadata…I just don’t know what. If for some reason you did manage to delete this, then I am going to bet the partition would be destroyed. If someone does know what that is from, then I’d like to know too.

#13

Sorry but could you elaborate? Do I run gparted before my “sudo losetup -Pf disk_image.raw” command? Does it matter where I “cd” into?

Where do I “cd” into to run this? The command line is giving me “Problem opening -l for reading! Error is 2. The specified file does not exist.”

Thanks again!

#14

gparted works on “block devices”. This means partitions and hard drives, or anything pretending to be one. It is the loop device which changes the image into a partition, and so gparted runs on the loop device. You can always use the “--show” option to know which device was created when you perform the losetup. So perhaps this example, assuming the first device is “/dev/loop0”:

sudo losetup -Pf --show disk_image.raw
sudo gparted /dev/loop0

Do note that once the loop device is associated with something that you do not run losetup on it again. You could “free” (detach) the specific loop device, e.g.:
sudo losetup -d /dev/loop0
…or just detach all which are detachable:
sudo losetup -D

The purpose of running gparted for most people would be to change partitions. For you this is not the case, the only purpose is to see what gparted thinks about the device. In this case, to see what gparted thinks of the clone partition when covered by loopback.

Also, you can run gparted on the entire eMMC if run directly on the Jetson, but you would want to be very certain you do not try to change anything. Your only use should be to gather information:
sudo gparted /dev/mmcblk0

The reason the specific partition is of interest (versus the disk as a whole) is because xdiskusage says there is content which is unreachable (and I think that content is just meta content, and not some deletable file). I don’t know what that content is, but I suspect it is related to either the ext4 itself, or to the solid state memory (which is different than what a traditional platter style hard drive is). Use gparted as a research tool and see how it compares to xdiskusage.

#15

In case somebody has the same problem, I ended up flashing the jetson with the most up to date L4T release aetina (carrier board manufacturer) is supporting. That seems to do it, “permission denied” disk space does not grow larger than about 50Mb now