Cleaning /tmp file (temporary logs) in Jetson

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hi every one,
I opened this topic though I know that the issue about tmp files in Jetson is like any ubuntu. But the problem is that most of us are using Jetson for very huge volume of computations which requires Jetson to work for a long time. Considering that when /tmp becomes full it makes serious problems which requires to be restarted-something that we do not want.
I want to ask How can we manage /tmp file and make it clean without restart?

Besied I am experiencing another issue (with Jetson device) even when I am trying to restart /tmp is steel full and I can’t do anything

nvidia@tegra-ubuntu:~$ df /tmp
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       28768380 27343960         0 100% /

nvidia@tegra-ubuntu:~$ df -k
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       28768380 26588604    695388  98% /
devtmpfs         7975976        0   7975976   0% /dev
tmpfs            8042776      184   8042592   1% /dev/shm
tmpfs            8042776    14064   8028712   1% /run
tmpfs               5120        4      5116   1% /run/lock
tmpfs            8042776        0   8042776   0% /sys/fs/cgroup
tmpfs             804280       12    804268   1% /run/user/106
tmpfs             804280        0    804280   0% /run/user/1001
tmpfs             804280        0    804280   0% /run/user/1000

I already tried something like

sudo find /tmp -type f -atime +10 -delete

or directly changing etc/default/rcS file by

TMPTIME=10

to clean /tmp but it does not work!
even

TMPTIME=0

does not work for me.
I will appreciate for your comments.

Thanks

#2

I don’t have an answer, but more information might help. Most of what is in “/tmp” isn’t a regular file, e.g., there are sockets or fuse file systems mounted. Lock files will be an exception, e.g., every X login has an X lock file. By default the actual “/tmp” is part of the main mount which serves “/”…so technically if “/tmp” fills so does the entire disk (but there may be limits imposed which restrict something before a disk fills). A way to show that “/tmp” itself is or isn’t filled is:

df -H -T /tmp

Note: “/dev/root” is “/”, and “/tmp” is a subset.

If you were to actually try to find the users of “/tmp” you’ll see an error regarding fuse file system type:

lsof /tmp/*

Those directories which are gvfs/fuse are actually other file systems mounted on “/tmp”…and are in themselves not even real file systems. These probably are tied to a running process and thus have no file content. If you are running out of space perhaps it is RAM and not disk…fuse/gvfs could report that way.

You’ll want to post some exact errors to decide if it is a quota, an actual disk filling, or some sort of virtual/non-real file system. You will also want to show this at the time you run out of space (you might want to go to “sudo -s” sudo shell first):

# To see if error messages are extreme (it might be error logs filling it up and not "/tmp"):
du -h -s /var/log
df -H -T /tmp
df -H -t ext4
du -sc /tmp/* /tmp/.[^.]* | sort -n
# Multiple versions of find so one can see what is or is not real, plus a master file list:
ls -ld `find /tmp -type f`
ls -ld `find /tmp -type d`
ls -ld `find /tmp -type s`
find /tmp

I’d expect that “/tmp” itself is not actually filled unless “/” is filled. If there are a lot of files in “/tmp” it could fill the file system even if they are empty (the 4k block size would be taken up by empty files…though some file systems can group empty files in the same block…I don’t know what ext4 does with that).

You might want to install “htop” (you can just use top, but htop is nicer) and run this as the process complains of too much disk use…see if memory is maxing out. As a test you could also add swap space via SD card or SATA, but keep in mind GPU memory and many specialty types of memory cannot be swapped…the purpose would be to allow other processes to swap out…if this increases time before failure than it likely means you ran out of physical ram.

NOTE: The “filelight” package could show a lot of information…but it is a large package. You might find it worth your time to add that and drill down on file system content in a graphical manner.

#3

I put here the results of your proposed commands:

root@tegra-ubuntu:~# du -h -s /var/log
32M	/var/log
root@tegra-ubuntu:~# df -H -T /tmp
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/root      ext4   30G   29G     0 100% /
root@tegra-ubuntu:~# df -H -t ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G   29G     0 100% /
root@tegra-ubuntu:~# du -sc /tmp/* /tmp/.[^.]* | sort -n
0	/tmp/argus_socket
0	/tmp/nvcamera_socket
0	/tmp/tmp.Pzr4hTNz5T
4	/tmp/.ICE-unix
4	/tmp/.Test-unix
4	/tmp/.X0-lock
4	/tmp/.X11-unix
4	/tmp/.XIM-unix
4	/tmp/.font-unix
8	/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-colord.service-bQNqSc
8	/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-rtkit-daemon.service-d6i1ED
8	/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-systemd-timesyncd.service-dtUrx4
48	total
root@tegra-ubuntu:~# ls -ld `find /tmp -type f`
-r--r--r-- 1 root    root    11 Nis 28 17:15 /tmp/.X0-lock
-rw------- 1 lightdm lightdm  0 Nis 29 07:35 /tmp/tmp.Pzr4hTNz5T
root@tegra-ubuntu:~# ls -ld `find /tmp -type d`
drwxrwxrwt 10 root root 12288 Nis 29 08:19 /tmp
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/.ICE-unix
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/.Test-unix
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/.X11-unix
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/.XIM-unix
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/.font-unix
drwx------  3 root root  4096 Nis 28 17:15 /tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-colord.service-bQNqSc
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-colord.service-bQNqSc/tmp
drwx------  3 root root  4096 Nis 28 17:15 /tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-rtkit-daemon.service-d6i1ED
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-rtkit-daemon.service-d6i1ED/tmp
drwx------  3 root root  4096 Nis 28 17:15 /tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-systemd-timesyncd.service-dtUrx4
drwxrwxrwt  2 root root  4096 Nis 28 17:15 /tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-systemd-timesyncd.service-dtUrx4/tmp
root@tegra-ubuntu:~# ls -ld `find /tmp -type s`
srwxrwxrwx 1 root root 0 Nis 28 17:15 /tmp/.X11-unix/X0
srwxrwxrwx 1 root root 0 Nis 28 17:15 /tmp/argus_socket
srwxrwxrwx 1 root root 0 Nis 28 17:15 /tmp/nvcamera_socket
root@tegra-ubuntu:~# find /tmp
/tmp
/tmp/.font-unix
/tmp/.X0-lock
/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-colord.service-bQNqSc
/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-colord.service-bQNqSc/tmp
/tmp/.XIM-unix
/tmp/.X11-unix
/tmp/.X11-unix/X0
/tmp/argus_socket
/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-systemd-timesyncd.service-dtUrx4
/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-systemd-timesyncd.service-dtUrx4/tmp
/tmp/nvcamera_socket
/tmp/.Test-unix
/tmp/tmp.Pzr4hTNz5T
/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-rtkit-daemon.service-d6i1ED
/tmp/systemd-private-9c23969d4ebd425eb7b748d8d02e3601-rtkit-daemon.service-d6i1ED/tmp
/tmp/.ICE-unix

and also

root@tegra-ubuntu:~# df -h --total
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        28G   27G     0 100% /
devtmpfs        7,7G     0  7,7G   0% /dev
tmpfs           7,7G  184K  7,7G   1% /dev/shm
tmpfs           7,7G   46M  7,7G   1% /run
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           7,7G     0  7,7G   0% /sys/fs/cgroup
tmpfs           786M   12K  786M   1% /run/user/106
tmpfs           786M     0  786M   0% /run/user/1001
tmpfs           786M     0  786M   0% /run/user/1000
total            61G   27G   33G  45% -

Beside soem errors related to disk space,
I get terrible results by compiling codes on “gpu” with warnings related to the memory.

018-04-26 13:14:40.069245: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.53GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

Are these warnings related to low disk space in tmp?

I am running object detection codes which reported around 30 fps (e.g. tiny yolo) on Jetson (tx2 with jetpack 3.2) while it takes 3 seconds per frame!! for my case (same code using tensorflow) I am sure that gpu is working on Jetson.

Can this low space effect computation speed?

#4

It isn’t “/tmp” which is full…this just happens to be a subdirectory of “/”. It is “/” which is full…there is nothing left free on the entire disk. Logins can become impossible if lock files can’t be written. All use which touches the disk will likely fail.

One option is to mount a SATA drive (or SD card) and perhaps mount a partition on “/usr/local/” after copying that content to the SATA drive (or SD card).

#5

How can I format Jetson and free it up? Does flashing formats entire disk? -Thanks

#6

Flashing would rewrite the whole filesystem. But it is not mandatory, you probably just need to free some space on your filesystem.
You may find where most disk space is used with:

sudo du -sh --exclude=/proc --exclude=/run --exclude=/media /*

and then recursively look at biggest directories such as:

sudo du -sh --exclude=/proc --exclude=/run --exclude=/media /home/*

If you have some projects with big sources trees and build directories, you may just move these to an external disk. If it is not enough and it ends up that installed software is so big, then you may want to have an external partition mounted as suggested by @linuxdev. In all cases you need some external storage. You may use a SD Card, SATA or USB drive, and make a partition with ext4 filesystem.

#7

Moreover, largest files can be found with

du -a /home | sort -n -r | head -n 5

source
or with

find / -xdev -type f -size +100M

source

#8

Many thanks for your kind help.

#9

So you can’t solve it after all?

#10

If you ran out of disk space you simply have to free up some of it. The original post mentioned being out of space in “/tmp”, but this location is just coincidence, and it wouldn’t matter where you free space (though you could remove large content in “/tmp”).

Fixing running out of space depends on how much access you still have. For example, a regular GUI login will use temp files, and if there isn’t room for this, then no login is possible by that method. Serial console has far fewer requirements, and might get you in to delete excess content. If space is so filled as to not even be able to use serial console, then you have to flash. The trick to this is that you can clone first, repair the clone by removing content on the host PC, and then flash the clone. Much more involved than just deleting files, but you can save your content and flash using what the Jetson already had on it. Then the clone makes a very nice safety mechanism for backups.

#11

Sorry, I’m a beginner in Linux, so I’m telling you this. I may have a slightly different topic, but what I’m paying attention to is the tmpfs file system area. So, can’t I delete the tmpfs file system certainly, and can’t I get disk capacity by removing it?

#12

Tmpfs is not a real filesystem. This is known as a pseudo filesystem because this is content in RAM which is being presented like a file so that ordinary read/write commands can talk to the drivers. You cannot delete this content because it is not on the hard drive, and is a reflection of drivers and kernel code currently running. If you managed to shut down all of the drivers and kernel services creating that content, then you’d have a bit more RAM, but the actual eMMC disk storage would not change by even a single byte.