Please provide the following information when requesting support.
• Hardware RTX3090
Configuration of the TAO Toolkit Instance
• Training spec fileunet_train_resnet_6S100.txt (1.3 KB)
run the training step as follows:
!tao unet train --gpus=1 --gpu_index=$GPU_INDEX \
-e $SPECS_DIR/unet_train_resnet_6S100.txt \
-r $USER_EXPERIMENT_DIR/unpruned \
-m $USER_EXPERIMENT_DIR/pretrained_resnet18/resnet_18.hdf5 \
-n model \
-k $KEY
Stops with error
OSError: [Errno 28] No space left on device: ‘/tmp/tmp0ld14dko’
The smallest amount of empty space in all my drives is 35GB
Training log attached here…training log 2022 11 01.txt (203.1 KB)
THANKS!!
Please check the result folder and delete some old .tlt models.
in order to generate less .tlt model.
That’s not the problem because the drive where the tao folder NOTEBOOK_ROOT is located has about 1TB unused, and the OS drive has 35GB unused…
Is the /tmp directory, referred to in the error, inside the docker image? If so, can I control on which drive the docker image gets saved?
My local /tmp folder has a total on 300kB so the error for No space left on device: ‘/tmp/tmp0ld14dko’ is mysterious…
Please check your ~/.tao_mounts.json file. It will map local folder to docker.
See more in TAO Toolkit Launcher — TAO Toolkit 3.22.05 documentation .
All mappings in the ~/.tao_mounts.json point to a drive with more that 1TB free. The folder ‘/tmp/tmp0ld14dko’ appears to be inside the docker.
Can you run below ?
More, can you share the ~/.tao_mounts.json file?
You can also login the docker to check why there is no space error.
2022-11-03 22:51:04,897 [INFO] root: Registry: [‘nvcr.io ’]nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
39GB total with 488 GB free (see below)
.tao_mounts.json (338 Bytes)
Not sure what I can do. The docker is supposed to grow as you put stuff in it. Wihich leads me to propose two ideas:
Allow control of the storage location of the tao docker (I am not very experienced with docker), or
Also map the tmp directory from within the docker so that the temporary directory can be mapped to whatever drive I have mopre space…
Thanks
From your tao_mount.json,
"Mounts": [
{
"source": "/mnt/DATA/MP/6S004C",
"destination": "/workspace/tao-experiments"
},
{
"source": "/mnt/DATA/MP/6S004C/specs",
"destination": "/workspace/tao-experiments/specs"
}
],
Did you ever mount other machine’s folder into your local /mnt/DATA folder ?
If the /mnt/DATA is really in your local machine, please try to use another directory and retry.
For example,
"Mounts": [
{
"source": "/home/yourname/DATA/MP/6S004C",
"destination": "/workspace/tao-experiments"
},
{
"source": "/home/yourname/DATA/MP/6S004C/specs",
"destination": "/workspace/tao-experiments/specs"
}
],
(TAO) david@AI01:~$ tao unet run /bin/bashnvcr.io ’]nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 @bf45ee3af7fb :/workspace$ df -h@bf45ee3af7fb :/workspace$
What’s your logic behind this? anything rooted in /home/user is in the OS root drive, and running out of space there can negatively and seriously impede the normal operation of the workstation!!!
In fact, when I started with TAO I had it setup that way and quickly crashed the computer because of that,
and /mnt/DATA has 488 GB free!!!
Still having the problem, which results in not being able to train overnight because it keeps stopping…
Morganh
November 4, 2022, 7:38am
11
When you run
What is the $USER_EXPERIMENT_DIR ?
env: USER_EXPERIMENT_DIR=/workspace/tao-experiments
Morganh
November 4, 2022, 7:44am
13
Can you run $df -h again outside the docker?
Morganh:
df -h
I have no name!@213f351247be :/workspace$ df -h
Seems to be mounting on the OS Boot Drive
If I could mount in another drive that would be a solution…!!!
Morganh
November 4, 2022, 7:49am
15
No, this is the result inside tao docker.
Morganh
November 4, 2022, 12:35pm
17
Can you try to change as below?
"Mounts": [
{
"source": "/mnt/sdb2/MP/6S004C",
"destination": "/workspace/tao-experiments"
},
{
"source": "/mnt/sdb2/MP/6S004C/specs",
"destination": "/workspace/tao-experiments/specs"
}
],
Morganh
November 7, 2022, 2:49am
18
Modify my previous comment as above. Try /mnt/sdb2
Morganh
November 7, 2022, 2:58am
19
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one.
More, below is the culprit for the error.
If it is /var/lib/docker, then there is not enough space for you to save the unet .tlt model because there is only 30G under “/”.
So, please change docker root dir.
system
December 6, 2022, 6:16am
21
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
