tensorrt-8.0.1.6
Quadro RTX 5000 dual GPU
Driver Version: 470.82.00
CUDA Version: 11.4
Ubuntu 18.04
python 3.6
Yolo_v4
nvidia/tao/tao-toolkit-tf:
docker_registry: nvcr.io
docker_tag: v3.21.08-py3
Current dependency
Ubuntu 18.04 LTS
18.04
python
=3.6.9
docker-ce
19.03.5
docker-API
1.40
nvidia-container-toolkit
1.3.0-1
nvidia-docker2
2.5.0-1
nvidia-driver
455
python-pip
21.06
I have earlier trained my custom model using this TAO toolkit,but now i can’t mount my folder properly
I have upgraded my driver from 455 to 470 after that
and also upgraded tensort to 8
and cuda from 11.1 to 11.4 after that
all the dependencies are satisfied expect that i can’t figure out how to check the version of my current nvidia-container-runtime
i looked up forms and they asked to perform i get this
tlt yolo_v4 run ls /home/vaaan/tlt-experiments/yolo_v4/specs/yolo_v4_retrain_resnet18_kitti.txt
/home/vaaan/.local/lib/python3.6/site-packages/tlt/__init__.py:20:
DeprecationWarning:
The nvidia-tlt
package will be deprecated soon. Going forward please migrate to using the nvidia-tao
package.
warnings.warn(message, DeprecationWarning)
2021-12-20 17:05:15,879 [INFO] root: Registry: [‘nvcr.io’]
2021-12-20 17:05:15,958 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module’s documentation for alternative uses
import fnmatch, glob, traceback, errno, sys, atexit, locale, imp, stat
Traceback (most recent call last):
File “/home/vaaan/.local/bin/tlt”, line 8, in
sys.exit(main())
File “/home/vaaan/.local/lib/python3.6/site-packages/tlt/entrypoint/entrypoint.py”, line 115, in main
args[1:]
File “/home/vaaan/.local/lib/python3.6/site-packages/tlt/components/instance_handler/local_instance.py”, line 319, in launch_command
docker_handler.run_container(command)
File “/home/vaaan/.local/lib/python3.6/site-packages/tlt/components/docker_handler/docker_handler.py”, line 284, in run_container
mount_data, env_vars, docker_options = self._get_mount_env_data()
File “/home/vaaan/.local/lib/python3.6/site-packages/tlt/components/docker_handler/docker_handler.py”, line 112, in _get_mount_env_data
raise ValueError(“Mount point source path doesn’t exist. {}”.format(mount[‘source’]))
ValueError: Mount point source path doesn’t exist. /home/vaaan/YOUR_LOCAL_PROJECT_DIR_PATH
I use these codes to launch docker :
wget -O ngccli_linux.zip https://ngc.nvidia.com/downloads/ngccli_linux.zip && unzip -o ngccli_linux.zip && chmod u+x ngc
md5sum -c ngc.md5
echo "export PATH=\"\$PATH:$(pwd)\"" >> ~/.bash_profile && source ~/.bash_profile
ngc config set
docker login nvcr.io
docker run --runtime=nvidia -it -v /home/vaaan/tlt-experiments:/workspace/tlt-experiments -p 8888:8888 nvcr.io/nvidia/tlt-streamanalytics:v3.0-py3 /bin/bash
jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root
Setting up env variables for cleaner command line commands.
import os
print(“Please replace the variable with your key.”)
%env KEY=aGJuM2dxZGltbjgwaGNnb3Fxc2h0ZXBqZGk6MzlkYjAxY2EtZWE2OC00NGRiLWI5ZmUtZWRlNDZjMTI4MjA5
%env USER_EXPERIMENT_DIR=/workspace/tlt-experiments/yolo_v4
%env DATA_DOWNLOAD_DIR=/workspace/tlt-experiments/data
Set this path if you don’t run the notebook from the samples directory.
#%env NOTEBOOK_ROOT=/home/vaaan/tlt_cv_samples_vv1.0.1/
Please define this local project directory that needs to be mapped to the TLT docker session.
The dataset expected to be present in $LOCAL_PROJECT_DIR/data, while the results for the steps
in this notebook will be stored at $LOCAL_PROJECT_DIR/detectnet_v2
%env LOCAL_PROJECT_DIR=/home/vaaan/tlt-experiments
os.environ[“LOCAL_DATA_DIR”] = os.path.join(os.getenv(“LOCAL_PROJECT_DIR”, os.getcwd()), “data”)
os.environ[“LOCAL_EXPERIMENT_DIR”] = os.path.join(os.getenv(“LOCAL_PROJECT_DIR”, os.getcwd()), “yolo_v4”)
The sample spec files are present in the same path as the downloaded samples.
os.environ[“LOCAL_SPECS_DIR”] = os.path.join(
os.getenv(“NOTEBOOK_ROOT”, os.getcwd()),
“specs”
)
%env SPECS_DIR=/workspace/tlt-experiments/yolo_v4/specs
Showing list of specification files.
!ls -rlt $LOCAL_SPECS_DIR
Please replace the variable with your key.
env: KEY=aGJuM2dxZGltbjgwaGNnb3Fxc2h0ZXBqZGk6MzlkYjAxY2EtZWE2OC00NGRiLWI5ZmUtZWRlNDZjMTI4MjA5
env: USER_EXPERIMENT_DIR=/workspace/tlt-experiments/yolo_v4
env: DATA_DOWNLOAD_DIR=/workspace/tlt-experiments/data
env: LOCAL_PROJECT_DIR=/home/vaaan/tlt-experiments
env: SPECS_DIR=/workspace/tlt-experiments/yolo_v4/specs
total 8
-rw------- 1 1001 1001 3277 Dec 17 07:06 yolo_v4_train_resnet18_kitti.txt
-rw------- 1 1001 1001 3175 Dec 17 07:06 yolo_v4_retrain_resnet18_kitti.txt
Mapping up the local directories to the TLT docker.
import json
mounts_file = os.path.expanduser(“~/.tlt_mounts.json”)
Define the dictionary with the mapped drives
drive_map = {
“Mounts”: [
# Mapping the data directory
{
“source”: os.environ[“LOCAL_PROJECT_DIR”],
“destination”: “/workspace/tlt-experiments”
},
# Mapping the specs directory.
{
“source”: os.environ[“LOCAL_SPECS_DIR”],
“destination”: os.environ[“SPECS_DIR”]
},
]
}
Writing the mounts file.
with open(mounts_file, “w”) as mfile:
json.dump(drive_map, mfile, indent=4)
`!cat ~/.tlt_mounts.json`
{
"Mounts": [
{
"source": "/home/vaaan/tlt-experiments",
"destination": "/workspace/tlt-experiments"
},
{
"source": "/workspace/tlt-experiments/yolo_v4/specs",
"destination": "/workspace/tlt-experiments/yolo_v4/specs"
}
]
}
View the versions of the TLT launcher
!tlt info
/usr/local/lib/python3.6/dist-packages/tlt/__init__.py:20: DeprecationWarning:
The `nvidia-tlt` package will be deprecated soon. Going forward please migrate to using the `nvidia-tao` package.
warnings.warn(message, DeprecationWarning)
~/.tao_mounts.json wasn't found. Falling back to obtain mount points and docker configs from ~/.tlt_mounts.json.
Please note that this will be deprecated going forward.
Configuration of the TAO Toolkit Instance
dockers: ['nvidia/tao/tao-toolkit-tf', 'nvidia/tao/tao-toolkit-pyt', 'nvidia/tao/tao-toolkit-lm']
format_version: 2.0
toolkit_version: 3.21.11
published_date: 11/08/2021
verify
import os
DATA_DIR = os.environ.get(‘LOCAL_DATA_DIR’)
num_training_images = len(os.listdir(os.path.join(DATA_DIR, “training/image_2”)))
num_training_labels = len(os.listdir(os.path.join(DATA_DIR, “training/label_2”)))
num_testing_images = len(os.listdir(os.path.join(DATA_DIR, “testing/image_2”)))
print(“Number of images in the train/val set. {}”.format(num_training_images))
print(“Number of labels in the train/val set. {}”.format(num_training_labels))
print(“Number of images in the test set. {}”.format(num_testing_images))
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-11-d9f1ad23d866> in <module>
3
4 DATA_DIR = os.environ.get('LOCAL_DATA_DIR')
----> 5 num_training_images = len(os.listdir(os.path.join(DATA_DIR, "training/image_2")))
6 num_training_labels = len(os.listdir(os.path.join(DATA_DIR, "training/label_2")))
7 num_testing_images = len(os.listdir(os.path.join(DATA_DIR, "testing/image_2")))
FileNotFoundError: [Errno 2] No such file or directory: '/home/vaaan/tlt-experiments/data/training/image_2'