$ nvidia-smi
Wed May 24 10:49:39 2023
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03 Driver Version: 530.41.03 CUDA Version: 12.1 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce GTX 1650 Ti Off| 00000000:01:00.0 Off | N/A |
| N/A 36C P0 5W / N/A| 10MiB / 4096MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1147 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 2887 G /usr/lib/xorg/Xorg 4MiB |
±--------------------------------------------------------------------------------------+
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
$ tao info
/home/mj/anaconda3/envs/tao/lib/python3.6/site-packages/tlt/init.py:20: DeprecationWarning:
The nvidia-tlt package will be deprecated soon. Going forward please migrate to using the nvidia-tao package.
warnings.warn(message, DeprecationWarning)
Configuration of the TAO Toolkit Instance
dockers: [‘nvidia/tao/tao-toolkit-tf’, ‘nvidia/tao/tao-toolkit-pyt’, ‘nvidia/tao/tao-toolkit-lm’]
format_version: 2.0
toolkit_version: 3.22.02
published_date: 02/28/2022
$docker version
Client: Docker Engine - Community
Cloud integration: v1.0.31
Version: 24.0.1
API version: 1.42 (downgraded from 1.43)
Go version: go1.20.4
Git commit: 6802122
Built: Fri May 19 18:06:24 2023
OS/Arch: linux/amd64
Context: desktop-linux
Server: Docker Desktop 4.19.0 (106363)
Engine:
Version: 23.0.5
API version: 1.42 (minimum version 1.12)
Go version: go1.19.8
Git commit: 94d3ad6
Built: Wed Apr 26 16:17:45 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.20
GitCommit: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
runc:
Version: 1.1.5
GitCommit: v1.1.5-0-gf19387a
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ ~/.tao_mounts.json :
{
“Mounts”: [
{
“source”: “/home/mj/TAO/workspace/tao-experiments/data”,
“destination”: “/home/mj/TAO/workspace/tao-experiments/data”
},
{
“source”: “/home/mj/TAO/workspace/tao-experiments/results”,
“destination”: “/home/mj/TAO/workspace/tao-experiments/results”
},
{
“source”: “/home/mj/TAO/workspace/tao-experiments/specs”,
“destination”: “/home/mj/TAO/workspace/tao-experiments/specs”
}
],
“Envs”: [
{
“variable”: “CUDA_DEVICE_ORDER”,
“value”: “PCI_BUS_ID”
}
],
“DockerOptions”: {
“shm_size”: “16G”,
“ulimits”: {
“memlock”: -1,
“stack”: 67108864
},
“user”: “1000:1000”,
“ports”: {
“8888”: 8888
}
}
}
i tried to run tao with docker but i faced many problemes with that can you help me ??
sudo docker run -it --rm --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/mj/TAO/workspace:/workspace nvcr.io/nvidia/tao/tao-toolkit:4.0.1-tf1.15.5 mask_rcnn dataset_convert -i /home/mj/TAO/workspace/tao-experiments/data/raw-data/val2017 -a /home/mj/TAO/workspace/tao-experiments/data/raw-data/annotations/instances_val2017.json -o /home/mj/TAO/workspace/tao-experiments/data --include_masks -t val -s 32
the error :
==============================
=== TAO Toolkit TensorFlow ===
NVIDIA Release 4.0.1-TensorFlow (build )
TAO Toolkit Version 4.0.1
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This container image and its contents are governed by the TAO Toolkit End User License Agreement.
By pulling and using the container, you accept the terms and conditions of this license:
Using TensorFlow backend.
2023-05-24 09:31:21.872547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
2023-05-24 09:31:25.034921: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-05-24 09:31:25.498955: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libnvinfer.so.8
2023-05-24 09:31:25.513538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/init.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
Traceback (most recent call last):
File “</usr/local/lib/python3.6/dist-packages/iva/mask_rcnn/scripts/dataset_convert.py>”, line 3, in
File “”, line 415, in
File “”, line 403, in
File “”, line 292, in main
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/mj/TAO/workspace/tao-experiments/data’
Telemetry data couldn’t be sent, but the command ran successfully.
[WARNING]: <urlopen error [Errno -2] Name or service not known>
Execution status: FAIL