Running tao toolkit in google colab

luis95.garay · July 3, 2023, 10:15pm

Some weeks ago I was running without any problems in google colab the notebooks for training: yolo_v4, yolo_v4tiny, yolo_v3 and ssd (TAO Toolkit | NVIDIA NGC). Now, I am running the same notebooks from the same github repository without making any change and tao setup is not performed properly. Now I cannot run any model. When I run “https://github.com/NVIDIA-AI-IOT/nvidia-tao/blob/main/tensorflow/setup_env.sh” I got next messages

Successfully built addict future mpi4py posix-ipc prettytable pycocotools-fix pycuda recordclass retrying seaborn semver simplejson tabulate DLLogger promise pytools fire pathtools termcolor
Installing collected packages: zipp, typing-extensions, six, ipython-genutils, decorator, traitlets, setuptools, pyrsistent, importlib-metadata, attrs, wcwidth, tornado, pyzmq, python-dateutil, pyparsing, pycparser, ptyprocess, parso, nest-asyncio, jupyter-core, jsonschema, entrypoints, webencodings, pygments, prompt-toolkit, pickleshare, pexpect, packaging, nbformat, MarkupSafe, jupyter-client, jedi, cffi, backcall, async-generator, testpath, pandocfilters, nbclient, mistune, jupyterlab-pygments, jinja2, ipython, defusedxml, dataclasses, bleach, argon2-cffi-bindings, terminado, Send2Trash, prometheus-client, numpy, nbconvert, ipykernel, argon2-cffi, urllib3, smmap, protobuf, notebook, jmespath, h5py, docutils, widgetsnbextension, termcolor, scipy, qtpy, PyYAML, platformdirs, Pillow, orderedmultidict, onnx, kiwisolver, keras-preprocessing, keras-applications, jupyterlab-widgets, idna, gitdb, cycler, chardet, certifi, botocore, uritemplate, tifffile, threadpoolctl, shortuuid, setproctitle, sentry-sdk, s3transfer, requests, qtconsole, PyWavelets, pytz, pytools, pyjwt, psutil, promise, pathtools, pathlib2, onnxconverter-common, networkx, matplotlib, mako, llvmlite, keras, jupyter-console, joblib, ipywidgets, imageio, GitPython, future, furl, flatbuffers, fire, docker-pycreds, cython, Click, appdirs, xmltodict, wandb, uplink, uff, tqdm, toposort, tf2onnx, tabulate, simplejson, shapely, semver, seaborn, scikit-learn, scikit-image, retrying, requests-toolbelt, recordclass, pycuda, pycocotools-fix, pyarrow, prettytable, posix-ipc, pandas, opencv-python, onnxruntime, onnx-graphsurgeon, nvidia-ml-py, numba, mpi4py, keras2onnx, keras-metrics, jupyter, grpcio, graphsurgeon, DLLogger, cryptography, clearml, boto3, argparse, argcomplete, addict
ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nvidia-tao 4.0.0 requires idna==2.10, but you have idna 2.7 which is incompatible.
nvidia-tao 4.0.0 requires six==1.15.0, but you have six 1.13.0 which is incompatible.
nvidia-tao 4.0.0 requires tabulate==0.8.7, but you have tabulate 0.7.5 which is incompatible.
nvidia-tao 4.0.0 requires urllib3>=1.26.5, but you have urllib3 1.24.3 which is incompatible.
google-colab 1.0.0 requires ipykernel~=4.6.0, but you have ipykernel 5.5.6 which is incompatible.
google-colab 1.0.0 requires ipython~=5.5.0, but you have ipython 7.16.3 which is incompatible.
google-colab 1.0.0 requires notebook~=5.2.0, but you have notebook 6.4.10 which is incompatible.
google-colab 1.0.0 requires pandas~=0.24.0, but you have pandas 0.25.3 which is incompatible.
google-colab 1.0.0 requires requests~=2.21.0, but you have requests 2.20.1 which is incompatible.
google-colab 1.0.0 requires six~=1.12.0, but you have six 1.13.0 which is incompatible.
google-colab 1.0.0 requires tornado~=4.5.0, but you have tornado 6.1 which is incompatible.

Then, when I try to run any command of tao like !tao yolo_v4 -h I got next error:

Using TensorFlow backend.
Traceback (most recent call last):
File “/usr/local/bin/yolo_v4”, line 5, in
from iva.yolo_v4.entrypoint.yolo_v4 import main
File “/usr/local/lib/python3.6/dist-packages/iva/init.py”, line 10, in
import third_party.keras.mixed_precision as MP
File “/usr/local/lib/python3.6/dist-packages/third_party/keras/mixed_precision.py”, line 11, in
import keras
File “/usr/local/lib/python3.6/dist-packages/keras/init.py”, line 3, in
from . import utils
File “/usr/local/lib/python3.6/dist-packages/keras/utils/init.py”, line 6, in
from . import conv_utils
File “/usr/local/lib/python3.6/dist-packages/keras/utils/conv_utils.py”, line 9, in
from … import backend as K
File “/usr/local/lib/python3.6/dist-packages/keras/backend/init.py”, line 89, in
from .tensorflow_backend import *
File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 5, in
import tensorflow as tf
ModuleNotFoundError: No module named ‘tensorflow’

What could be the problem? maybe recent updates of google colab?

General information:
• Hardware (T4)
• Google Colab (Yolo_v4/Yolo_v4tiny/Yolo_v3/etc). I want to emphasize that I am running in google colab and not in a docker container. In google colab we cannot run the containers. I am running in colab because I do not have a local machine with GPU

• How to reproduce the issue ? You could reproduce it by only running the notebooks from the oficial github in google colab:

Morganh · July 4, 2023, 2:03pm

There is not update recently. Could you save the .ipynb and upload here?

luis95.garay · July 4, 2023, 2:28pm

Sure, I share the notebook:
yolo_v4.ipynb (34.1 KB)

It is the same of the repository:

github.com

NVIDIA-AI-IOT/nvidia-tao/blob/main/tensorflow/yolo_v4/yolo_v4.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"LGLBrzF8hKgS"},"source":["## Switch to CPU Instance (Advisable only for Non Colab-Pro instance)\n","\n","1. Switch to CPU Instance for until Step 2 for non GPU dependent tasks\n","2. This increases your time available for the GPU dependent tasks on a Colab instance\n","2. Change Runtime type to CPU by Runtime(Top Left tab)->Change Runtime Type->None(Hardware Accelerator)\n","3.   Then click on Connect (Top Right)\n","\n"]},{"cell_type":"markdown","metadata":{"id":"SjpjyNg5c2V9"},"source":["## Mounting Google drive\n","Mount your Google drive storage to this Colab instance"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":18056,"status":"ok","timestamp":1658620835786,"user":{"displayName":"tao-team-google-admins nvidia","userId":"12428471056663112246"},"user_tz":420},"id":"EvUVkYw0hzqG","outputId":"a8f580a7-bd55-4a9c-8620-c0795752fbc9"},"outputs":[],"source":["try:\n","    import google.colab\n","    %env GOOGLE_COLAB=1\n","    from google.colab import drive\n","    drive.mount('/content/drive', force_remount=True)\n","except:\n","    %env GOOGLE_COLAB=0\n","    print(\"Warning: Not a Colab Environment\")"]},{"cell_type":"markdown","metadata":{"id":"0g04STEGxKFM"},"source":["# Object Detection using TAO YOLOv4\n","\n","Transfer learning is the process of transferring learned features from one application to another. It is a commonly used training technique where you use a model trained on one task and re-train to use it on a different task. \n","\n","Train Adapt Optimize (TAO) Toolkit  is a simple and easy-to-use Python based AI toolkit for taking purpose-built AI models and customizing them with users' own data.\n","\n","<img align=\"center\" src=\"https://developer.nvidia.com/sites/default/files/akamai/TAO/tlt-tao-toolkit-bring-your-own-model-diagram.png\" width=\"1080\">\n"]},{"cell_type":"markdown","metadata":{"id":"YlWylr65xKFO"},"source":["## Learning Objectives\n","In this notebook, you will learn how to leverage the simplicity and convenience of TAO to:\n","\n","* Take a pretrained resnet18 model and train a ResNet-18 Yolo_v4 model on the KITTI dataset\n","* Prune the trained yolo_V4 model\n","* Retrain the pruned model to recover lost accuracy\n","* Export the pruned model\n","* Quantize the pruned model using QAT\n","* Run Inference on the trained model\n","* Export the pruned, quantized and retrained model to a .etlt file for deployment to DeepStream\n","\n","## Table of Contents\n","\n","This notebook shows an example usecase of YOLO v4 object detection using Train Adapt Optimize (TAO) Toolkit.\n","\n","0. [Set up env variables](#head-0)\n","1. [Prepare dataset and pre-trained model](#head-1) <br>\n","     1.1 [Download the dataset](#head-1-1)<br>\n","     1.2 [Verify the downloaded dataset](#head-1-2)<br>\n","     1.3 [Download pretrained model](#head-1-3)\n","2. [Setup GPU environment](#head-2) <br>\n","    2.1 [Connect to GPU Instance](#head-2-1) <br>\n","    2.2 [Mounting Google drive](#head-2-2) <br>\n","    2.3 [Setup Python environment](#head-2-3) <br>\n","    2.4 [Reset env variables](#head-2-4) <br>\n","3. [Generate TF records](#head-3)\n","4. [Provide training specification](#head-4)\n","5. [Run TAO training](#head-5)\n","6. [Evaluate trained models](#head-6)\n","7. [Prune trained models](#head-7)\n","8. [Retrain pruned models](#head-8)\n","9. [Evaluate retrained model](#head-9)\n","10. [Visualize inferences](#head-10)\n"]},{"cell_type":"markdown","metadata":{"id":"e1RkCyTnxKFP"},"source":["#### Note\n","1. This notebook currently is by default set up to run training using 1 GPU. To use more GPU's please update the env variable `$NUM_GPUS` accordingly\n","1. This notebook uses KITTI dataset by default, which should be around ~12 GB. If you are limited by Google-Drive storage, we recommend to:\n","\n","    i. Download the dataset onto the local system\n","\n","    ii. Run the utility script at $COLAB_NOTEBOOKS/tensorflow/utils/generate_kitti_subset.py in your local system\n","\n","    iii. This generates a subset of coco dataset with number of sample images you wish for\n","\n","    iv. Upload this subset onto Google Drive\n","\n","1. Using the default config/spec file provided in this notebook, each weight file size of yolo_v4 created during training will be ~400 MB\n","\n","## 0. Set up env variables and set FIXME parameters <a class=\"anchor\" id=\"head-0\"></a>\n","\n","*Note: This notebook currently is by default set up to run training using 1 GPU. To use more GPU's please update the env variable `$NUM_GPUS` accordingly*\n","\n","#### FIXME\n","1. NUM_GPUS - set this to <= number of GPU's availble on the instance\n","1. COLAB_NOTEBOOKS_PATH - for Google Colab environment, set this path where you want to clone the repo to; for local system environment, set this path to the already cloned repo\n","1. EXPERIMENT_DIR - set this path to a folder location wh

This file has been truncated. show original

Morganh · July 4, 2023, 2:30pm

Do you have any log when you run this notebook? If yes, please save it and share with us.

img1518 · July 4, 2023, 3:50pm

I’m having the same error as this guy. When I dug into the problem and checked it, I found that there was an error “python3.6 -m pip install https://developer.download.nvidia.com/compute/redist/nvidia-tensorflow/nvidia_tensorflow-1.15.4+nv20.10-cp36-cp36m-linux_x86_64.whl” in the “setup_env.sh”

The error code that occurred when executing the code is as follows.
ERROR: Could not find a version that satisfies the requirement nvidia-cuda-cupti==11.1.69 (from nvidia-tensorflow) (from versions: 0.0.1.dev4, 0.0.1.dev5)
ERROR: No matching distribution found for nvidia-cuda-cupti==11.1.69

How can I solve it?

Morganh · July 4, 2023, 4:11pm

Can you save the .ipynb file and share with us?

img1518 · July 4, 2023, 5:59pm

I didn’t save any files separately, but after going through that process, I tried running the error code that occured again.

[Set up env variables]
[Prepare dataset and pre-trained model]
1.1 [Download the dataset](
1.2 [Verify the downloaded dataset]
1.3 [Download pretrained model]
[Setup GPU environment]
2.1 [Connect to GPU Instance]
2.2 [Mounting Google drive]
2.3 [Setup Python environment]
2.4 [Reset env variables]

and I tried "!pip install tensorflow==1.15.4 before the method presented above, but the process also encountered an error while executing “!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_kitti_train.txt \ -o $DATA_DIR/training/tfrecords/train” code.

Morganh · July 5, 2023, 1:34am

Can you click as below and then save the log to an .ipynb file?

img1518 · July 5, 2023, 12:39pm

yolo_v4_tiny.ipynb (615.0 KB)

Morganh · July 5, 2023, 12:48pm

Thanks, I can also reproduce the error. Will check further internally.

luis95.garay · July 5, 2023, 2:50pm

Getting the exact same error messages as @img1518

aneetachristopher94 · July 6, 2023, 5:25pm

I ma also getting the same error. What should we do to resolve this?

Morganh · July 7, 2023, 12:30am

It is a sudden issue since last week. We are still working on that. Will update to you if any. Sorry for inconvenient.

Morganh · July 9, 2023, 11:11am

Please modify setup_env.sh to below.

#!/bin/sh

# Install Python 3.6 as the default version
sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt-get update
sudo apt-get install python3.6 -y
apt install python3-pip -y
apt-get install python3.6-distutils
apt-get install python3.6-dev

# Set Python 3.6 as the default version
rm /usr/bin/python
ln -sf /usr/bin/python3.6 /usr/bin/python3
ln -sf /usr/bin/python3.6 /usr/local/bin/python

python3.6 -m pip install --upgrade pip
python3.6 -m pip install google-colab
python3.6 -m pip install nvidia-pyindex

# Install Tensorflow
python3.6 -m pip install https://developer.download.nvidia.com/compute/redist/nvidia-horovod/nvidia_horovod-0.20.0+nv20.10-cp36-cp36m-linux_x86_64.whl
python3.6 -m pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-tensorflow==1.15.4+nv20.10

# Install Cmake
cd /tmp
wget https://github.com/Kitware/CMake/releases/download/v3.14.4/cmake-3.14.4-Linux-x86_64.sh
chmod +x cmake-3.14.4-Linux-x86_64.sh
./cmake-3.14.4-Linux-x86_64.sh --prefix=/usr/local --exclude-subdir --skip-license
rm ./cmake-3.14.4-Linux-x86_64.sh

# Install dependencies
python3.6 -m pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-eff==0.5.3
python3.6 -m pip install nvidia-tao==4.0.0
python3.6 -m pip install --ignore-installed PyYAML -r /content/drive/MyDrive/nvidia-tao/tensorflow/requirements-pip.txt -f https://download.pytorch.org/whl/torch_stable.html --extra-index-url https://developer.download.nvidia.com/compute/redist

# Install code related wheels
python3.6 -m pip install nvidia-tao-tf1==4.0.0.657.dev0

Topic		Replies	Views
Can't run the provided TAO toolkit sample code TAO Toolkit	20	2322	July 31, 2023
TAO Toolkit Google colab Error TAO Toolkit	6	929	February 22, 2024
Model training using Tao toolkit with colab not working anymore TAO Toolkit	4	150	September 9, 2025
Running TAO Toolkit on Google Colab TAO Toolkit	8	805	February 22, 2024
Google Colab ssd.ipynb no python3.6 TAO Toolkit	35	3692	February 22, 2024
Tao in Google Colab error: /bin/bash: line 1: tao: command not found TAO Toolkit	4	1948	February 22, 2024
TAO on Google Colab TAO Toolkit	6	1063	February 22, 2024
Nvidia TAO Yolo_v3 training failure TAO Toolkit yolo , tao	7	552	June 11, 2024
TAO example for YoloV3 in Google Colab fails to execute the training TAO Toolkit	1	412	March 1, 2023
TAO yolo_v3 google colab training failure TAO Toolkit	5	345	May 14, 2024

Running tao toolkit in google colab

Related topics