Giving error with installing wheel for pycocotools-fix when installing Tao-deploy using pip

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware : T4
• Network Type Dino
• TLT Version
• Training spec file(If have, please share here)
• How to reproduce the issue ? : running pip install nvidia-tao-deploy

Error
root@91f59cfea2c7:/workspace/ai_deployment_pipeline# pip install nvidia-tao-deploy
Collecting nvidia-tao-deploy
Using cached nvidia_tao_deploy-4.0.0.1-py3-none-any.whl (2.5 MB)
Requirement already satisfied: Pillow<9.0.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (8.4.0)
Requirement already satisfied: PyYAML>=5.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (6.0.1)
Requirement already satisfied: absl-py>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (2.0.0)
Requirement already satisfied: h5py==3.7.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.7.0)
Requirement already satisfied: hydra-core==1.2.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.2.0)
Requirement already satisfied: matplotlib>=3.0.3 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.7.3)
Requirement already satisfied: mpi4py>=3.0.3 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.1.5)
Requirement already satisfied: omegaconf==2.2.2 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (2.2.2)
Requirement already satisfied: onnx in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.12.0)
Requirement already satisfied: onnxruntime in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.16.2)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (4.8.1.78)
Requirement already satisfied: protobuf==3.20.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.20.1)
Collecting pycocotools-fix (from nvidia-tao-deploy)
Using cached pycocotools-fix-2.0.0.9.tar.gz (124 kB)
Preparing metadata (setup.py) … done
Requirement already satisfied: pynvml==11.0.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (11.0.0)
Requirement already satisfied: scikit-image==0.17.2 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (0.17.2)
Requirement already satisfied: scikit-learn==0.24.2 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (0.24.2)
Requirement already satisfied: scipy==1.5.4 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.5.4)
Requirement already satisfied: seaborn==0.7.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (0.7.1)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.16.0)
Requirement already satisfied: tqdm==4.64.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (4.64.0)
Requirement already satisfied: numpy>=1.14.5 in /usr/local/lib/python3.8/dist-packages (from h5py==3.7.0->nvidia-tao-deploy) (1.23.5)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy) (4.9.3)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy) (23.2)
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy) (6.1.1)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (3.1)
Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (2.32.0)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (2023.7.10)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (1.4.1)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.8/dist-packages (from scikit-learn==0.24.2->nvidia-tao-deploy) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn==0.24.2->nvidia-tao-deploy) (3.2.0)
Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from seaborn==0.7.1->nvidia-tao-deploy) (2.0.3)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (4.44.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (2.8.2)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.8/dist-packages (from onnx->nvidia-tao-deploy) (4.5.0)
Requirement already satisfied: coloredlogs in /usr/local/lib/python3.8/dist-packages (from onnxruntime->nvidia-tao-deploy) (15.0.1)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.8/dist-packages (from onnxruntime->nvidia-tao-deploy) (23.5.26)
Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from onnxruntime->nvidia-tao-deploy) (1.12)
Requirement already satisfied: setuptools>=18.0 in /usr/local/lib/python3.8/dist-packages (from pycocotools-fix->nvidia-tao-deploy) (68.2.2)
Requirement already satisfied: cython>=0.27.3 in /usr/local/lib/python3.8/dist-packages (from pycocotools-fix->nvidia-tao-deploy) (3.0.5)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources->hydra-core==1.2.0->nvidia-tao-deploy) (3.17.0)
Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.8/dist-packages (from coloredlogs->onnxruntime->nvidia-tao-deploy) (10.0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.8/dist-packages (from pandas->seaborn==0.7.1->nvidia-tao-deploy) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.8/dist-packages (from pandas->seaborn==0.7.1->nvidia-tao-deploy) (2023.3)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->onnxruntime->nvidia-tao-deploy) (1.3.0)
Building wheels for collected packages: pycocotools-fix
Building wheel for pycocotools-fix (setup.py) … error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [45 lines of output]
/tmp/pip-install-92q2yghx/pycocotools-fix_6f071d4e9a684254b35d5e5daf468175/setup.py:12: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************
  
  !!
    dist.Distribution().fetch_build_eggs(install_requires)
  /usr/local/lib/python3.8/dist-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!
  
          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************
  
  !!
    dist.fetch_build_eggs(dist.setup_requires)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-38
  creating build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  running build_ext
  Compiling pycocotools/_mask.pyx because it changed.
  [1/1] Cythonizing pycocotools/_mask.pyx
  /usr/local/lib/python3.8/dist-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-92q2yghx/pycocotools-fix_6f071d4e9a684254b35d5e5daf468175/pycocotools/_mask.pyx
    tree = Parsing.p_module(s, pxd, full_module_name)
  building 'pycocotools._mask' extension
  creating build/common
  creating build/temp.linux-x86_64-cpython-38
  creating build/temp.linux-x86_64-cpython-38/common
  creating build/temp.linux-x86_64-cpython-38/pycocotools
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/local/lib/python3.8/dist-packages/numpy/core/include -I./common -I/usr/include/python3.8 -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-38/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
  x86_64-linux-gnu-gcc: error: ../common/maskApi.c: No such file or directory
  x86_64-linux-gnu-gcc: fatal error: no input files
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycocotools-fix
Running setup.py clean for pycocotools-fix
Failed to build pycocotools-fix
ERROR: Could not build wheels for pycocotools-fix, which is required to install pyproject.toml-based projects

3

You are running in T4 instead of Jetson devices. It is not needed to run pip install nvidia-tao-deploy. You can run

4

Tao launcher and docker options are using some docker service to run the tao commands isn’t it

Can’t we directly use the tao features without running any additional docker service

I want to run inference of Tao models , the dino within my docker container

5

For your case, you can leverage the tao deploy source code for Dino. https://github.com/NVIDIA/tao_deploy/blob/main/nvidia_tao_deploy/cv/dino/scripts/inference.py.

Or tao pytorch source code for Dino. https://github.com/NVIDIA/tao_pytorch_backend/blob/main/nvidia_tao_pytorch/cv/dino/scripts/inference.py.

6

Morganh

I am currently trying on the solution you gave

When i try to leverage the script , with the tensorrt engine file that i have converted using TAO launcher , i got some errors, can you advise me on this

(tao_custom_deploy) ubuntu@ip-192-168-1-156:~/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts$ python inference.py
sys:1: UserWarning:
'config' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See [https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching](https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching) for migration instructions.
/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/common/hydra/hydra_runner.py:99: UserWarning:
'config' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See [https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching](https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching) for migration instructions.
_run_hydra(
/opt/conda/envs/tao_custom_deploy/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See [https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/](https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/) for more information.
ret = run_job(
Log file already exists at /home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/results/status.json
Starting dino inference.
################### engine path /home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/results/dino_model_v1.engine
[11/14/2023-06:11:26] [TRT] [E] 1: [runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
############# loaded engine <class 'bytes'>
############ TensorRT version: 8.6.1
'NoneType' object has no attribute 'create_execution_context'
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/common/decorators.py", line 63, in _func
raise e
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/common/decorators.py", line 47, in _func
runner(cfg, **kwargs)
File "inference.py", line 55, in main
trt_infer = DDETRInferencer(cfg.inference.trt_engine,
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/deformable_detr/inferencer.py", line 55, in __init__
super().__init__(engine_path)
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/inferencer/trt_inferencer.py", line 58, in __init__
self.context = self.engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Exception ignored in: <function DDETRInferencer.__del__ at 0x7f44e15145e0>
Traceback (most recent call last):
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/deformable_detr/inferencer.py", line 137, in __del__
if self.context:
AttributeError: 'DDETRInferencer' object has no attribute 'context'

image.png
image.png1293×760 97.8 KB

7

Above error log implied that the TensorRT version is not the same between building and running.
I think you ware building tensorrt engine in tao deploy docker. You can login it and check the TensorRT version and then make sure your custom docker has the same TensorRT version.

Or, since you are using your custom docker, maybe you can build a new docker while using the tao deploy docker as the base container.

8

Hi I ran with updated tensorrt version , the inference is working with the inference script from tao_deploy repository

Now i am experiencing and issue when i run my application with Flask app
when i initialize the inference class and run the prediction i get below warning and i get empty predictions

[TRT] [E] 1: [shapeHostToDeviceRunner.cpp::execute::48] Error Code 1: Cuda Runtime (invalid resource handle)

But i can able to run the script from same Flask app by using subprocess and run the inference script

Can you advice how i can make the inference work with the 1st way just directly initializing from the python using Flask

9

So, the inference is working with the inference script from tao_deploy repository now. Can you leverage it and check the difference why you get failed in your application?