Giving error with installing wheel for pycocotools-fix when installing Tao-deploy using pip

Please provide the following information when requesting support.

• Hardware : T4
• Network Type Dino
• TLT Version
• Training spec file(If have, please share here)
• How to reproduce the issue ? : running pip install nvidia-tao-deploy

Error
root@91f59cfea2c7:/workspace/ai_deployment_pipeline# pip install nvidia-tao-deploy
Collecting nvidia-tao-deploy
Using cached nvidia_tao_deploy-4.0.0.1-py3-none-any.whl (2.5 MB)
Requirement already satisfied: Pillow<9.0.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (8.4.0)
Requirement already satisfied: PyYAML>=5.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (6.0.1)
Requirement already satisfied: absl-py>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (2.0.0)
Requirement already satisfied: h5py==3.7.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.7.0)
Requirement already satisfied: hydra-core==1.2.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.2.0)
Requirement already satisfied: matplotlib>=3.0.3 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.7.3)
Requirement already satisfied: mpi4py>=3.0.3 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.1.5)
Requirement already satisfied: omegaconf==2.2.2 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (2.2.2)
Requirement already satisfied: onnx in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.12.0)
Requirement already satisfied: onnxruntime in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.16.2)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (4.8.1.78)
Requirement already satisfied: protobuf==3.20.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (3.20.1)
Collecting pycocotools-fix (from nvidia-tao-deploy)
Using cached pycocotools-fix-2.0.0.9.tar.gz (124 kB)
Preparing metadata (setup.py) … done
Requirement already satisfied: pynvml==11.0.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (11.0.0)
Requirement already satisfied: scikit-image==0.17.2 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (0.17.2)
Requirement already satisfied: scikit-learn==0.24.2 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (0.24.2)
Requirement already satisfied: scipy==1.5.4 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.5.4)
Requirement already satisfied: seaborn==0.7.1 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (0.7.1)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (1.16.0)
Requirement already satisfied: tqdm==4.64.0 in /usr/local/lib/python3.8/dist-packages (from nvidia-tao-deploy) (4.64.0)
Requirement already satisfied: numpy>=1.14.5 in /usr/local/lib/python3.8/dist-packages (from h5py==3.7.0->nvidia-tao-deploy) (1.23.5)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy) (4.9.3)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy) (23.2)
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.8/dist-packages (from hydra-core==1.2.0->nvidia-tao-deploy) (6.1.1)
Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (3.1)
Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (2.32.0)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (2023.7.10)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from scikit-image==0.17.2->nvidia-tao-deploy) (1.4.1)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.8/dist-packages (from scikit-learn==0.24.2->nvidia-tao-deploy) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn==0.24.2->nvidia-tao-deploy) (3.2.0)
Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from seaborn==0.7.1->nvidia-tao-deploy) (2.0.3)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (1.1.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (4.44.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib>=3.0.3->nvidia-tao-deploy) (2.8.2)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.8/dist-packages (from onnx->nvidia-tao-deploy) (4.5.0)
Requirement already satisfied: coloredlogs in /usr/local/lib/python3.8/dist-packages (from onnxruntime->nvidia-tao-deploy) (15.0.1)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.8/dist-packages (from onnxruntime->nvidia-tao-deploy) (23.5.26)
Requirement already satisfied: sympy in /usr/local/lib/python3.8/dist-packages (from onnxruntime->nvidia-tao-deploy) (1.12)
Requirement already satisfied: setuptools>=18.0 in /usr/local/lib/python3.8/dist-packages (from pycocotools-fix->nvidia-tao-deploy) (68.2.2)
Requirement already satisfied: cython>=0.27.3 in /usr/local/lib/python3.8/dist-packages (from pycocotools-fix->nvidia-tao-deploy) (3.0.5)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources->hydra-core==1.2.0->nvidia-tao-deploy) (3.17.0)
Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.8/dist-packages (from coloredlogs->onnxruntime->nvidia-tao-deploy) (10.0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.8/dist-packages (from pandas->seaborn==0.7.1->nvidia-tao-deploy) (2023.3.post1)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.8/dist-packages (from pandas->seaborn==0.7.1->nvidia-tao-deploy) (2023.3)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.8/dist-packages (from sympy->onnxruntime->nvidia-tao-deploy) (1.3.0)
Building wheels for collected packages: pycocotools-fix
Building wheel for pycocotools-fix (setup.py) … error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [45 lines of output]
/tmp/pip-install-92q2yghx/pycocotools-fix_6f071d4e9a684254b35d5e5daf468175/setup.py:12: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************
  
  !!
    dist.Distribution().fetch_build_eggs(install_requires)
  /usr/local/lib/python3.8/dist-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!
  
          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************
  
  !!
    dist.fetch_build_eggs(dist.setup_requires)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-38
  creating build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/mask.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/__init__.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  copying pycocotools/coco.py -> build/lib.linux-x86_64-cpython-38/pycocotools
  running build_ext
  Compiling pycocotools/_mask.pyx because it changed.
  [1/1] Cythonizing pycocotools/_mask.pyx
  /usr/local/lib/python3.8/dist-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-92q2yghx/pycocotools-fix_6f071d4e9a684254b35d5e5daf468175/pycocotools/_mask.pyx
    tree = Parsing.p_module(s, pxd, full_module_name)
  building 'pycocotools._mask' extension
  creating build/common
  creating build/temp.linux-x86_64-cpython-38
  creating build/temp.linux-x86_64-cpython-38/common
  creating build/temp.linux-x86_64-cpython-38/pycocotools
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/local/lib/python3.8/dist-packages/numpy/core/include -I./common -I/usr/include/python3.8 -c ../common/maskApi.c -o build/temp.linux-x86_64-cpython-38/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
  x86_64-linux-gnu-gcc: error: ../common/maskApi.c: No such file or directory
  x86_64-linux-gnu-gcc: fatal error: no input files
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycocotools-fix
Running setup.py clean for pycocotools-fix
Failed to build pycocotools-fix
ERROR: Could not build wheels for pycocotools-fix, which is required to install pyproject.toml-based projects

You are running in T4 instead of Jetson devices. It is not needed to run pip install nvidia-tao-deploy. You can run

Tao launcher and docker options are using some docker service to run the tao commands isn’t it

Can’t we directly use the tao features without running any additional docker service

I want to run inference of Tao models , the dino within my docker container

For your case, you can leverage the tao deploy source code for Dino. https://github.com/NVIDIA/tao_deploy/blob/main/nvidia_tao_deploy/cv/dino/scripts/inference.py.

Or tao pytorch source code for Dino. https://github.com/NVIDIA/tao_pytorch_backend/blob/main/nvidia_tao_pytorch/cv/dino/scripts/inference.py.

Morganh

I am currently trying on the solution you gave

When i try to leverage the script , with the tensorrt engine file that i have converted using TAO launcher , i got some errors, can you advise me on this

(tao_custom_deploy) ubuntu@ip-192-168-1-156:~/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts$ python inference.py
sys:1: UserWarning:
'config' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See [https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching](https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching) for migration instructions.
/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/common/hydra/hydra_runner.py:99: UserWarning:
'config' is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See [https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching](https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching) for migration instructions.
_run_hydra(
/opt/conda/envs/tao_custom_deploy/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See [https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/](https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/) for more information.
ret = run_job(
Log file already exists at /home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/results/status.json
Starting dino inference.
################### engine path /home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/results/dino_model_v1.engine
[11/14/2023-06:11:26] [TRT] [E] 1: [runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
############# loaded engine <class 'bytes'>
############ TensorRT version: 8.6.1
'NoneType' object has no attribute 'create_execution_context'
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/common/decorators.py", line 63, in _func
raise e
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/common/decorators.py", line 47, in _func
runner(cfg, **kwargs)
File "inference.py", line 55, in main
trt_infer = DDETRInferencer(cfg.inference.trt_engine,
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/deformable_detr/inferencer.py", line 55, in __init__
super().__init__(engine_path)
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/inferencer/trt_inferencer.py", line 58, in __init__
self.context = self.engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Exception ignored in: <function DDETRInferencer.__del__ at 0x7f44e15145e0>
Traceback (most recent call last):
File "/home/ubuntu/repositories/tao_deploy/nvidia_tao_deploy/cv/dino/scripts/../../../../nvidia_tao_deploy/cv/deformable_detr/inferencer.py", line 137, in __del__
if self.context:
AttributeError: 'DDETRInferencer' object has no attribute 'context'

Above error log implied that the TensorRT version is not the same between building and running.
I think you ware building tensorrt engine in tao deploy docker. You can login it and check the TensorRT version and then make sure your custom docker has the same TensorRT version.

Or, since you are using your custom docker, maybe you can build a new docker while using the tao deploy docker as the base container.

Hi I ran with updated tensorrt version , the inference is working with the inference script from tao_deploy repository

Now i am experiencing and issue when i run my application with Flask app
when i initialize the inference class and run the prediction i get below warning and i get empty predictions

[TRT] [E] 1: [shapeHostToDeviceRunner.cpp::execute::48] Error Code 1: Cuda Runtime (invalid resource handle)

But i can able to run the script from same Flask app by using subprocess and run the inference script

Can you advice how i can make the inference work with the 1st way just directly initializing from the python using Flask

So, the inference is working with the inference script from tao_deploy repository now. Can you leverage it and check the difference why you get failed in your application?

Hi

I tried to check the data is available (while doing the prediction for an image)in the GPU memory , by again copy the device memory to host and check the data is there in the GPU memory , i can see the data is there

But still , when i run the inference by directly initializing the inference class within Flask app, i get below Error from tensorRT

Error:
[TRT] [E] 1: [shapeHostToDeviceRunner.cpp::execute::48] Error Code 1: Cuda Runtime (invalid resource handle)

and get prediction with zero as bbx coordinates and always return 0,5 as the score (some pointless predictions)

Sample prediction portion :

[[ 90 0.5 0 0 0 0]
[ 2 0.5 0 0 0 0]
[ 86 0.5 0 0 0 0]
[ 87 0.5 0 0 0 0]
[ 88 0.5 0 0 0 0]
[ 89 0.5 0 0 0 0]
[ 90 0.5 0 0 0 0]
[ 0 0.5 0 0 0 0]
[ 1 0.5 0 0 0 0]
[ 3 0.5 0 0 0 0]

Can you guide me how can i debug this issue , do you think this is anything to do with Flask app

FYI: when i run the inference script using python subprocess within Flask itself , it works and i get correct prediction without the Error

Appreciate your support on this , let me know if you need any other information

Can you share reproduce steps with details info? Such as command, code, log, etc.
You can run inference successfully from tao_deploy repository, right?
Then, run the inference script using python subprocess within Flask successfully?
But cannot run the inference by directly initializing the inference class within Flask app?
Suggest to share more info for better checking.

Yes the way you understand my question is correct

Please find the code that i have used


"""Standalone TensorRT inference."""

import os
import logging
import numpy as np
from PIL import Image
from tqdm.auto import tqdm
import sys
import cv2

# sys.path.append("../../../../")


current_dir = os.getcwd()
inf_base_dir = os.path.abspath(os.path.join(current_dir, "../../../../"))
sys.path.append(inf_base_dir)

root_dir = os.path.abspath(os.path.join(current_dir, "inference_base/"))
sys.path.append(root_dir)
sys.path.append(os.path.join(os.path.dirname(__file__), "..", "inference_base"))


from nvidia_tao_deploy.cv.deformable_detr.inferencer import DDETRInferencer
from nvidia_tao_deploy.cv.deformable_detr.utils import post_process


from nvidia_tao_deploy.utils.image_batcher import ImageBatcher

from utils.inference import process_dino_pred, load_classes
from utils.datasets import letterbox


logging.basicConfig(
    format="%(asctime)s [TAO Toolkit] [%(levelname)s] %(name)s %(lineno)d: %(message)s",
    level="INFO",
)
logger = logging.getLogger(__name__)
spec_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


class Inference:
    def __init__(
        self,
        trt_model_path,
        img_size,
        device,
        class_path,
        accepted_cls,
        verbose=False,
        classwise_conf=None,
    ):
        if not os.path.exists(trt_model_path):
            raise FileNotFoundError(
                f"Provided inference.trt_engine at {trt_model_path} does not exist!"
            )

        self.trt_infer = DDETRInferencer(trt_model_path, batch_size=1, num_classes=91)
        self.image_size = img_size
        self.accepted_cls = accepted_cls
        self.names = load_classes(class_path)
        self.class_path = class_path

 

    # @torch.no_grad()
    def predict_video(
        self, source, augment, conf, iou_thresh, agnostic_nms, draw=False
    ):
        # bmp_image_paths = [os.path.join(source, file) for file in os.listdir(directory_path) if file.lower().endswith(".bmp")]

        detections = []
        i = 0

        # if self.classwise_conf is None:
        #     conf_thresh = conf
        # else:
        #     conf_thresh = min(self.classwise_conf.values())

        c, h, w = self.trt_infer._input_shape
        batcher = ImageBatcher(
            source,
            (1, c, h, w),
            self.trt_infer.inputs[0].host.dtype,
            preprocessor="DDETR",
        )

        # with open(self.class_path, "r", encoding="utf-8") as f:
        #     classmap = [line.rstrip() for line in f.readlines()]
        # classes = {c: i + 1 for i, c in enumerate(classmap)}
        # inv_classes = {v: k for k, v in classes.items()}

        for batches, img_paths, scales in tqdm(
            batcher.get_batch(), total=batcher.num_batches, desc="Producing predictions"
        ):
            # Handle last batch as we artifically pad images for the last batch idx
            if len(img_paths) != len(batches):
                batches = batches[: len(img_paths)]
            pred_logits, pred_boxes = self.trt_infer.infer(batches)
            target_sizes = []
            for batch, scale in zip(batches, scales):
                _, new_h, new_w = batch.shape
                orig_h, orig_w = int(scale[0] * new_h), int(scale[1] * new_w)
                target_sizes.append([orig_w, orig_h, orig_w, orig_h])

            class_labels, scores, boxes = post_process(
                pred_logits, pred_boxes, target_sizes
            )
            y_pred_valid = np.concatenate(
                [class_labels[..., None], scores[..., None], boxes], axis=-1
            )

            for img_path, pred in zip(img_paths, y_pred_valid):
               

                draw = (None,)
                path = None
                colors = None
                auto_size = 640
                img0 = cv2.imread(img_path)
                img = letterbox(img0, new_shape=self.image_size, auto_size=auto_size)[0]
                frame_detections = process_dino_pred(
                    pred,
                    draw,
                    path,
                    img,
                    img0,
                    self.names,
                    colors,
                    self.accepted_cls,
                )[0]

                # frame_detections = self.filter_detections(
                #     frame_detections, self.classwise_conf
                # )
                detections.append({"boundingPoly": frame_detections, "frame": i})
                i += 1

        import json

        # Save the dictionary as a JSON file
        path_to_detection_file = os.path.abspath(
            os.path.join(current_dir, "yolor/weights/", "detections.json")
        )
        with open(path_to_detection_file, "w") as json_file:
            json.dump(detections, json_file)

        return detections


def main():


    import sys

    
    arguments = sys.argv[1:]

  

 "/workspace/ai_deployment_pipeline/yolor/weights/dino_test_images/exported_images/"
    IMG_SIZE = 640
    TEST_VIDEO_FRAMES_DIR_PATH = arguments[0]
    WEIGTHS_PATH = arguments[1]
    NAMES_PATH = arguments[2]

    ACCEPTED_CLS = [0, 1, 2, 3, 4]

    import time
    start_time = time.time()
    

    inference = Inference(
        trt_model_path=WEIGTHS_PATH,
        img_size=IMG_SIZE,
        device=None,
        class_path=NAMES_PATH,
        accepted_cls=ACCEPTED_CLS,
        verbose=False,
    )

    
    AUGMENT = False
    CONFIDENCE_THRESHOLD = 0.3
    IOU_THRESHOLD = 0.5
    AGNOSTIC_NMS = False

  

    predictions = inference.predict_video(
        TEST_VIDEO_FRAMES_DIR_PATH,
        AUGMENT,
        CONFIDENCE_THRESHOLD,
        IOU_THRESHOLD,
        AGNOSTIC_NMS,
        True,
    )




if __name__ == "__main__":
    main()

Above is the inference class

Below is how i am accessing that inside another script

from inference_base.nvidia_tao_deploy.cv.dino.scripts.dino_inference import Inference

class ServerInference(Inference, CommonInference):

    def __init__(
            self,
            config_ai_parameters,
            isLowFrameVideo=False,
            num_download_tries=3,
        ):
            """
            The constructor for the class from the configuration file.
    
            Parameters
            ----------
            config_file_path : str
                The path to the configuration file.
            """
    
            CommonInference.__init__(
                self, config_ai_parameters, isLowFrameVideo, num_download_tries
            )
    
            (
                self.detector_weights_path,
                self.detector_names_path,
                self.model_info_path,
                self.class_wise_conf_path,
            ) = self.download_model_weights()
            self.isLowFrameVideo = isLowFrameVideo
    
            with open(self.model_info_path, "r") as stream:
                self.model_info = yaml.load(stream, Loader=yaml.FullLoader)
            if self.model_info is None:
                self.model_info = self.config["model_info"]
    
           if torch.cuda.is_available():
            device_id = torch.cuda.current_device()
            device = "cuda:" + str(device_id)
            Inference.__init__(
                self,
                trt_model_path=self.detector_weights_path,
                img_size=self.model_info.get(
                    "image_size", self.detector_config["image_size"]
                ),
                device=device,
                class_path=self.detector_names_path,
                accepted_cls=self.model_info.get(
                    "accepted_classes", self.detector_config["accepted_classes"]
                ),
                verbose=False,
            )
        else:
            print("CUDA is not available. Cannot initialize the Inference class.")


    def predict_video_images(self, request, file_id, timestamps):
            """
            Predict on the exported images using the model
    
            Parameters
            ----------
            request : dict
                The request object. Containing lower_confidence, vehicle_confidence
            file_id : str
                The file id of the video to be predicted
    
            Returns
            -------
                predictions: list
                debug object: dict
            """
            human_confidence = request.get(
                "lower_confidence", self.detector_config["lower_confidence_threshold"]
            )
            vehicle_confidence = request.get(
                "vehicle_confidence", self.detector_config["vehicle_confidence_threshold"]
            )
    
            confidence = min(human_confidence, vehicle_confidence)
            images_path = get_abs_path("..", "temp/{}/exported_images/".format(file_id))
            is_raw_predictions_enabled = request.get("is_raw_predictions_enabled", False)
    
            # predictions on video
        predictions = self.predict_video(
            images_path,
            self.detector_config["augment"],
            confidence,
            self.detector_config["iou_threshold"],
            self.detector_config["agnostic_nms"],
            draw=False,
        )
        print(predictions)
        ############ test prediction with subprocess ##########33
            import subprocess
    
            WEIGTHS_PATH = get_abs_path("../yolor/weights/dino_model_v1.engine")
            NAMES_PATH = get_abs_path("../yolor/weights/class.names")
    
            script_path = get_abs_path(
                "../inference_base/nvidia_tao_deploy/cv/dino/scripts/dino_inference.py"
            )
            command = ["python", script_path, images_path, WEIGTHS_PATH, NAMES_PATH]
            process = subprocess.Popen(
                command, stdout=subprocess.PIPE, stderr=subprocess.PIPE
            )
            stdout, stderr = process.communicate()
            if process.returncode != 0:
                print(f"Error occurred: {stderr.decode('utf-8')}")
            else:
                print(f"Output: {stdout.decode('utf-8')}")
    
            # Load the JSON file back into a variable
            path_to_detection_file = os.path.abspath(
                os.path.join("yolor/weights/", "detections.json")
            )
            with open(path_to_detection_file, "r") as json_file:
                predictions = json.load(json_file)
            #############################################################
    
            if is_raw_predictions_enabled:
                return predictions, None, None
            return predictions

Above scripts contains both the sub process method and the direct initialization method

OK, can you tell me which code can “run the inference script using python subprocess within Flask successfully”?
And which code " cannot run the inference by directly initializing the inference class within Flask app?"

When running below code which is inside the function " def predict_video_images(self, request, file_id, timestamps):
", which is not working , just reterning some zero values as prediction

predictions = self.predict_video(
            images_path,
            self.detector_config["augment"],
            confidence,
            self.detector_config["iou_threshold"],
            self.detector_config["agnostic_nms"],
            draw=False,
        )

When running as below , with sub process , the prediction works

import subprocess
    
            WEIGTHS_PATH = get_abs_path("../yolor/weights/dino_model_v1.engine")
            NAMES_PATH = get_abs_path("../yolor/weights/class.names")
    
            script_path = get_abs_path(
                "../inference_base/nvidia_tao_deploy/cv/dino/scripts/dino_inference.py"
            )
            command = ["python", script_path, images_path, WEIGTHS_PATH, NAMES_PATH]
            process = subprocess.Popen(
                command, stdout=subprocess.PIPE, stderr=subprocess.PIPE
            )
            stdout, stderr = process.communicate()
            if process.returncode != 0:
                print(f"Error occurred: {stderr.decode('utf-8')}")
            else:
                print(f"Output: {stdout.decode('utf-8')}")
    
            # Load the JSON file back into a variable
            path_to_detection_file = os.path.abspath(
                os.path.join("yolor/weights/", "detections.json")
            )
            with open(path_to_detection_file, "r") as json_file:
                predictions = json.load(json_file)

Since it is 3rd-party code and also one can work and another cannot work, could you please try to debug firstly by yourself? Thanks.

Can you advice on where to look for and any debuging guide when we get Error like below, only occurs with running with Flask app, not otherwise
[TRT] [E] 1: [shapeHostToDeviceRunner.cpp::execute::48] Error Code 1: Cuda Runtime (invalid resource handle)

Which is coming when we run the inference on Dino with below line

        cuda.memcpy_htod_async(inp.device, inp.host, stream)
    # Run inference.
    if execute_v2:
        context.execute_async_v2(bindings=bindings, stream_handle=stream.handle)
    else:
        context.execute_async(
            batch_size=batch_size, bindings=bindings, stream_handle=stream.handle
        )
    # Transfer predictions back from the GPU.
    for out in outputs:
        cuda.memcpy_dtoh_async(out.host, out.device, stream)
    # Synchronize the stream
    stream.synchronize()```

above is the extract from the inference code in tao_deploy repository 

I already checked the GPU memory, which is contains the image data when doing inference 

Can you give me some pointers to look into this 
Thanks

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

You can try as below.

import pycuda.driver as cuda0
cuda0.init()

Then in the inference class, adding

self.cfx = cuda0.Device(0).make_context()

Then add self.cfx.push() before inference code and add self.cfx.pop() after inference code.

 self.cfx.push()

 #inference code

 self.cfx.pop()

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.