Unable to run Inference on Tensorflow Hub models that do not support batching with Triton InferServer

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.0
• TensorRT Version 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only) 440.33.01

**• Steps to reproduce the issue **

  • Create a container from the image : ‘nvcr.io/nvidia/deepstream:6.0-triton’ using the below command sudo docker run -it --gpus all -p 8554:8554 -w /opt/nvidia/deepstream/deepstream-6.0
  • Unzip the attached folder in the current working directory (deepstream-6.0).
  • Go to efficientdet folder.
  • Run the following command python3 main.py file:///opt/nvidia/deepstream/deepstream-6.0/efficientdet/data/sample_720p.h264
  • The screenshot of the error is attached.
  • Note : We are facing this problem with tensorflow models that do not support batching, whereas with models that support dynamic batching this problem disappears.
    efficientdet.zip (45.2 MB)

Error
Error1920×817 203 KB

4

Any update regarding this issue?

5

Sorry for the late response, we will do the investigation to have the update soon. Thanks

6

Ok. Awaiting for your response!

8

Please refer to NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream (github.com)

9

Thanks for the update.
But we want to deploy any TensorFlow 2 hub Model ( Not only efficientdet ) using triton-inferserver ( nvinferserver ). We observed Models that don’t support batching cannot be deployed with triton-inference.
We would appreciate any help in this regard. Thanks!

11

Any updates ?