GPU difference inference

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hello,

Can the tlt models generated on GPU 1080 be inferred on Triton server with GPU 3070, 3080?

2

Yes, but please note that it is .etlt model. You can run tao export and then copy the .etlt model to Trition server.

3

Thank you!

4

Hi,

How to run a detectnet_v2 model on the triton server, it is a little confusing?

5

For running detectnet_v2 model in triton server, you can refer to Integrating TAO CV Models with Triton Inference Server — TAO Toolkit 3.22.05 documentation.

6

I saw this documentation earlier, but it is not clear…
Is there another client that I have to install for Detectnet_v2?

7

After you have trained a detectnet_v2 model, you can deploy the model to run inference.
For inference, usually there are below ways.

  1. Directly run “tao detectnet_v2 infernece xxx”
  2. Run it with deepstream , refer to https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/detectnet_v2.html#deploying-to-deepstream
  3. Run with triton server. You can run it refer to the NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton (github.com) . Peoplenet or dashcamnet is actually based on detectnet_v2 network.
  4. You can also run with your standalone inference code. For example, Run PeopleNet with tensorrt - #21 by carlos.alvarez
8

How about the post processing, is it a file or a folder?
If it is a file then what is the extension?

The DetectNet_v2 inference sample has 2 components that can be configured

  1. [Model Repository]
  2. [Post Processor]
9

For running in triton server, please refer to the README.
For example, running official released peoplenet model with it, see its postprocessing config file in tao-toolkit-triton-apps/clustering_config_peoplenet.prototxt at main · NVIDIA-AI-IOT/tao-toolkit-triton-apps (github.com)

10

Can I run classification with a standalone inference code?

11

Yes, it can. You can leverage GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton
or search/find some topics in forum. For example,

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.