Wrong output after converting model from .etlt to model.plan

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

• Hardware (RTX3050 and T4)
• Network Type (Classification)

I have trained a resnet 18 model. The model is working fine in etlt format with deepstream. Below is the config file for deepstream:

[property]
gpu-id=0
# preprocessing parameters: These are the same for all classification models generated by TLT.
net-scale-factor=1.0
model-color-format=1
batch-size=8

# Model specific paths. These need to be updated for every classification model.
model-engine-file=catdog_v3.0_resnet18_acc95_fp32.etlt_b8_gpu0_fp32.engine
tlt-encoded-model=catdog_v3.0_resnet18_acc95_fp32.etlt
tlt-model-key=model_key
labelfile-path=labels_CatDog.txt
input-dims=3;224;224;0
uff-input-blob-name=input_1
#output node name for classification
output-blob-names=predictions/Softmax

## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
process-mode=2
interval=2
# defines that the model is a classifier.
network-type=1
gie-unique-id=1
classifier-threshold=0.6
classifier-async-mode=1

When I converted the model from etlt to model.plan via tao-converter to deploy on triton. The model output is wrong. Below is the sample run script:

from itertools import tee
from tritonclient.utils import *
import tritonclient.http as httpclient
import sys
import numpy as np
import cv2
model_name = "CatDog"
shape =  [1,3,224,224]
path = "CuteCat.jpg"

with httpclient.InferenceServerClient("localhost:8000") as client:
    input_file = cv2.imread(path)
    input_file = cv2.resize(input_file, (224,224))
    input_file = input_file.astype(np.float32)
    # input_file/=255 # I have tried normalizing also.
    
    input0_data = np.reshape(input_file, shape)    
    
    inputs = [
        httpclient.InferInput("input_1", input0_data.shape,
                              np_to_triton_dtype(input0_data.dtype)),
    ]

    inputs[0].set_data_from_numpy(input0_data)

    outputs = [
        httpclient.InferRequestedOutput("predictions/Softmax"),
    ]

    response = client.infer(model_name,
                            inputs,
                            request_id=str(1),
                            outputs=outputs)

    result = response.get_response()
    output0_data = response.as_numpy("predictions/Softmax")
    print("output0_data : {}".format(output0_data))
    print('PASS: CatDog')
    sys.exit(0)

And this is the model config in triton:

name: "CatDog"
platform: "tensorrt_plan"
max_batch_size: 16
input [
  {
    name: "input_1"
    data_type: TYPE_FP32
    format: FORMAT_NCHW
    dims: [ 3, 224, 224 ]
  }
]
output [
  {
    name: "predictions/Softmax"
    data_type: TYPE_FP32
    dims: [ 2, 1, 1 ]
  }
]

Please, verify if above script is correct…

And please let me know what could be wrong with the models?

3

Please use official tao-triton-app GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton . Firstly, you can run the default setting. Then replace your .etlt model with existing one.
Then generate your own model.plan .
https://github.com/NVIDIA-AI-IOT/tao-toolkit-triton-apps/blob/main/scripts/download_and_convert.sh#L30

4

The problem was with my script. I was doing mistake in preprocessing. Here is the correct script:

from itertools import tee
from tritonclient.utils import *
import tritonclient.http as httpclient
import sys
import numpy as np
import cv2
from logzero import logger
model_name = "CatDog"
shape =  [1,3,224,224]

path = "CuteCat.jpg" # Cat

def preprocess(image, w, h, dtype):
    image = cv2.resize(image,(w, h))#, Image.ANTIALIAS)
    nparray = np.asarray(image).astype(dtype)
    nparray = np.transpose(nparray, (2, 0, 1))
    return nparray

with httpclient.InferenceServerClient("localhost:8000") as client:
    input_file = cv2.imread(path)
    input0_data = preprocess(input_file, 224, 224, np.float32)
    input0_data = np.stack([input0_data], axis=0)
 
    inputs = [
        httpclient.InferInput("input_1", input0_data.shape,
                              np_to_triton_dtype(input0_data.dtype)),
    ]

    inputs[0].set_data_from_numpy(input0_data)

    outputs = [
        httpclient.InferRequestedOutput("predictions/Softmax"),
    ]

    response = client.infer(model_name,
                            inputs,
                            request_id=str(1),
                            outputs=outputs)

    result = response.get_response()
    output0_data = response.as_numpy("predictions/Softmax")

    print("output0_data : {}".format(output0_data))

    print('PASS: CatDog')
    sys.exit(0)
1 Like
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.