Inferring resnet18 classification etlt model with python

Okay. I was trying to infer it in my linux system, inside the tlt-container, just like i inferred detectnet_v2 and yolo models for object detection… is there any tutorial or sample code i can refer to understand how to read the resnet18 classification model in python and to get the predicted result out of it?

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
trt_runtime = trt.Runtime(TRT_LOGGER)
with open(engine_path, "rb") as f:
    engine_data = f.read()
    engine = trt_runtime.deserialize_cuda_engine(engine_data)

I am able to read the engine file with this method…

got those sample files at GitHub - NVIDIA/TensorRT: TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

Yes, the same as https://github.com/NVIDIA/TensorRT/tree/master/samples/python/introductory_parser_samples

1 Like

I could run my script and i am able to get the output from the model… But the results are not same as i get using tlt-infer…

so this is my memory allocation function and image normalising function…

def allocate_buffers(engine):
    h_input = cuda.pagelocked_empty(trt.volume(engine.get_binding_shape(0)), dtype=trt.nptype(trt.float32))
    h_output = cuda.pagelocked_empty(trt.volume(engine.get_binding_shape(1)), dtype=trt.nptype(trt.float32))
    # Allocate device memory for inputs and outputs.
    d_input = cuda.mem_alloc(h_input.nbytes)
    d_output = cuda.mem_alloc(h_output.nbytes)
    # Create a stream in which to copy inputs/outputs and run inference.
    stream = cuda.Stream()
    return h_input, d_input, h_output, d_output, stream

def load_normalized_test_case(test_image, pagelocked_buffer):
    # Converts the input image to a CHW Numpy array
    def normalize_image(image):
        # Resize, antialias and transpose the image to CHW.
        c, h, w = 3,120,120
        return np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel()

    # Normalize the image and copy to pagelocked memory.
    np.copyto(pagelocked_buffer, normalize_image(Image.open(test_image)))
    return test_image

is the classification output i am getting is wrong because of something here?

Could you please share your full script?

Yes @Morganh … This is my script…

infer_resnet18_classification.py (4.3 KB)

Hi @Morganh ,
Do i need to do something else in the preprocessing? Currently this is what i am using which was recommended in
TensorRT/samples/python/introductory_parser_samples/caffe_resnet50.py ,

np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel()

should i do something like this which is given in TensorRT/samples/python/introductory_parser_samples/onnx_resnet50.py,

#This particular ResNet50 model requires some preprocessing, specifically, mean normalization.
return (image_arr / 255.0 - 0.45) / 0.225

I have trained the model from resnet18 pretrained model…

Firstly, have you run the python files under TensorRT/samples/python/introductory_parser_samples/ successfully?

For the first time, you can serialize the engine and write to a file. Then after that, when you run again, read the engine from the file and deserialize. It will help you load an existing engine file.
Reference: Developer Guide :: NVIDIA Deep Learning TensorRT Documentation

Hi @Morganh ,
There is no error when i run my script. It loads the engine file and gives the output in expected format…
So i have 2 classes for the classification model to predict, say positive and negative… So what happens is, i have a folder where 31 images are there. When i classify those images using tlt-infer classification, all of them are predicted as positive, which is correct…

but when i classify it in python 25 items are coming to the category negative… and only 6 of them are predicted as positive… that’s why i think there is something need to be done in the pre processing stage (in normalize_image function)…

In onnx_rest50.py they have given something like this,

#This particular ResNet50 model requires some preprocessing, specifically, mean normalization.
return (image_arr / 255.0 - 0.45) / 0.225

like that is there any different pre-processing needed for resnet18 models?

Why i am getting different classification output from what i get with tlt-infer? I have shared the full script above…

Can you run inference with deepstream?
For tlt-infer, it just only run inference against tlt model.

The preprocessing in your script looks ok.

@jazeel.jk hey … I am also trying to run classification model. I tried your script . I load .trt model inside that but I am getting this issue . Do you have any idea what I am doing wrong . my error message is :
[TensorRT] ERROR: coreReadArchive.cpp (38) - Serialization Error in verifyHeader: 0 (Version tag does not match)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
Traceback (most recent call last):
File “infer_resnet18_classification.py”, line 81, in
h_input, d_input, h_output, d_output, stream = allocate_buffers(trt_engine)
File “infer_resnet18_classification.py”, line 35, in allocate_buffers
h_input = cuda.pagelocked_empty(trt.volume(engine.get_binding_shape(0)), dtype=trt.nptype(trt.float32))
AttributeError: ‘NoneType’ object has no attribute ‘get_binding_shape’
any insights will be appreciable.
thanks

Hi @senbhaskar26 ,
Please double check the path you given in the line,
trt_engine_path = os.path.join(“final_model.trt”), if .trt file is not in the same directory you need to give the full path…

I have given absolute path still facing same error. @jazeel.jk
is anything else which I am missing.

@senbhaskar26, have you mounted any volume inside the docker? and are you giving the full path of the file in the system? soemthing like /home/****/tlt-experiments/model.trt ? or are you giving the path as /workspace/tlt-experiments/model.trt ?

Hi @Morganh,
Okay so i have tried the the way you suggested, first serializing the engine and writing to a file… then reading the engine from the file and deserializing it… it gives the same outputs as i received in my script…
25 negative classes and 6 positive classes… The ground truth is that all of them are positive… and that’s what i am getting from tlt-infer… so the model was trained for an input shape of (3,120,120). so i am resizing the testing image in the preprocessing stage to same resolution… ideally it should give the same outputs like i received from tlt-infer right? If everything at the preprocessing stage is correct? is there anything i am missing ?

hey thanks got the issue . I have trained the model using tlt docker but running predictions on Jetson-nano . that’s why causing the issue. I converted the model first into engine on nano . I am able to load the model .
now the issue is in images.
File “infer_resnet18_classification.py”, line 49, in normalize_image
return np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel()
ValueError: axes don’t match array
Any idea ? @jazeel.jk

1 Like

now the issue is in images.
File “infer_resnet18_classification.py”, line 49, in normalize_image
return np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel()
ValueError: axes don’t match array
Any idea ? @jazeel.jk

@senbhaskar26 what is the input shape of your image? is it a colour image or greyscale image?
i am not sure about the preprocessing part… i am able to load the image and predict without any error… but the output model predicts, is not right… but the same model gives correct output with tlt-infer… that’s what i was discussing with @Morganh .

in your case were you using resnet18 pretrained model for training ? there could be different preprocessing step for different models…

I have used resnet18 model as well. I have changed input shape of the image as it is trained on (3,224,224) . not sure what is happening with the pre-processing . @jazeel.jk

@jazeel.jk
How did you generate the etlt file and trt engine?
With fp16 or int8?