I have an .etlt model and a corresponding .trt classification model which i trained with nvidia tlt, from resnet18 pretrained model. How can i infer it in python? Any suggestion would be appreciated… Thank you.
caffe_resnet50.py onnx_resnet50.py README.md requirements.txt uff_resnet50.py
Thank you for your reply. Where i can find these files ? Is this files part of some repo or available in some docker container?
When you install TensorRT with Jetpack or other way, the files are available at your device.
For example, in Jetson device,
Okay. I was trying to infer it in my linux system, inside the tlt-container, just like i inferred detectnet_v2 and yolo models for object detection… is there any tutorial or sample code i can refer to understand how to read the resnet18 classification model in python and to get the predicted result out of it?
TRT_LOGGER = trt.Logger(trt.Logger.WARNING) trt_runtime = trt.Runtime(TRT_LOGGER) with open(engine_path, "rb") as f: engine_data = f.read() engine = trt_runtime.deserialize_cuda_engine(engine_data)
I am able to read the engine file with this method…
I could run my script and i am able to get the output from the model… But the results are not same as i get using tlt-infer…
so this is my memory allocation function and image normalising function…
def allocate_buffers(engine): h_input = cuda.pagelocked_empty(trt.volume(engine.get_binding_shape(0)), dtype=trt.nptype(trt.float32)) h_output = cuda.pagelocked_empty(trt.volume(engine.get_binding_shape(1)), dtype=trt.nptype(trt.float32)) # Allocate device memory for inputs and outputs. d_input = cuda.mem_alloc(h_input.nbytes) d_output = cuda.mem_alloc(h_output.nbytes) # Create a stream in which to copy inputs/outputs and run inference. stream = cuda.Stream() return h_input, d_input, h_output, d_output, stream def load_normalized_test_case(test_image, pagelocked_buffer): # Converts the input image to a CHW Numpy array def normalize_image(image): # Resize, antialias and transpose the image to CHW. c, h, w = 3,120,120 return np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel() # Normalize the image and copy to pagelocked memory. np.copyto(pagelocked_buffer, normalize_image(Image.open(test_image))) return test_image
is the classification output i am getting is wrong because of something here?
Could you please share your full script?
Hi @Morganh ,
Do i need to do something else in the preprocessing? Currently this is what i am using which was recommended in
np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel()
should i do something like this which is given in TensorRT/samples/python/introductory_parser_samples/onnx_resnet50.py,
#This particular ResNet50 model requires some preprocessing, specifically, mean normalization. return (image_arr / 255.0 - 0.45) / 0.225
I have trained the model from resnet18 pretrained model…
Firstly, have you run the python files under TensorRT/samples/python/introductory_parser_samples/ successfully?
For the first time, you can serialize the engine and write to a file. Then after that, when you run again, read the engine from the file and deserialize. It will help you load an existing engine file.
Reference: Developer Guide :: NVIDIA Deep Learning TensorRT Documentation
Hi @Morganh ,
There is no error when i run my script. It loads the engine file and gives the output in expected format…
So i have 2 classes for the classification model to predict, say positive and negative… So what happens is, i have a folder where 31 images are there. When i classify those images using tlt-infer classification, all of them are predicted as positive, which is correct…
but when i classify it in python 25 items are coming to the category negative… and only 6 of them are predicted as positive… that’s why i think there is something need to be done in the pre processing stage (in normalize_image function)…
In onnx_rest50.py they have given something like this,
#This particular ResNet50 model requires some preprocessing, specifically, mean normalization.
return (image_arr / 255.0 - 0.45) / 0.225
like that is there any different pre-processing needed for resnet18 models?
Why i am getting different classification output from what i get with tlt-infer? I have shared the full script above…
Can you run inference with deepstream?
For tlt-infer, it just only run inference against tlt model.
The preprocessing in your script looks ok.
@jazeel.jk hey … I am also trying to run classification model. I tried your script . I load .trt model inside that but I am getting this issue . Do you have any idea what I am doing wrong . my error message is :
[TensorRT] ERROR: coreReadArchive.cpp (38) - Serialization Error in verifyHeader: 0 (Version tag does not match)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
Traceback (most recent call last):
File “infer_resnet18_classification.py”, line 81, in
h_input, d_input, h_output, d_output, stream = allocate_buffers(trt_engine)
File “infer_resnet18_classification.py”, line 35, in allocate_buffers
h_input = cuda.pagelocked_empty(trt.volume(engine.get_binding_shape(0)), dtype=trt.nptype(trt.float32))
AttributeError: ‘NoneType’ object has no attribute ‘get_binding_shape’
any insights will be appreciable.
Hi @senbhaskar26 ,
Please double check the path you given in the line,
trt_engine_path = os.path.join(“final_model.trt”), if .trt file is not in the same directory you need to give the full path…
I have given absolute path still facing same error. @jazeel.jk
is anything else which I am missing.
@senbhaskar26, have you mounted any volume inside the docker? and are you giving the full path of the file in the system? soemthing like /home/****/tlt-experiments/model.trt ? or are you giving the path as /workspace/tlt-experiments/model.trt ?
Okay so i have tried the the way you suggested, first serializing the engine and writing to a file… then reading the engine from the file and deserializing it… it gives the same outputs as i received in my script…
25 negative classes and 6 positive classes… The ground truth is that all of them are positive… and that’s what i am getting from tlt-infer… so the model was trained for an input shape of (3,120,120). so i am resizing the testing image in the preprocessing stage to same resolution… ideally it should give the same outputs like i received from tlt-infer right? If everything at the preprocessing stage is correct? is there anything i am missing ?
hey thanks got the issue . I have trained the model using tlt docker but running predictions on Jetson-nano . that’s why causing the issue. I converted the model first into engine on nano . I am able to load the model .
now the issue is in images.
File “infer_resnet18_classification.py”, line 49, in normalize_image
return np.asarray(image.resize((w, h), Image.ANTIALIAS)).transpose([2, 0, 1]).astype(trt.nptype(trt.float32)).ravel()
ValueError: axes don’t match array
Any idea ? @jazeel.jk