HELP running tensorflow2 model

[user75788]

I have trained a tensorflow model and saved it using the savedmodel format, (I have a folder and saved_model.pb is in it)

Now how do i convert it to tf-trt and run it?

I have tried converting it to ONNX format and from ONNX to .trt (using TensorRT/create_onnx.py at main · NVIDIA/TensorRT · GitHub) now im completely lost on how to use these models to classify using my webcam.

[AastaLLL]

Hi,

Do you want to convert it into TF-TRT or pure TensorRT.

For TF-TRT, you don’t need to use the ONNX format.
The conversion can just be done via TensorFlow API.

Below is a tutorial for your reference:
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#worflow-with-savedmodel

Thanks.

[user75788]

Hey AastaLLL,

I have created a tensorflow model, converted it into onnx format, and used trtexec to generate a .trt file. (I followed along with this notebook: tensorflow2 through onnix)

I’m able to run the .trt file using images loaded in numpy format as shown in the notebook.

I need help regarding using cuda images (<class ‘jetson.utils.cudaImage’>).

please help how to use the cuda image class and .trt file for inference

Edit:
I’m providing more detatils:
So i have trained a resnet50 model with input shape (1,224,224,3) and output shape (1,1) so it just predicts the probability of 1 class.

I’ve done the conversion:
TF (saved model) → ONNX → .trt

I was able to run inference using the .trt file using images loaded with numpy.
Now i wish to use this .trt model to classify based on webcam video at /dev/video0.
I wanted to use the jetson.utils module to read images from webcam and classify but im getting a cuda image and I don’t know how to use cuda image with .trt

so any help regarding this is appreciated. Thank you

[AastaLLL]

Hi,

...
cuda.memcpy_htod_async(cuda_inputs[0], host_inputs[0], stream)
context.execute_async(bindings=bindings, stream_handle=stream.handle)
cuda.memcpy_dtoh_async(host_outputs[0], cuda_outputs[0], stream)
...

In the sample above, host_inputs[0] is the CPU buffer and the cuda_inputs[0] is the corresponding GPU buffer.
So to run TensorRT with jetson-utils, you can just copy the CUDA image into the cuda_inputs[0].

Ex.

...
jetson.utils.cudaMemcpy(cuda_inputs[0], input_img)   # dst, src
context.execute_async(bindings=bindings, stream_handle=stream.handle)
cuda.memcpy_dtoh_async(host_outputs[0], cuda_outputs[0], stream)
...

Thanks.