I have converted this saved_model to TensorRT model by using tf-trt converter but the model size after conversion was around 800MB which is uncommon and while inferecing on the Xavier NX it takes more than 20GB RAM, almost 1 hour to load the model but even after i was getting only 3 FPS.
I have tried also with ONNX runtime but the result is same.
but the model loading memory is reduced to 4GB with few solutions,
however the FPS remains the same.
In my laptop with 1650max-Q GPU it gives around 13FPS.
Im using Python with Tensorflow implementation.
we have sucessfully converted the model to ONNX format and inferenced with ONNX runtime in the xavier NX but there is not much improvement in the FPS (3-4FPS)
I have tried to convert the ONNX to Nvidia tensorRT but getting the below error
XavierNX is an embedded device.
It’s expected that 1650 will have a better performance compared to Jetson.
You can try to inference it with fp16 or int8 for extra acceleration.
The unsupported data type error is a known issue.
It is caused by the different default input data type between ONNX and TensorRT.
Please check below comment for the solution: