Description
When I used the TF-TRT tool to deploy the tensorflow model, I found that when deployed as FP32 precision, FP16 precision, and INT8 precision, the resulting file sizes varied greatly and were all much larger than the original tensorflow files. INT8 precision has the largest deployment file size. In theory, the deployment method with the lowest precision saves the smallest number of weight digits, and the file size should be reduced. I would like to ask a few questions.
- Why is the deployment file much larger than the original tensorflow model file?
- Why does the lower the deployment accuracy, the larger the file size?
Environment
TensorRT Version: 6.0.1.5
GPU Type: Titan Xp
Nvidia Driver Version: 440.36
CUDA Version: 10.1
CUDNN Version: 7.6.5.32
Operating System + Version: Ubuntu18.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 2.3.0
Relevant Files
tensorflow model: 45M
FP32-deploymodel: 87M
FP16-deploymodel: 87M
INT8-deploymodel: 121M
Hi,
Looks like you’re using a very old version of the TensorRT, we recommend you to please try on the latest TensorRT version and let us know if you still face this issue.
Thank you.
I installed jetpack5.0DP on jetson agx xavier, its environment is:
TensorRT Version : 8.4.0.11
CUDA Version : 11.4.4
CUDNN Version : 8.3.2
Operating System + Version : Ubuntu20.04
Python Version (if applicable) : 3.8.10
TensorFlow Version (if applicable) : 2.8.0
After doing the same TF-TRT conversion experiment, the obtained files still have the above two problems. The specific file size after deployment is:
tensorflow model: 45M
FP32-deploymodel: 87M
FP16-deploymodel: 87M
INT8-deploymodel: 99M
Could you please tell me why this happens?
Hi,
We will get back to you on your queries, could you please share with us the issue repro script/model for better debugging here or via DM.
Thank you.
Thank you for your reply. The tensorflow model file and the deployed model file of the resnet18 classification model I used are in the following files.
tf_model.zip (39.7 MB)
tf_model_FP32.zip (79.2 MB)
tf_model_FP16.zip (79.2 MB)
tf_model_INT8.zip (89.1 MB)
The tf-trt based deployment code and demo image files I used are in the following files：
tf-trt.zip (21.3 KB)
Hi,
Currently, there would be both TF and TRT portions of the network included. That’s why its expected to model size larger.
Even if INT precision is enabled, there is no guarantee that it will be used (TRT is allowed to use Fp16/32 if that is faster). So having the same engine size in all the cases is possible. INT models also save the calibration table, which could further increase the size. We are verifying to confirm.
Thank you.
Thank you very much for your reply. I have a general understanding of the reason. If there is a relevant detailed description published, please inform me.
@648976749,
Thank you so much for bringing this issue to our attention.
Actually, the calibration table is lightweight, There is some other reason for TF-TRT that results in a large saved model size.
We are tracking this issue internally and also created an issue in Tensorflow GIT - Variables saved in converted model · Issue #305 · tensorflow/tensorrt · GitHub.
Note that for the FP32 and FP16 conversion, the model was not built (which means the TRT engines were not saved to disk, they are created on the fly). If we call converter.build(input_fn) (where input_fn can be the same function that was used for calibration) before convert.save() then we shall see FP32 model size >= FP16 model size >= INT8 model size
Thank you.
I’m facing a similar issue when generating the engine files with the tensorRT utility on a Jetson AGX Orin device (using ONNX files as input), where the INT8 engines have a bigger size compared with those generated in FP32 for resnet-based models.
Sizes obtained for .engine files:
LW-ResNet-FP32: 1.69 MB
LW-ResNet-BF16: 1.65 MB
LW-ResNet-FP16: 1.60 MB
LW-ResNet-INT8: 2.06 MB
However, when doing the same process for transformer encoder-based models, the results are different and INT8 files are smaller than FP32 (as it should be expected):
LW-Transformer-FP32: 1.57 MB
LW-Transformer-BF16: 1.50 MB
LW-Transformer-FP16: 0.89 MB
LW-Transformer-INT8: 0.89 MB
The environment is (versions):
Jetpack: 6.2
TensorRT: 10.3.0.30
CUDA: 12.6
cuDNN: 9.3.0.75
NVIDIA driver: 540.4.0
Hi @6708893100, thanks for sharing the numbers!
This thread is from 2022 and the OP was using TF-TRT (TensorFlow + TensorRT integration) where the saved file contains both the TF graph and the TRT subgraphs, which is what drove the size deltas @spolisetty explained back then. Your stack (TRT 10.3 on JetPack 6.2 building straight from ONNX) is a completely different code path, so the same mechanism does not apply.
A quick orientation on what you’re seeing in pure-TRT engines:
-
Engine size is dominated by the actual weights TRT chose to keep, plus per-tensor scaling metadata, plus any tactic-specific reformat/scale tensors. INT8 saves bytes per weight but adds Q/DQ scale tensors and may keep some layers in FP16/FP32 if those tactics are faster. On a small model like LW-ResNet (~1.6 MB), the metadata overhead can outweigh the weight savings. On a larger transformer where the weight matrices dominate, INT8 wins back a real fraction. That matches what you’re seeing.
-
To see exactly which layers stayed FP16 vs ran INT8, build with --exportLayerInfo=layers.json --profilingVerbosity=detailed and inspect the per-layer precisions. That tells you whether your INT8 engine is actually mostly INT8 or whether TRT fell back to FP16/FP32 for the ResNet branches.
Could you open a new topic in the TensorRT category with:
-
Your exact trtexec build commands for FP32 / FP16 / INT8
-
The full verbose log for one of the runs (--verbose --profilingVerbosity=detailed)
-
The layers.json from --exportLayerInfo
-
Optionally the ONNX (or a representative one)
Link this thread for context, that way the older 2022 TF-TRT answer doesn’t get conflated with your ONNX-on-Orin case.
Closing this one as it’s old and not directly relevant to the new question.
Thanks!