We are having issues with high memory consumption on Jetson Xavier NX especially when using TensorRT via ONNX RT.
By default our NN models are in FP32, so we tried converting to FP16 which makes the NN model smaller. However, during the model inference the memory consumption is the same as with FP32.
I did enable FP16 inference using ORT_TENSORRT_FP16_ENABLE=1 as suggested by ONNX Runtime documentation (TensorRT - onnxruntime). But it didn’t help.
Does Jetson Xavier NX support both FP16 and FP32, especially for CUDA and TensorRT?
Is there any other way to reduce the memory consumption when using CUDA and TensorRT?
We also found out that with any NN model the process using ONNX RT with CUDA execution provider utilizes at least 1.5GB of RAM and with TensorRT execution provider at least 2 GB of RAM. Is this expected?
Is there maybe any light version of the libraries to be used on a device with limited RAM like Jetson?
To inference with cuDNN (TensorRT), it requires at least 600 MB of memory for loading the library.
If you run it with ONNX Runtime, you need more memory to load the all required library.
Using trtexec reduces the memory consumption by 150-200 MB comparing to ONNX Runtime with TRT. So this does not seem to be the issue.
In case of some NN models the RAM consumption is much lower when the models is loaded from existing engine file. Is it expected the model optimization (engine file creation) requires much more memory in comparison to inference only?
We have observed the FP16 inference is slower comparing to FP32. Is it possible? Our expectation is the performance would be at least the same, maybe better, but not worse.
And the last issue is the TRT optimization of FP16 model takes extremely long comparing to FP32. How is it possible?
My understanding is Jetson Xavier NX supports FP16 inference.
First, have you maximized the device performance before benchmarking?
$ sudo nvpmodel -m 0
$ sudo jetson_clocks
Since TensorRT loads the corresponding library when required.
It’s possible that the memory consumption with or without conversion will be different.
In general, FP16 should be faster than FP32, but it might take a longer time for conversion.
Do you measure the inference time only or all the execution time (including conversion and benchmark)?
XavierNX (sm=7.2) supports FP32, FP16, and INT8 inference.
You can find the support matrix below:
Yes, we have maximized the performance of the device.
We tested two approaches:
The first is original FP 32 NN model but FP16 enabled for TensorRT.
MEMORY: We did not observe any significant change in memory consumption when the caching was disabled. Only with the TRT engine file caching enabled we were able to save up to 700 MB of memory.
INFERENCE PERF: For some NN models the inference performance has improved 2 times.
NN MODEL LOAD: However, the load/compilation of the NN model is 4-5 times slower.
Second approach was converting the ONNX model to FP16 using a script provided by ONNX (Microsoft) and testing both FP32 and FP16 inference.
MEMORY: There is no difference in memory usage when using FP 16 model.
INFERENCE PERF: The inference performance has decreased significantly with FP16 model.
NN MODEL LOAD: The FP16 ONNX model load/compilation is much slower comparing to original FP32 model.
Usually, we recommend users to use approach 1.
Since the conversion (model → engine) is a one-time job.
You can deserialize the engine file at runtime to save memory and get performance.
For the ONNX integrated version, have you checked this issue with the ONNX team?
If not, would you mind doing so? They may have more information about the ONNX-based implementation.
