# Same memory usage for fp16 and int8

**URL:** <https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338>\
**Category:** Jetson Xavier NX\
**Tags:** tensorrt\
**Created:** [July 11, 2021, 3:23pm UTC](https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338 "2021-07-11T15:23:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tigran.khachatryann1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tigran.khachatryann1](https://forums.developer.nvidia.com/u/tigran.khachatryann1)\
**Post date:** [July 11, 2021, 3:23pm UTC](https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338/1 "2021-07-11T15:23:18Z")

</div>

Hello,

I wanted to benchmark depth estimation model on Jetson Xavier NX in terms of speed and memory usage. For that purpose I have converted pytorch model to ONNX format and than I have created TensorRT engines with fp32, fp16 and int8 precisions. In case of speed(FPS) everything seems to be correct, fp16 model is faster than fp32 and int8 model is the fastest.  
The memory usage is around 1.9 Gb in case of fp32 and around 1.1 Gb in case of fp16 and int8. I guess the difference between fp32 and fp16 memory usages is reasonable, but I can not understand why it is similar for fp16 and int8 engines.  
Could someone explain is this behavior correct?  
Could you please advise how can I profile memory usage ? (My application is written in Python)  
Is there any method to calculate FLOPs or TOPs for TensorRT engine?

Thanks,  
Tigran

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 12, 2021, 4:10am UTC](https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338/3 "2021-07-12T04:10:50Z")

</div>

Hi,

The memory usage depends on the inference algorithm used by TensorRT.  
It doesn’t guarantee that lower precision will use less memory.

But you can specify the preferred memory amount when creating the TensorRT engine.  
For example:

```auto
$ /usr/src/tensorrt/bin/trtexec --workspace=16 ...

```

To get FLOPs information, you can use _nvprof_ with `--metrics` flag.  
For example:

```auto
$ sudo /usr/local/cuda-10.2/bin/nvprof --metrics flops_sp /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx

```

```auto
==30550== Profiling result:
==30550== Metric result:
Invocations Metric Name Metric Description Min Max Avg
Device "Xavier (0)"
    Kernel: void fused::fusedConvolutionReluKernel<fused::SrcChwcPtr_FltTex_Reader<float, int=1, int=1, int=1, int=1>, fused::KpqkPtrWriter<float, int=1, int=1, int=1>, float, float, int=3, int=4, int=1, int=5, int=5, int=1, int=1>(fused::ConvolutionParams<floatSrcType, int=1, int=1Type>)
         80 flop_count_sp Floating Point Operations(Single Precision) 1350272 3075136 2212704
    Kernel: void fused::fusedConvolutionReluKernel<fused::SrcChwcPtr_FltTex_Reader<float, int=1, int=1, int=1, int=2>, fused::KpqkPtrWriter<float, int=1, int=1, int=2>, float, float, int=2, int=4, int=1, int=5, int=5, int=1, int=1>(fused::ConvolutionParams<floatSrcType, int=1, int=1Type>)
         80 flop_count_sp Floating Point Operations(Single Precision) 2515072 5737536 4126304
    Kernel: void nvinfer1::tiled_pooling::poolCHW_PQT<int=2, int=2, int=2, int=2, int=2, int=16, int=128, int=1, int=1, bool=1, nvinfer1::ITiledPooling::PoolingMode, bool=0>(nvinfer1::TiledPoolingParams, int)
...

```

Thanks.

---

<div class="post-metadata">

**Author:** ![tigran.khachatryann1](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tigran.khachatryann1](https://forums.developer.nvidia.com/u/tigran.khachatryann1)\
**Post date:** [July 13, 2021, 10:27pm UTC](https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338/4 "2021-07-13T22:27:23Z")

</div>

Hi,

Thank you for explanation @AastaLLL .  
One more question please, how can I get the amount of int8 operations?  
I was able to measure the amount of fp32 and fp16 operations using flops\_sp and flop\_count\_hp metrics respectively, but I can not find any metric for int8 operations.

Thanks.

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 22, 2021, 8:16am UTC](https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338/5 "2021-07-22T08:16:58Z")

</div>

Hi,

You can use `tensor_int_fu_utilization` mentioned in the below document:

> **[Profiler :: CUDA Toolkit Documentation](https://docs.nvidia.com/cuda/archive/11.0_GA/profiler-users-guide/index.html#metrics-reference-7x)**
>
> The user manual for NVIDIA profiling tools for optimizing performance of CUDA applications.

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [September 27, 2021, 1:06pm UTC](https://forums.developer.nvidia.com/t/same-memory-usage-for-fp16-and-int8/183338/7 "2021-09-27T13:06:42Z")

</div>

Closed on September 27, 2021, 1:06pm UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.