Subnormal FP16 values detected

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

A clear and concise description of the bug or issue.

Environment

TensorRT Version: 8.4.1.5
GPU Type: discrete
Nvidia Driver Version: 460.73.01
CUDA Version: 11.2
CUDNN Version: 8.2
Operating System + Version: ubuntu 20.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag): nvidia-tensorrt==8.4.1.5

When converting to tensorrt FP16 I see this:

Weights [name=Conv_0 + Relu_1.weight] had the following issues when converted to FP16:
[07/07/2022-18:30:26] [TRT] [W]  - Subnormal FP16 values detected. 
[07/07/2022-18:30:26] [TRT] [W]  - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value. 
[07/07/2022-18:30:26] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
[07/07/2022-18:30:26] [TRT] [W] Weights [name=Conv_0 + Relu_1.bias] had the following issues when converted to FP16:
[07/07/2022-18:30:26] [TRT] [W]  - Subnormal FP16 values detected. 
[07/07/2022-18:30:26] [TRT] [W]  - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value. 
[07/07/2022-18:30:26] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
[07/07/2022-18:30:26] [TRT] [V] Conv_0 + Relu_1 Set Tactic Name: trt_turing_cutlass_image_network_first_layer_hmma_fprop_f16f16f32_nhwc_nhwc_k64r7s7c4_stride2x2 Tactic: 0xe2222883a6602489
[07/07/2022-18:30:26] [TRT] [W] Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
[07/07/2022-18:30:26] [TRT] [W]  - Subnormal FP16 values detected. 
[07/07/2022-18:30:26] [TRT] [W]  - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value. 
[07/07/2022-18:30:26] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
[07/07/2022-18:30:26] [TRT] [V] Conv_3 + Relu_4 Set Tactic Name: turing_h1688cudnn_256x64_ldg8_relu_exp_small_nhwc_tn_v1 Tactic: 0x1dcf9babce3d9b3b
[07/07/2022-18:30:26] [TRT] [W] Weights [name=Conv_5 + Add_6 + Relu_7.weight] had the following issues when converted to FP16:
[07/07/2022-18:30:26] [TRT] [W]  - Subnormal FP16 values detected.

Weights are out of range ?
I think this causes FP16 to underperform. Is there anyway to resolve this either by retraining or modifyying weights?
Complete logs if that helps.
logs.txt (2.4 MB)

Thanks

2 Likes