Negative values encountered in unsigned quantization

sjtu_llq · September 28, 2021, 2:39am

Description

I use the Trt8.0 quantization function in my model. I just emplyed its PTQ.
The quantization setting are shown as follows:
#######################################################################################
quant_desc_input = QuantDescriptor(calib_method=‘histogram’, unsigned = True)
quant_desc_weight = QuantDescriptor(unsigned = True)
quant_nn.QuantConv2d.set_default_quant_desc_input(quant_desc_input)
quant_nn.QuantConv2d.set_default_quant_desc_weight(quant_desc_weight)
#######################################################################################
After calibration, I got the corresponding amax.
Then I wanted to test the performance, but I got the error like this:
#######################################################################################
File “/usr/lib/python3.7/runpy.py”, line 193, in _run_module_as_main
" main “, mod_spec)
File “/usr/lib/python3.7/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/home/tiger/.vscode-server/extensions/ms-python.python-2021.9.1246542782/pythonFiles/lib/python/debugpy/ main .py”, line 45, in
cli.main()
File “/home/tiger/.vscode-server/extensions/ms-python.python-2021.9.1246542782/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py”, line 444, in main
run()
File “/home/tiger/.vscode-server/extensions/ms-python.python-2021.9.1246542782/pythonFiles/lib/python/debugpy/…/debugpy/server/cli.py”, line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str(” main "))
File “/usr/lib/python3.7/runpy.py”, line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File “/usr/lib/python3.7/runpy.py”, line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File “/usr/lib/python3.7/runpy.py”, line 85, in run_code
exec(code, run_globals)
File “/home/tiger/llq/BasicSR-master/4_quantize_test_frame_rgb.py”, line 111, in
output = model(img_LR).data.squeeze().float().cpu().clamp (0, 1).numpy()
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “/home/tiger/llq/BasicSR-master/4_quantize_test_frame_rgb.py”, line 60, in forward
fea = self.conv_first(x)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/pytorch_quantization-2.1.0-py3.7-linux-x86_64.egg/pytorch_quantization/nn/modules/quant_conv.py”, line 119, in forward
quant_input, quant_weight = self._quant(input)
File “/usr/local/lib/python3.7/dist-packages/pytorch_quantization-2.1.0-py3.7-linux-x86_64.egg/pytorch_quantization/nn/modules/quant_conv.py”, line 85, in _quant
quant_weight = self._weight_quantizer(self.weight)
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 1051, in _call_impl
return forward_call(*input, **kwargs)
File “/usr/local/lib/python3.7/dist-packages/pytorch_quantization-2.1.0-py3.7-linux-x86_64.egg/pytorch_quantization/nn/modules/tensor_quantizer.py”, line 345, in forward
outputs = self._quant_forward(inputs)
File “/usr/local/lib/python3.7/dist-packages/pytorch_quantization-2.1.0-py3.7-linux-x86_64.egg/pytorch_quantization/nn/modules/tensor_quantizer.py”, line 309, in _quant_forward
outputs = fake_tensor_quant(inputs, amax, self._num_bits, self._unsigned, self._narrow_range)
File “/usr/local/lib/python3.7/dist-packages/pytorch_quantization-2.1.0-py3.7-linux-x86_64.egg/pytorch_quantization/tensor_quant.py”, line 305, in forward
outputs, scale = _tensor_quant(inputs, amax, num_bits, unsigned, narrow_range)
File “/usr/local/lib/python3.7/dist-packages/pytorch_quantization-2.1.0-py3.7-linux-x86_64.egg/pytorch_quantization/tensor_quant.py”, line 326, in _tensor_quant
raise TypeError(“Negative values encountered in unsigned quantization.”)
TypeError: Negative values encountered in unsigned quantization.
#######################################################################################

My Quantization setting is
“ quant_desc_input = QuantDescriptor(num_bits=8, unsigned=True)
quant_desc_weight = QuantDescriptor(num_bits=8, unsigned=True)
quant_nn.QuantConv2d.set_default_quant_desc_input(quant_desc_input)
quant_nn.QuantConv2d.set_default_quant_desc_weight(quant_desc_weight)”

I have debuged the error, and found it happened error in the network forward process. But I checked the input, and found the input are all positive. I don’t know why?

Environment

TensorRT Version: 8.0.2
GPU Type: v100（32GB）
Nvidia Driver Version: 450.80.02
CUDA Version: 11.1
CUDNN Version:
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.7.3
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.9.0
Baremetal or Container (if container which image + tag):

Relevant Files

Steps To Reproduce

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

NVES · September 28, 2021, 5:38am

Hi , UFF and Caffe Parser have been deprecated from TensorRT 7 onwards, hence request you to try ONNX parser.
Please check the below link for the same.

Thanks!

sjtu_llq · October 3, 2021, 11:33am

Hi, thanks for your reply.
However, I think it is the problem of the UFF and Caffe Parser, even the onnx parser.
In fact, I did not carry on that step, I just test the quantized network’s inference functionality in Pytorch.

It seems the pytorch-quantization tool does not support unsigned quantization well.
I have found the reason. The following is an example:
if a weight tensor have negtive values and positve values, and we use the unsigned quantization, the tool would call something like this:
“quant_weight = self._weight_quantizer(self.weight)”, and this line will call something like this:
“if self._if_quant: outputs = self._quant_forward(inputs)”, and this line will call:
“outputs = fake_tensor_quant(inputs, amax, self._num_bits, self._unsigned, self._narrow_range)”, and this will call:
def _tensor_quant(inputs, amax, num_bits=8, unsigned=False, narrow_range=True):
“”“Shared function body between TensorQuantFunction and FakeTensorQuantFunction”“”
# Fine scale, per channel scale will be handled by broadcasting, which could be tricky. Pop a warning.
if isinstance(amax, torch.Tensor) and inputs.dim() != amax.dim():
logging.debug(“amax %s has different shape than inputs %s. Make sure broadcast works as expected!”,
amax.size(), inputs.size())

logging.debug("{} bits quantization on shape {} tensor.".format(num_bits, inputs.size()))

if unsigned:
    if inputs.min() < 0.:
        raise TypeError("Negative values encountered in unsigned quantization.")

# Computation must be in FP32 to prevent potential over flow.
input_dtype = inputs.dtype
if inputs.dtype == torch.half:
    inputs = inputs.float()
if amax.dtype == torch.half:
    amax = amax.float()

min_amax = amax.min()
if min_amax < 0:
    raise ValueError("Negative values in amax")

max_bound = torch.tensor((2.0**(num_bits - 1 + int(unsigned))) - 1.0, device=amax.device)
if unsigned:
    min_bound = 0
elif narrow_range:
    min_bound = -max_bound
else:
    min_bound = -max_bound - 1
scale = max_bound / amax

epsilon = 1. / (1<<24)
if min_amax <= epsilon:  # Treat amax smaller than minimum representable of fp16 0
    zero_amax_mask = (amax <= epsilon)
    scale[zero_amax_mask] = 0  # Value quantized with amax=0 should all be 0

outputs = torch.clamp((inputs * scale).round_(), min_bound, max_bound)

if min_amax <= epsilon:
    scale[zero_amax_mask] = 1.  # Return 1 makes more sense for values quantized to 0 with amax=0

if input_dtype == torch.half:
    outputs = outputs.half()

return outputs, scale

but the input weight tensor have negative values. but it seems not right, if the tensor have negative values, then it cannot quantize
please

spolisetty · October 5, 2021, 8:42am

Hi,

We are looking into this issue, please allow us sometime to get back on this.

Thank you.

spolisetty · October 5, 2021, 1:13pm

Hi @sjtu_llq,

Thank you for letting us know this. We will work on this issue.

spolisetty · October 13, 2021, 4:11am

Hi,

It is TensorRT design that we would throw an error when apply unsigned quant descriptor on negative weights.

Thank you.

sjtu_llq · October 20, 2021, 8:14am

Thank you for your reply
I know that
I think this should be fixed as the weights for the first layer must have negative values

spolisetty · October 22, 2021, 11:35am

Hi,

When the weights have negative values, let’s use signed quantization. else all negative values will clip to 0, we cannot guarantee the accuracy because of this clip error.

Thank you.

Topic		Replies	Views
How exactly are you supposed to do explicit quantization? TensorRT	1	56	March 4, 2025
Fake quantization ONNX model parse ERROR using TensorRT TensorRT	6	1773	September 26, 2021
TensorRT conversion issues of ONNX model trained with Quantization Aware Training + custom quantization scale TensorRT tensorrt	5	1374	April 14, 2021
Fake quantization ONNX model parse ERROR using TensorRT 8 TensorRT	3	790	September 27, 2021
mobilenet onnx problem TensorRT	10	1877	October 12, 2021
Cannot create the calibration cache for the QAT model in tensorRT 8 TensorRT tensorrt , pytorch , calibration , onnx	4	2229	March 15, 2022
Model quantized with explicit precision mode (with Q/DQ nodes) failed in engine generation TensorRT	6	959	December 17, 2022
TensorRT get different result in python and c++ TensorRT	21	2876	August 24, 2022
How to quantize a model for Tensorrt? TensorRT tensorrt , pytorch , python	0	56	February 6, 2025
TensorRT quantization bug on Jetpack 6.0 Jetson AGX Orin tensorrt , pytorch	6	603	January 22, 2024

Negative values encountered in unsigned quantization

Description

Environment

Relevant Files

Steps To Reproduce

Related topics