Bug of Torch-TensorRT DataLoaderCalibrator class

Description

Recently I used Torch-TensorRT to quantized YOLOx. I calibrated the model using PTQ and saved cache file. However, when I want to use this cache file in DataLoaderCalibrator, an error occured:

ERROR: [Torch-TensorRT] - Input cache file is None but use_cache is set to True in INT8 mode.

I believe I have set the right path to cache file so I checked the source code of DataLoaderCalibrator. I found out that in __new__() method the following lines were used to determine whether to use cache file or not:

class DataLoaderCalibrator(object):
    def __init__(self, **kwargs):
        pass
    
    def __new__(cls, *args, **kwargs):
        dataloader = args[0]
        algo_type = kwargs.get("algo_type", CalibrationAlgo.ENTROPY_CALIBRATION_2)
        cache_file = kwargs.get("cache_file", None)
        use_cache = kwargs.get("use_cache", False)
        device = kwargs.get("device", torch.device("cuda:0"))
    
        if not isinstance(dataloader, torch.utils.data.DataLoader):
            log(Level.Error,
                "Dataloader : {} is not a valid instance of torch.utils.data.DataLoader".format(dataloader))
    
        if not cache_file: # here not is wrong
            if use_cache:
                log(Level.Debug, "Using existing cache_file {} for calibration".format(cache_file))
            else:
                log(Level.Debug, "Overwriting existing calibration cache file.")
    
        else:
            if use_cache:
                log(Level.Error, "Input cache file is None but use_cache is set to True in INT8 mode.")

and I think that if not cache_file: should be if cache_file: instead.

Environment

TensorRT Version: 8.4.0.6
GPU Type: NVIDIA GeForce RTX 3090
Nvidia Driver Version: 470.57.02
CUDA Version: 11.3
CUDNN Version: 8.4.0
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.8
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.11.0
Baremetal or Container (if container which image + tag):

Relevant Files

link to source code of DataLoaderCalibrator torch_tensorrt.ptq — Torch-TensorRT master documentation

Hi,

Could you please share with us the issue repro model and script for better debugging.

Thank you.

Hi I’ve created a repo in Github, you can repro this issue using following commands:

git clone https://github.com/QQQQ00243/Torch-TensorRT-VGG16.git
python main.py --epochs 10
python issue.py 

You will see the error message:

ERROR: [Torch-TensorRT] - Input cache file is None but use_cache is set to True in INT8 mode.

though the cache file is already set.

Hi,

Could you please let us know original source for the above repo you referred.
You can try the same VGG16 with the CIFAR dataset QAT sample if interested.

Thank you.

Hi,
in fact the source of this repo is VGG16 with the CIFAR dataset provided in Torch-TensorRT repo. I calibrated the model and got calibration.cache file. However, when I want to use this file, I set cache_file to calibration.cache and use_cache to True in DataLoaderCalibrator(), an error was raised:

ERROR: [Torch-TensorRT] - Input cache file is None but use_cache is set to True in INT8 mode.

I checked the source code of DataLoaderCalibrator() in Torch-TensorRT library and I think it would be better without not in __new__() method.

class DataLoaderCalibrator(object):
    def __init__(self, **kwargs):
        pass
    
    def __new__(cls, *args, **kwargs):
        dataloader = args[0]
        algo_type = kwargs.get("algo_type", CalibrationAlgo.ENTROPY_CALIBRATION_2)
        cache_file = kwargs.get("cache_file", None)
        use_cache = kwargs.get("use_cache", False)
        device = kwargs.get("device", torch.device("cuda:0"))
    
        if not isinstance(dataloader, torch.utils.data.DataLoader):
            log(Level.Error,
                "Dataloader : {} is not a valid instance of torch.utils.data.DataLoader".format(dataloader))
    
        if not cache_file: # here not is wrong
            if use_cache:
                log(Level.Debug, "Using existing cache_file {} for calibration".format(cache_file))
            else:
                log(Level.Debug, "Overwriting existing calibration cache file.")
    
        else:
            if use_cache:
                log(Level.Error, "Input cache file is None but use_cache is set to True in INT8 mode.")

source of above code: torch_tensorrt.ptq — Torch-TensorRT master documentation

Hi,

This issue looks more in the scope of Pytorch, we recommend you to please reach out Pytorch forum to get better help.
Or you can use TensorRT’s Quantization Toolkit.

Thank you.