Recently I used Torch-TensorRT to quantized YOLOx. I calibrated the model using PTQ and saved cache file. However, when I want to use this cache file in DataLoaderCalibrator, an error occured:
ERROR: [Torch-TensorRT] - Input cache file is None but use_cache is set to True in INT8 mode.
I believe I have set the right path to cache file so I checked the source code of DataLoaderCalibrator. I found out that in __new__() method the following lines were used to determine whether to use cache file or not:
class DataLoaderCalibrator(object):
def __init__(self, **kwargs):
pass
def __new__(cls, *args, **kwargs):
dataloader = args[0]
algo_type = kwargs.get("algo_type", CalibrationAlgo.ENTROPY_CALIBRATION_2)
cache_file = kwargs.get("cache_file", None)
use_cache = kwargs.get("use_cache", False)
device = kwargs.get("device", torch.device("cuda:0"))
if not isinstance(dataloader, torch.utils.data.DataLoader):
log(Level.Error,
"Dataloader : {} is not a valid instance of torch.utils.data.DataLoader".format(dataloader))
if not cache_file: # here not is wrong
if use_cache:
log(Level.Debug, "Using existing cache_file {} for calibration".format(cache_file))
else:
log(Level.Debug, "Overwriting existing calibration cache file.")
else:
if use_cache:
log(Level.Error, "Input cache file is None but use_cache is set to True in INT8 mode.")
and I think that if not cache_file: should be if cache_file: instead.
Environment
TensorRT Version: 8.4.0.6 GPU Type: NVIDIA GeForce RTX 3090 Nvidia Driver Version: 470.57.02 CUDA Version: 11.3 CUDNN Version: 8.4.0 Operating System + Version: Ubuntu 20.04 Python Version (if applicable): 3.8 TensorFlow Version (if applicable): PyTorch Version (if applicable): 1.11.0 Baremetal or Container (if container which image + tag):
Could you please let us know original source for the above repo you referred.
You can try the same VGG16 with the CIFAR dataset QAT sample if interested.
Hi,
in fact the source of this repo is VGG16 with the CIFAR dataset provided in Torch-TensorRT repo. I calibrated the model and got calibration.cache file. However, when I want to use this file, I set cache_file to calibration.cache and use_cache to True in DataLoaderCalibrator(), an error was raised:
ERROR: [Torch-TensorRT] - Input cache file is None but use_cache is set to True in INT8 mode.
I checked the source code of DataLoaderCalibrator() in Torch-TensorRT library and I think it would be better without not in __new__() method.
class DataLoaderCalibrator(object):
def __init__(self, **kwargs):
pass
def __new__(cls, *args, **kwargs):
dataloader = args[0]
algo_type = kwargs.get("algo_type", CalibrationAlgo.ENTROPY_CALIBRATION_2)
cache_file = kwargs.get("cache_file", None)
use_cache = kwargs.get("use_cache", False)
device = kwargs.get("device", torch.device("cuda:0"))
if not isinstance(dataloader, torch.utils.data.DataLoader):
log(Level.Error,
"Dataloader : {} is not a valid instance of torch.utils.data.DataLoader".format(dataloader))
if not cache_file: # here not is wrong
if use_cache:
log(Level.Debug, "Using existing cache_file {} for calibration".format(cache_file))
else:
log(Level.Debug, "Overwriting existing calibration cache file.")
else:
if use_cache:
log(Level.Error, "Input cache file is None but use_cache is set to True in INT8 mode.")
This issue looks more in the scope of Pytorch, we recommend you to please reach out Pytorch forum to get better help.
Or you can use TensorRT’s Quantization Toolkit.