Classification model of densenet converted to int8 that outputs result is error!

Linux distro and version - Linux-x86_64, Ubuntu, 16.04
GPU type - GeForce GTX 1080ti
nvidia driver version - 384.130
CUDA version - Release 9.0, V9.0.176
CUDNN version - 7.5.3
TensorRT version – 5.1.5.0

I’m using the TensorRT C++ APIs in order to inference a CNN model (densenet) that was developed and trained using caffe. The model is used for classification for six class.
When I transfer it to the fp32 model, its result is consistent with that of Caffe, but it is totally inconsistent with that of int8. No matter what image I enter, the ouput of Int8 model is the same.
Looking closely at each layer of the model and finds that the results are quite different when I find the sixth convolutional layer. This model has no special layer.I don’t know how to solve this problem now.
Looking forward to your answer! thinks

Hi,

This sounds like it might be a problem with int8 calibration.

How did you handle int8 calibration when creating your int8 model?

Thanks,
NVIDIA Enterprise Support

Thank you for your reply!
I used the calibrator of Int8EntropyCalibrator2, and used 600 pictures as the data of calibrator.The preprocess of images is the same as the forward of algorithm .Strangely,I processed other classfication models used the same codes for int8 calibrator, which is successed!

// preprocess calibrator images
auto data = PreProcess(img);// The preprocess is right!
calibData.emplace_back(data);
std::random_shuffle(calibData.begin(),calibData.end());

//the part code of calibrator process
if (calibratorData.size() > 0 ){
  auto endPos= deployFile.find_last_of(".");
  auto beginPos= deployFile.find_last_of('/') + 1;
  std::string calibratorName = deployFile.substr(beginPos,endPos - beginPos);
  std::cout << "create calibrator,Named:" << calibratorName << std::endl;
  calibrator = new wave::Int8EntropyCalibrator(maxBatchSize,calibratorData, inputBlobName, calibratorName);
}
   
if (mTrtRunMode == RUN_MODE::INT8){
   printf("WaveTrt::loadModelAndCreateEngine:setInt8Mode!\n");
   if (!builder->platformHasFastInt8()) 
      printf(" Notice: the platform do not has fast for int8!\n");
      builder->setInt8Mode(true);
      builder->setInt8Calibrator(calibrator);
    }

Looking forward to your answer! thinks

Thank you for your reply!
I used the calibrator of Int8EntropyCalibrator2, and used 600 pictures as the data of calibrator.The preprocess of images is the same as the forward of algorithm .Strangely,I processed other classfication models used the same codes for int8 calibrator, which is successed!

// preprocess calibrator images
auto data = PreProcess(img);// The preprocess is right!
calibData.emplace_back(data);
std::random_shuffle(calibData.begin(),calibData.end());


//the part code of calibrator process
if (calibratorData.size() > 0 ){
  auto endPos= deployFile.find_last_of(".");
  auto beginPos= deployFile.find_last_of('/') + 1;
  std::string calibratorName = deployFile.substr(beginPos,endPos - beginPos);
  std::cout << "create calibrator,Named:" << calibratorName << std::endl;
  calibrator = new wave::Int8EntropyCalibrator(maxBatchSize,calibratorData, inputBlobName, calibratorName);
}
   
if (mTrtRunMode == RUN_MODE::INT8){
   printf("WaveTrt::loadModelAndCreateEngine:setInt8Mode!\n");
   if (!builder->platformHasFastInt8()) 
      printf(" Notice: the platform do not has fast for int8!\n");
      builder->setInt8Mode(true);
      builder->setInt8Calibrator(calibrator);
    }

Looking forward to your answer! thinks

Hi,

I’m not sure why it would work for your previous models and not this one.

One thing I noticed from your code is that you’re using Int8EntropyCalibrator, but that is now legacy code and a new version IInt8EntropyCalibrator2 has been added: https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/Int8/EntropyCalibrator2.html

Can you try inheriting from this calibrator instead?

Also, you could try to run this with TensorRT 6.0 and see if that helps as well. To avoid reinstalling everything, you could easily run TensorRT6.0 with our NGC container using Docker + Nvidia-Docker: https://ngc.nvidia.com/catalog/containers/nvidia:tensorrt