Description
I have a running program, that usually works fine. But sometimes after restarting it fails with logs below:
terminate called after throwing an instance of ‘char const*’
ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
ERROR: INVALID_STATE: std::exception
ERROR: …/rtSafe/coreReadArchive.cpp (63) - Serialization Error in verifyHeader: 0 (CRC-32 checksum does not match value in archive).
Then it can start again, without any issues. This happens not so often. From logs I see, that it happens in one of the functions below. In the first function I write an engine to a file, and in the second function I read it and use it further. What can cause this problem?
1 - function
void serializeEngine(ICudaEngine* engine, const string& engineFilename) {
ofstream engineFile(engineFilename, ios::binary);
unique_ptr<IHostMemory, TensorRtDeleter> trtModelStream{engine->serialize(), TensorRtDeleter()};
engineFile.write((char*) trtModelStream->data(), trtModelStream->size());
if (engineFile.bad())
cerr << “Char_const. SerializeEngine: I/O error while write” << endl;
else if (engineFile.eof())
cerr << “Char_const. SerializeEngine: End of file reached successfully” << endl;
else if (engineFile.fail())
cerr << “Char_const. SerializeEngine: Non-integer data encountered” << endl;
else
cout << “Char_const. SerializeEngine: “<< trtModelStream->size() <<” bytes were written to enginefile” << endl;
}
2 - function
ICudaEngine* readEngine(const string& engineFilename) {
ifstream engineFile(engineFilename);
engineFile.seekg(0, ios::end);
const int modelSize = engineFile.tellg();
engineFile.seekg(0, ios::beg);
vector<char> engineData(modelSize);
engineFile.read(engineData.data(), modelSize);
if (engineFile.bad())
cerr << "Char_const. ReadEngine: I/O error while reading" << endl;
else if (engineFile.eof())
cerr << "Char_const. ReadEngine: End of file reached successfully" << endl;
else if (engineFile.fail())
cerr << "Char_const. ReadEngine: Non-integer data encountered" << endl;
cout << "Char_const. ReadEngine: "<< engineFile.gcount() <<" bytes were read from enginefile" << endl;
TrtLogger trtLogger;
auto infer = unique_ptr<IRuntime, TensorRtDeleter>(createInferRuntime(trtLogger), TensorRtDeleter());
ICudaEngine* engineReturn;
try {
engineReturn = (infer->deserializeCudaEngine(engineData.data(), modelSize, nullptr));
}
catch (const exception& ex) {
cerr << "Char_const. Error in readEngine - " << ex.what() << endl;
return nullptr;
}
return engineReturn;
}
Environment
TensorRT Version: 6.0.1.5
GPU Type: GeForce GTX 1050 Ti
Nvidia Driver Version: 455.23.05
CUDA Version: 11.1
CUDNN Version: 7.6.5.32
Operating System + Version: cudagl:10.1-devel-ubuntu16.04
Python Version (if applicable): -
TensorFlow Version (if applicable): -
PyTorch Version (if applicable): -
Baremetal or Container (if container which image + tag):-