Serialization Error in verifyHeader: 0 (CRC-32 checksum does not match value in archive)

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Description

I have a running program, that usually works fine. But sometimes after restarting it fails with logs below:

terminate called after throwing an instance of ‘char const*’
ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
ERROR: INVALID_STATE: std::exception
ERROR: …/rtSafe/coreReadArchive.cpp (63) - Serialization Error in verifyHeader: 0 (CRC-32 checksum does not match value in archive).

Then it can start again, without any issues. This happens not so often. From logs I see, that it happens in one of the functions below. In the first function I write an engine to a file, and in the second function I read it and use it further. What can cause this problem?

1 - function
void serializeEngine(ICudaEngine* engine, const string& engineFilename) {
ofstream engineFile(engineFilename, ios::binary);
unique_ptr<IHostMemory, TensorRtDeleter> trtModelStream{engine->serialize(), TensorRtDeleter()};
engineFile.write((char*) trtModelStream->data(), trtModelStream->size());
if (engineFile.bad())
cerr << “Char_const. SerializeEngine: I/O error while write” << endl;
else if (engineFile.eof())
cerr << “Char_const. SerializeEngine: End of file reached successfully” << endl;
else if (engineFile.fail())
cerr << “Char_const. SerializeEngine: Non-integer data encountered” << endl;
else
cout << “Char_const. SerializeEngine: “<< trtModelStream->size() <<” bytes were written to enginefile” << endl;
}

2 - function
ICudaEngine* readEngine(const string& engineFilename) {
ifstream engineFile(engineFilename);

engineFile.seekg(0, ios::end);
const int modelSize = engineFile.tellg();
engineFile.seekg(0, ios::beg);

vector<char> engineData(modelSize);
engineFile.read(engineData.data(), modelSize);
if (engineFile.bad())
    cerr << "Char_const. ReadEngine: I/O error while reading" << endl;
else if (engineFile.eof())
    cerr << "Char_const. ReadEngine: End of file reached successfully" << endl;
else if (engineFile.fail())
    cerr << "Char_const. ReadEngine: Non-integer data encountered" << endl;
cout << "Char_const. ReadEngine: "<< engineFile.gcount() <<" bytes were read from enginefile" << endl;

TrtLogger trtLogger;
auto infer = unique_ptr<IRuntime, TensorRtDeleter>(createInferRuntime(trtLogger), TensorRtDeleter());

ICudaEngine* engineReturn;
try {
    engineReturn = (infer->deserializeCudaEngine(engineData.data(), modelSize, nullptr));
}
catch (const exception& ex) {
    cerr << "Char_const. Error in readEngine - " << ex.what() << endl;
    return nullptr;
}
return engineReturn;

}

Environment

TensorRT Version: 6.0.1.5
GPU Type: GeForce GTX 1050 Ti
Nvidia Driver Version: 455.23.05
CUDA Version: 11.1
CUDNN Version: 7.6.5.32
Operating System + Version: cudagl:10.1-devel-ubuntu16.04
Python Version (if applicable): -
TensorFlow Version (if applicable): -
PyTorch Version (if applicable): -
Baremetal or Container (if container which image + tag):-

2

The error message indicates that the program failed to deserialize the CUDA engine. This can occur when the data being read from the file is not in the correct format or is corrupt.

Please check if there are any memory allocation errors when creating or deserializing the engine. If it fails, please make sure the serialization of the engine works fine, or try to generate the engine again.