How to serialize TensorRT model into file and deserialize from file?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I managed to get a network parsed from caffe and I serialized it.

According to the following program, I saved the serialized result to “serialized_engine.txt” file successfully .

IHostMemory serializedModel = engine->serialize();
…
// store model to disk
std::ofstream ofs(“serialized_engine.txt”, std::ios::out | std::ios::binary);
ofs.write((char)(serialized_model ->data()), serialized_model ->size());
ofs.close();

But I don’t know how to use the “serialized_engine.txt” and haven’t found something useful in the examples.

Anyone knows how to transform the “serialized_engine.txt” to the serializedModel in inference ?

2

Hi.

Please take a look at the latest blog post below. Section “Reuse the TensorRT Engine” describes deserialization process. In short, you need to read the binary file into the memory (e.g. character array), create IRuntime object and use it’s method to recreate engine from serialized part.

I hope this helps. Here is the link to blog post:

Best regards,
Piotr Wojciechowski

1 Like
3

Hi Piotr,

Thanks for the blog.

In the code example within “Reuse the TensorRT Engine”, there is the following line:

string buffer = readBuffer(enginePath);

I assume the “readBuffer” function is just taking the contents of the engine file and placing it withing the string buffer. Could you please elaborate a little on your preferred method of doing this?

1 Like
4

Hi Deenz.

You are correct. The purpose of readBuffer function is to read binary file and place it’s content into a collection of characters. You can find it’s source code at the following GitHub location below. For a reference, I also copy-paste function definition.

Best regards,
Piotr Wojciechowski

source: https://github.com/parallel-forall/code-samples/blob/master/posts/TensorRT-introduction/ioHelper.cpp

// Returns empty string iff can't read the file
string readBuffer(string const& path)
{
    string buffer;
    ifstream stream(path.c_str(), ios::binary);

    if (stream)
    {
        stream >> noskipws;
        copy(istream_iterator<char>(stream), istream_iterator<char>(), back_inserter(buffer));
    }

    return buffer;
}