How to load and deserialize the .engine file?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

My device is Jetson TX2. I used the deepstream to accelerate my yolov3-tiny model. The deepstream sample helped me generate an .engine file. I want to load and deserialize this .engine file with Tensor RT C++ API.
I referenced this file:

https://github.com/dusty-nv/jetson-inference/blob/e168b746d540fd739a6fa6ba652b5a5e8c6ffa08/tensorNet.cpp

However, I met these prolems when deserializing the .engine file:
could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin YoloLayerV3_TRT version 1 namespace
Cannot deserialize plugin YoloLayerV3_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin LReLU_TRT version 1 namespace
Cannot deserialize plugin LReLU_TRT
getPluginCreator could not find plugin YoloLayerV3_TRT version 1 namespace
Cannot deserialize plugin YoloLayerV3_TRT
Press to close this window…

Here are my codes:
[b]bool mEnableFP16 = true;
bool mOverride16 = false;
std::stringstream gieModelStream;
gieModelStream.seekg(0, gieModelStream.beg);

const char* cache_path = “/home/nvidia/Documents/Documents/deepstreamYOLO/tensorrtengine/model_b1_fp16.engine”;
std::ifstream cache(cache_path);

if( !cache )
{
cout<<“file doesn’t exist!”<<endl;
}
else
{
cout<< “loading network profile from cache…”<<endl;
gieModelStream << cache.rdbuf();
cache.close();
// test for half FP16 support
nvinfer1::IBuilder* builder = CREATE_INFER_BUILDER(gLogger);

if( builder != NULL )
{
mEnableFP16 = !mOverride16 && builder->platformHasFastFp16();
printf( “platform %s FP16 support.\n”, mEnableFP16 ? “has” : “does not have”);
builder->destroy();
}
}

printf(“%s loaded\n”, cache_path);

/*

  • create runtime inference engine execution context
    /
    nvinfer1::IRuntime     infer = CREATE_INFER_RUNTIME(gLogger);

if( !infer )
{
printf(“failed to create InferRuntime\n”);
return 0;
}

// support for stringstream deserialization was deprecated in TensorRT v2
// instead, read the stringstream into a memory buffer and pass that to TRT.
gieModelStream.seekg(0, std::ios::end);
const int modelSize = gieModelStream.tellg();
gieModelStream.seekg(0, std::ios::beg);
printf(“the model size is %d\n”,modelSize);
void* modelMem = malloc(modelSize);

if( !modelMem )
{
printf(“failed to allocate %i bytes to deserialize model\n”, modelSize);
return 0;
}

gieModelStream.read((char*)modelMem, modelSize);
nvinfer1::ICudaEngine* engine = infer->deserializeCudaEngine(modelMem, modelSize, NULL);
free(modelMem);

if( !engine )
{
printf(“failed to create CUDA engine\n”);
return 0;
}[/b]

1 Like
2

Following is the ideal code

std::vector<char> trtModelStream_;
size_t size{ 0 };

std::ifstream file("outtest.engine", std::ios::binary);
if (file.good())
{
file.seekg(0, file.end);
size = file.tellg();
file.seekg(0, file.beg);
trtModelStream_.resize(size);
std::cout << "size" << trtModelStream_.size() << std::endl;
file.read(trtModelStream_.data(), size);
file.close();
}
std::cout << "size" << size;
IRuntime* runtime = createInferRuntime(gLogger);
assert(runtime != nullptr);
ICudaEngine* engine = runtime->deserializeCudaEngine(trtModelStream_.data(), size, nullptr);
4 Likes
3

If you get some plugin deserialization errors, don’t forget:

initLibNvInferPlugins(&this->m_Logger, "");
1 Like
4

Hello, I am getting the same error, but the plugin it is referring to is FancyActivation. I tried including the initLibNvInferPlugins function you referenced, but with no luck in resolving the issue. Here is my code for loading the .engine file and deserializing it:

initLibNvInferPlugins(&gLogger, “”);

std::stringstream modelStream_Pose;
modelStream_Pose.seekg(0, modelStream_Pose.beg);
std::ifstream cache(“./pose_estimation.engine”);
modelStream_Pose << cache.rdbuf();
cache.close();

runtime = createInferRuntime(gLogger);

modelStream_Pose.seekg(0, std::ios::end);
const int modelSize_Pose = modelStream_Pose.tellg();
modelStream_Pose.seekg(0, std::ios::beg);
void* modelMem_Pose = malloc(modelSize_Pose);
modelStream_Pose.read((char*)modelMem_Pose, modelSize_Pose);

engine = runtime->deserializeCudaEngine(modelMem_Pose, modelSize_Pose, NULL);

Do I need to add that line somewhere else in the code? Thanks.