Cannot load the network after upgrading to TensorRT 8 from 7

Description

I decided to upgrade to tensorrt 8 and when i tried to run the app i was working on i encountered errors when loading the onnx network or serialized engine.

I checked the documentation and i saw that the function buildEngineWithConfig was deprecated so i changed the code to use buildSerializedNetwork but both functions are failing to return anything except a null pointer. I set the logger severity to display all information available and i could see that the builder is trying to optimize the network but exits the function too early.

I tried to use the serialized model generated with TensorRT 7 but the runtime object fails to return an engine object just like the builder.

Environment

TensorRT Version: 8.0.1.1
GPU Type: GTX 1660
Nvidia Driver Version: 440.82
CUDA Version: 10.2
CUDNN Version:
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

nvinfer1::IBuilder* builder = nvinfer1::createInferBuilder(gLogger);
nvinfer1::INetworkDefinition* network = builder->createNetworkV2(1u << (int)nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
nvonnxparser::IParser* parser = nvonnxparser::createParser(network, gLogger);
// parse ONNX
if (!parser->parseFromFile(“densenet.onnx”, static_cast< int >(nvinfer1::ILogger::Severity::kVERBOSE)))
{
std::cerr << “ERROR: could not parse the model.\n”;
return;
}
nvinfer1::IBuilderConfig
config = builder->createBuilderConfig();
config->setMaxWorkspaceSize(1ULL << 25);
// use FP16 mode if possible
if (builder->platformHasFastFp16())
{
config->setFlag(nvinfer1::BuilderFlag::kFP16);
}
// we have only one image in batch
builder->setMaxBatchSize(1);
auto profile = builder->createOptimizationProfile();
profile->setDimensions(“input”, nvinfer1::OptProfileSelector::kMIN, nvinfer1::Dims4(1,3,256,256));
profile->setDimensions(“input”, nvinfer1::OptProfileSelector::kOPT, nvinfer1::Dims4(1,3,256,256));
profile->setDimensions(“input”, nvinfer1::OptProfileSelector::kMAX, nvinfer1::Dims4(1,3,256,256));
config->addOptimizationProfile(profile);
auto data = builder->buildSerializedNetwork(*network, config);
std::ofstream out(“densenet.engine”, std::ios::binary);
out.write(static_cast<char
>(data->data()), data->size());
out.close();

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:
https://docs.nvidia.com/deeplearning/tensorrt/quick-start-guide/index.html#onnx-export

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

Hello,

I uploaded the onnx model here: WeTransfer - Send Large Files & Share Photos Online - Up to 2GB Free
Unfortunately I cannot access the github page, it gives me error 404 - Not Found.
I executed the python commands but check_model doesn’t seem to do anything, no errors and no output info.

Thank you for your help,

Hello,

So i managed to run trtexec after all and here are the last lines:

[08/27/2021-14:26:55] [E] Error[2]: [ltWrapper.cpp::setupHeuristic::327] Error Code 2: Internal Error (Assertion cublasStatus == CUBLAS_STATUS_SUCCESS failed.)
[08/27/2021-14:26:55] [E] Error[2]: [builder.cpp::buildSerializedNetwork::417] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed.)
Segmentation fault (core dumped)

I managed to generate the engine file from onnx using the command

trtexec --onnx=xxx.onnx --tacticSources=-cublasLt,+cublas --workspace=2048 --fp16 --saveEngine=xxx.engine

I used the output engine file as an input to my main app and the runtime builder was able to generate the ICudaEngine object and inferences are working again (twice as fast compared to TensorRT 7 :)) )