Description
I would like to make a TensorRT inference module that is callable from the outside. I would like to make two methods:
- An initialization method to load the engine from a file
- An inference method to actually run the inference
The idea is that the initialization should occur once after my application is started, but there will thereupon be many inference calls.
I am using the “sampleOnnxMNIST” as an example.
Original example has everything in one method:
int main(int argc, char** argv)
{
samplesCommon::Args args;
bool argsOK = samplesCommon::parseArgs(args, argc, argv);
if (!argsOK)
{
sample::gLogError << "Invalid arguments" << std::endl;
printHelpInfo();
return EXIT_FAILURE;
}
if (args.help)
{
printHelpInfo();
return EXIT_SUCCESS;
}
auto sampleTest = sample::gLogger.defineTest(gSampleName, argc, argv);
sample::gLogger.reportTestStart(sampleTest);
SampleOnnxMNIST sample(initializeSampleParams(args));
sample::gLogInfo << "Building and running a GPU inference engine for Onnx MNIST" << std::endl;
if (!sample.build())
{
return sample::gLogger.reportFail(sampleTest);
}
if (!sample.infer())
{
return sample::gLogger.reportFail(sampleTest);
}
return sample::gLogger.reportPass(sampleTest);
}
I basically want to do something like the following;
bool BuildTensorRTEngine(char* inputLayerName, char* outputLayerName, char* engineFile)
{
// Do some initialization stuff here...
SampleOnnxMNIST sample(initializeSampleParams(inputLayerName, outputLayerName));
sample::gLogInfo << "Building and running a GPU inference engine for Onnx MNIST" << std::endl;
if (!sample.build(engineFile))
{
return sample::gLogger.reportFail(sampleTest);
}
return sample::gLogger.reportPass(sampleTest);
}
bool RunInference(float* input)
{
if (!sample.infer(input))
{
return sample::gLogger.reportFail(sampleTest);
}
return sample::gLogger.reportPass(sampleTest);
}
What’s missing above is the creation and sharing of the logger between the two callable methods. What is the recommended approach in this scenario?
Environment
TensorRT Version: 7.2 C++ API
CUDA Version: 11.1
CUDNN Version: 8.0.5
Operating System + Version: Windows 10