TensorRT engine cannot be safely deleted in deconstruction of a global or static instance

Description

app developer is complaining about the long loading time trt plan file. so i decided to put the loading part at the global init phase and they should be deleted at the end of global lifetime.
As described in title, with the minimum code in attachment. when calling delete m_trtEngine; a segment fault happens.
By the way. if i put gItem in the main function, i works good.

I want to Ask

whether i could delete the context/engine/runtime with C++ RAII mechanism in a global/static instance.

Environment

TensorRT Version: 8.6 8.4
GPU Type: RTX3090 A5000
Nvidia Driver Version: 540, 535
CUDA Version: 12.3 11.8
CUDNN Version: 8.9 8.6
Operating System + Version: Ubuntu 22.04

Relevant Files

#include <NvInfer.h>
#include <string>
#include <iostream>
#include <fstream>
#include <memory>
#include <iterator>
#include <chrono>
#include <thread>
class TrtLogger : public nvinfer1::ILogger
{
public:
    void log(Severity severity, const char *msg) noexcept override {
        std::cerr << msg << std::endl;
    }
};
class Engine {
  public:
    Engine(){}
    Engine(const std::string& trt){
      std::string value;
      std::ifstream stream(trt.c_str(), std::ios::binary);
      stream >> std::noskipws;
      std::copy(std::istream_iterator<char>(stream), std::istream_iterator<char>(), back_inserter(value));
      m_trtRunTime = nvinfer1::createInferRuntime(gLogger);
      m_trtEngine = m_trtRunTime->deserializeCudaEngine(value.data(), value.size());

    }
    ~Engine() {
      if (m_trtEngine) {
        delete m_trtEngine;
        m_trtEngine = nullptr;
      }

      if (m_trtRunTime) {
        delete m_trtRunTime;
        m_trtRunTime = nullptr;
      }
      
    } 
  private:
    TrtLogger gLogger;
    nvinfer1::ICudaEngine *m_trtEngine{nullptr};
    nvinfer1::IRuntime *m_trtRunTime{nullptr};

};
class gItem {
public:
  std::shared_ptr<Engine> ee;
};
gItem gitem;
int main(int argc, char** argv) {
  gitem.ee = std::make_shared<Engine>("../out.trt");
  return 0;
}

Hi @ahawelcome ,
Can you please share your onnx model and repro script and steps.

Thanks

This problem is not related to specific onnx file. I use the Resnet50.onnx provided by the TensorRT example package.
Use trtexec to generate trt plan file out.trt

trtexec --onnx=TensorRT/data/resnet50/ResNet50.onnx   --saveEngine=out.trt 

To build the cpp program, i use CMakeLists.txt below

cmake_minimum_required(VERSION 3.10)
project(prog)

find_package(CUDA REQUIRED)

link_directories(TensorRT/lib)
cuda_add_executable(main main.cpp)
target_include_directories(main PRIVATE TensorRT/include)
target_link_libraries(main nvinfer nvparsers)

project structure like
proj/
|–TensorRT/
|
|–build/
|
|–main.cpp
|
|–CMakeLists.txt
|
|–out.trt

TensorRT/ contain the the libs and headers of tensorrt

then,

cd build
cmake ..
make
./main

which produce

Loaded engine size: 109 MiB
Deserialization required 17090 microseconds.
[MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +107, now: CPU 0, GPU 107 (MiB)
[1]    2215031 segmentation fault (core dumped)  ./main

error massage may change every time you run the exacutable, Sometimes massage below may appear several times

1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (driver shutting down)

Hi @ahawelcome ,
Can you please try TRT 9,and try a repro.
Thanks

I am experiencing the same issue. Have you found a solution yet?

According to the discussion here: Cuda Run time library unload, The problem is caused by calling CUDA runtime API out of main function.

This error itself won’t end up with a core dump. The core dump is caused by TensorRT. maybe the overloaded delete operator on trt engine raised the problem.

Here are 2 solutions I can see:

  1. add a function to release the global object before end of main function.
  2. Define your global object as a pointer. so the destructor won’t be called in the end.

Personally, I prefer the 1st way, since It cleans everything up. So I can use the code of the object at anywhere else without worrying about errors caused by destructor.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.