[TensorRT] Create multiple engines in one python script

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello Everybody,

i have to load multiple tensorRT engines in one session, but when i do it my program crashes, while parsing the prototxt and caffemodel from the second engine.

My reproducible error code is:

G_LOGGER = tensorrt.infer.ConsoleLogger(tensorrt.infer.LogSeverity.ERROR)
    engine = tensorrt.utils.caffe_to_trt_engine(G_LOGGER, #Standard way to produces a FP32-Engine
                                           MODEL_PROTOTXT,
                                           CAFFE_MODEL,
                                           1,
                                           256 << 20,
                                           OUTPUT_LAYERS,
                                           tensorrt.infer.DataType.FLOAT)
    engine.destroy()
    del engine
    print ('|--------------------FAILS--------------------|')
    B_LOGGER = tensorrt.infer.ConsoleLogger(tensorrt.infer.LogSeverity.ERROR)
    engine = tensorrt.utils.caffe_to_trt_engine(B_LOGGER, #Standard way to produces a FP32-Engine
                                           MODEL_PROTOTXT,
                                           CAFFE_MODEL,
                                           1,
                                           256 << 20,
                                           OUTPUT_LAYERS,
                                           tensorrt.infer.DataType.FLOAT)

The python backtrace is:

File "/usr/local/lib/python2.7/dist-packages/tensorrt/utils/_utils.py", line 280, in caffe_to_trt_engine
    blob_name_to_tensor = parser.parse(deploy_file, model_file, network, model_datatype)
File "playground.py", line 80, in main
    tensorrt.infer.DataType.DataType_kHALF)
File "playground.py", line 89, in <module>
    main()

And the error itself is:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffc82f832b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

Does anybody have an idea what i am doing wrong?

Thanks!

2

I experienced the same error. Did you solve it yet?

3

I came up with one possible solution on how to load and use multiple engines.
It is not possible to build more than one engine in the same session. But it is possible to load multiple serialized engines from disk and use them simultaneously.
My suggestion is therefore to first build and serialize all of your models separately and then load them via tensorrt.utils.load_engine.

4

+1

5

I am trying to load multiple engines in a ROS nodelet and am running into the same issue. Everything works great with a single TensorRT engine, but when I add a second the nodelet manager crashes. Using the TensorRT C++ API and am using IRuntime::deserializeCudaEngine to load the engine rather than building it.