YOLOv3-tiny TensorRT C++ Plugins Issue

Environment

TensorRT Version: 6.0.1.10
DeepStream SDK Version: 4.0.2
GPU Type: Jetson Xavier running Jetpack 4.3
Nvidia Driver Version:
CUDA Version: 10.0
CUDNN Version: 7.6.3
Operating System + Version: Ubuntu 18.04
Baremetal or Container (if container which image + tag):

Description

I am trying to load a TensorRT based engine file for Darknet YOLOv3-tiny in my C++ program. I converted my darknet weights to a TensorRT engine file using the examples for YOLO located in /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/ directory. The major steps while doing so included: -

  • Updating the custom-network-config and model-file parameters to point towards my yolov3-tiny cfg file and yolov3-tiny weights file respectively in config_infer_primary_yoloV3_tiny.txt
  • Then running the application using the command deepstream-app -c deepstream_app_config_yoloV3_tiny.txt. It did run perfectly fine and I was able to see the output on the test stream (.mp4 file).

Note: I did go to the nvdsinfer_custom_impl_Yolo folder and updated the nvdsparsebbox_Yolo.cpp file according to use case and made the project resulting in the libnvdsinfer_custom_impl_Yolo.so file which was updated according to the use case.

Now I am trying to load this same engine file provided by deepstream to be loaded in my C++ program. I know that this engine file uses some custom tensorrt plugins. According to my understanding, these plugins could be found in the libnvdsinfer_custom_impl_Yolo.so, therefore I link my c++ program with this .so using the -lnvdsinfer_custom_impl_Yolo flag. Currently, my program is: -

#include <iostream>
#include <string>
#include <vector>
#include <array>
#include <sstream>
#include <fstream>
#include <cassert>
#include <numeric>
#include <stdexcept>
#include <stdio.h>

#include <unistd.h>

#include "NvInfer.h"
#include "NvInferPlugin.h"
#include <NvInferRuntimeCommon.h>

class Logger : public nvinfer1::ILogger
{
    void log(nvinfer1::ILogger::Severity severity, const char *msg) override
    {
        // suppress info-level messages
        if (severity != nvinfer1::ILogger::Severity::kINFO)
            std::cout << msg << std::endl;
    }
} gLogger;

int main(int argc, char const *argv[])
{
    initLibNvInferPlugins(&gLogger, "");

    nvinfer1::IRuntime *runtime{nullptr};
    nvinfer1::ICudaEngine *engine{nullptr};
    nvinfer1::IExecutionContext *context{nullptr};

    std::string engine_file_path = "model_b1_fp32.engine";

    std::stringstream gieModelStream; 
    gieModelStream.seekg(0, gieModelStream.beg); 
    std::ifstream cache( engine_file_path ); 
    gieModelStream << cache.rdbuf();
    cache.close();

    runtime = nvinfer1::createInferRuntime(gLogger); 
    assert(runtime != nullptr); 

    gieModelStream.seekg(0, std::ios::end);
    const int modelSize = gieModelStream.tellg(); 
    gieModelStream.seekg(0, std::ios::beg);
    void* modelMem = malloc(modelSize); 
    gieModelStream.read((char*)modelMem, modelSize);
    engine = runtime->deserializeCudaEngine(modelMem, modelSize, nullptr);
    assert(engine != nullptr); 
    free(modelMem);
    
    std::cout << "Loaded\n";

    context = engine->createExecutionContext();

    assert(engine->getNbBindings() == 2);

    context->destroy();
    engine->destroy();
    runtime->destroy();
    
    return 0;
}

The program compiles perfectly fine and I can load the plugins from the .so file (presumably). The output shows

mubeen@jetson-xavier-5157:~/workspace/yolo_trt$ ./main
Plugin Creator registration succeeded - GridAnchor_TRT
Plugin Creator registration succeeded - GridAnchorRect_TRT
Plugin Creator registration succeeded - NMS_TRT
Plugin Creator registration succeeded - Reorg_TRT
Plugin Creator registration succeeded - Region_TRT
Plugin Creator registration succeeded - Clip_TRT
Plugin Creator registration succeeded - LReLU_TRT
Plugin Creator registration succeeded - PriorBox_TRT
Plugin Creator registration succeeded - Normalize_TRT
Plugin Creator registration succeeded - RPROI_TRT
Plugin Creator registration succeeded - BatchedNMS_TRT
Plugin Creator registration succeeded - FlattenConcat_TRT
INVALID_ARGUMENT: getPluginCreator could not find plugin YoloLayerV3_TRT version 1
safeDeserializationUtils.cpp (259) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
INVALID_STATE: std::exception
INVALID_CONFIG: Deserialize the cuda engine failed.
main: main.cpp:66: int main(): Assertion `engine != nullptr' failed.
Aborted (core dumped)

But as you can see I am still unable to load the model and it cannot find the YoloLayerV3_TRT plugin, shouldn’t this plugin be in the .so file, or am I missing something? Any help would be really appreciated.

Hi @mmubeen,
To use TensorRT registered plugins in your application, the libnvinfer_plugin.so library must be loaded and all plugins must be registered. This can be done by calling initLibNvInferPlugins(void* logger, const char* libNamespace)() in your application code.
Please note that If you have your own plugin library, you can include a similar entry point to register all plugins in the registry under a unique namespace. This ensures there are no plugin name collisions during build time across different plugin libraries.
Please find the reference link for adding custom plugin
https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-713/developer-guide/index.html#add_custom_layer
Thanks!