getPluginCreator could not find plugin BatchedNMS_TRT version 1

gusghrlrl123 · November 1, 2019, 8:46am

Name: NVIDIA Jetson
Type: AGX Xavier
Jetpack: UNKNOWN [L4T 32.2.2] (JetPack 4.3. DP)
GPU-Arch: 7.2

Libraries:

CUDA: 10.0.326
cuDNN: 7.6.3.28-1+cuda10.0
TensorRT: 6.0.1.5-1+cuda10.0
VisionWorks: NOT_INSTALLED
OpenCV: 4.0.0 compiled CUDA: YES

Hi,

I want to connect BatchedNMSPlugin to my detector model.

So, I convert onnx file to trt and connect to batchedNMSPlugin.

I change onnx2trt’s main.cpp.

[main.cpp]

// set nms paramter
      nvinfer1::plugin::NMSParameters nms_parameter;
      nms_parameter.shareLocation = true;
      nms_parameter.backgroundLabelId = -1;
      nms_parameter.numClasses = 1;
      nms_parameter.scoreThreshold = 0.1f;
      nms_parameter.iouThreshold = 0.5f;
      nms_parameter.isNormalized = false;
      nms_parameter.keepTopK = 100;

      // create nms plugin
      nvinfer1::IPluginV2* nms_plugin = createBatchedNMSPlugin(nms_parameter);

      // set nms input tensor
      // 7: outputs_bbox0
      // 8: outputs_score0
      nvinfer1::ITensor* nms_input_tensors[] = {trt_network->getOutput(7), trt_network->getOutput(8)};

      // print nms input tensor
      for (auto i = 0; i < 2; ++i) {
        auto dims = nms_input_tensors[i]->getDimensions();
        cout << i << ": " << dims.nbDims << ": ";
        for (auto j = 0; j < dims.nbDims; ++j)
          cout << dims.d[j] << ", ";
        cout << endl;
      }

      // connect to original network
      auto nms_layer = trt_network->addPluginV2(&nms_input_tensors[0], 2, *nms_plugin);

      // set nms output tensor
      for (auto i = 0; i < nms_layer->getNbOutputs(); ++i)
        trt_network->markOutput(*(nms_layer->getOutput(i)));
      
      // print nms output tensor
      for (auto i = 0; i < nms_layer->getNbOutputs(); ++i) {
        auto dims = nms_layer->getOutput(i)->getDimensions();
        cout << i << ": " << dims.nbDims << ": ";
        for (auto j = 0; j < dims.nbDims; ++j)
          cout << dims.d[j] << ", ";
        cout << endl;
      }

      // remove original detector's output
      trt_network->unmarkOutput(*(trt_network->getOutput(7)));
      trt_network->unmarkOutput(*(trt_network->getOutput(7)));

      // set nms output's name
      char* nms_output_name[] = {"num_detections", "nmsed_boxes", "nmsed_scores", "nmsed_classes"};
      for (auto i = 0; i < 4; ++i)
        trt_network->getOutput(7 + i)->setName(nms_output_name[i]);

      // final nms engine's output
      for (auto i = 0; i < trt_network->getNbOutputs(); ++i) {
        auto nms_tensor = trt_network->getOutput(i);
        auto nms_out_dims = nms_tensor->getDimensions();

        cout << i << ": " << nms_tensor->getName() << ": ";
        for (auto j = 0; j < nms_out_dims.nbDims; ++j)
          cout << nms_out_dims.d[j] << ", ";
        cout << endl;
      }

So, I can get Tensor RT engine file from converted by onnx2trt.

And I want to use this engine file another source.

I want to deserialize this engine and execute engine.

But In this case, I got the error on deserializeCudaEngine().

INVALID_ARGUMENT: getPluginCreator could not find plugin BatchedNMS_TRT version 1
safeDeserializationUtils.cpp (259) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
INVALID_STATE: std::exception
INVALID_CONFIG: Deserialize the cuda engine failed.
Segmentation fault (core dumped)

How can I do?

Thanks.

yahya_qlue · November 13, 2019, 3:24am

Name: NVIDIA GeForce
Type: GTX 1050Ti
Docker container: TensorFlow 19.09-py3
GPU-Arch: 6.1

Hi,

I found same exact issue within TensorFlow container when running TF-TRT optimized graph that has CombinedNonMaxSuppression operator. However, it is stated in Accelerating Inference In TF-TRT User Guide :: NVIDIA Deep Learning Frameworks Documentation that the operator is supported. I ran on GTX 1050Ti laptop using NVIDIA NGC TensorFlow 19.09-py3 container.

2019-11-13 03:00:45.202276: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-11-13 03:00:51.470503: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2208000000 Hz
2019-11-13 03:00:51.470996: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5079d70 executing computations on platform Host. Devices:
2019-11-13 03:00:51.471027: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-11-13 03:00:51.473315: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2019-11-13 03:00:51.545852: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 03:00:51.546170: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5056840 executing computations on platform CUDA. Devices:
2019-11-13 03:00:51.546193: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1050 Ti, Compute Capability 6.1
2019-11-13 03:00:51.546380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 03:00:51.546677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
2019-11-13 03:00:51.546743: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2019-11-13 03:00:51.546786: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-11-13 03:00:51.546815: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2019-11-13 03:00:51.546853: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2019-11-13 03:00:51.548121: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2019-11-13 03:00:51.548738: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2019-11-13 03:00:51.548779: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-11-13 03:00:51.548864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 03:00:51.549126: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 03:00:51.549319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-11-13 03:00:52.084317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-13 03:00:52.084352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2019-11-13 03:00:52.084363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2019-11-13 03:00:52.084576: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 03:00:52.084866: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 03:00:52.085080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3055 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-11-13 03:00:54.905646: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-11-13 03:00:55.886505: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-11-13 03:00:56.228054: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger INVALID_ARGUMENT: getPluginCreator could not find plugin BatchedNMS_TRT version 1
2019-11-13 03:00:56.228100: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger safeDeserializationUtils.cpp (259) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
2019-11-13 03:00:56.229101: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger INVALID_STATE: std::exception
2019-11-13 03:00:56.229151: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger INVALID_CONFIG: Deserialize the cuda engine failed.
*** Aborted at 1573614056 (unix time) try "date -d @1573614056" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 2157 (TID 0x7f2857fff700) from PID 0; stack trace: ***
    @     0x7f2d22615f20 (unknown)
    @     0x7f2ae60c3ac8 tensorflow::tensorrt::TRTEngineOp::GetEngine()
    @     0x7f2ae60c9263 tensorflow::tensorrt::TRTEngineOp::ComputeAsync()
    @     0x7f2b05e49de3 tensorflow::BaseGPUDevice::ComputeAsync()
    @     0x7f2b05ea9416 tensorflow::(anonymous namespace)::ExecutorState::Process()
    @     0x7f2b05ea9b0f _ZNSt17_Function_handlerIFvvEZN10tensorflow12_GLOBAL__N_113ExecutorState13ScheduleReadyERKN4absl13InlinedVectorINS3_10TaggedNodeELm8ESaIS6_EEEPNS3_20TaggedNodeReadyQueueEEUlvE_E9_M_invokeERKSt9_Any_data
    @     0x7f2b05f501f1 Eigen::ThreadPoolTempl<>::WorkerLoop()
    @     0x7f2b05f4d0e8 _ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data
    @     0x7f2d2014466f (unknown)
    @     0x7f2d223bf6db start_thread
    @     0x7f2d226f888f clone

rbainQT3.14 · January 31, 2020, 11:23am

I am having the exact same errors as the original poster. I tried both versions 19.12 and 20.1 through pip3. I also flashed down to JetPack 4.2 with version 19.11, but I get a similar error about not finding the BatchedNMS_TRT plugin…

Name: NVIDIA Jetson
Type: AGX Xavier
Jetpack: UNKNOWN [L4T 32.2.3] (JetPack 4.3. DP)
GPU-Arch: 7.2

Libraries:

CUDA: 10.2
TensorRT: 6

rbainQT3.14 · February 3, 2020, 12:01am

So I had been converting a SavedModel to TRT in one python script, then trying to load it in another. I did both in one script: Load non-TRT model, convert to TRT, save TRT, load TRT, and run inference. Then it worked! So it’s probably something to do with converter.convert, the default parameters, etc. Hope this helps.

LucasJin · December 23, 2020, 7:30am

No, it’s not help

NVES · December 23, 2020, 7:37am

Hi, Request you to check the below reference links for custom plugin implementation.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleOnnxMnistCoordConvAC

Thanks!

Topic		Replies	Views
Calibration failed: INTERNAL: Failed to build TensorRT engine (INT8 precision mode) in Jetson Xavier NX (16GB) Jetson Xavier NX tensorrt	9	762	April 12, 2023
Tf-trt conversion got killed TensorRT tensorrt , tensorflow , jetson-inference	3	752	April 22, 2021
TensorRT problem on NVIDIA APEX ORIN NX TensorRT tensorrt , jetson-inference , cudnn	1	45	August 29, 2024
Error while converting my model to a TensorRT model. Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_0) TensorRT tensorrt	1	2578	December 9, 2021
getPluginCreator could not find plugin: EfficientNMS_TRT version: 1 error with C++ API but works fine with Python API TensorRT tensorrt	14	2967	January 11, 2024
TF-TRT issue Jetson TX2	26	3848	October 18, 2021
Version mismatch help to rectify please :) Jetson Nano	7	921	October 14, 2021
Configuring multiple versions of TensorRT and Tensorflow on HPC share cluster; TF-TRT Warning: Cannot dlopen some TensorRT libraries TensorRT	8	13105	June 28, 2023
Jetson Nano: Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory TensorRT	4	8096	August 3, 2022
Jetson TX2 Tensorrt l4t-tensorflow NGC Segmentation fault at build trt graphconverterV2 Jetson TX2 tensorrt	4	487	May 17, 2023

getPluginCreator could not find plugin BatchedNMS_TRT version 1

Related topics