JetPack 4.4 Jetson Xavier NX can not inference tlt yolov3_resnet18.etlt_b2_gpu0_fp16.engine

load model code:

std::ifstream file(model_path, std::ios::binary);
size_t size{0};
char* model_stream{nullptr};
if (file.good()) {
    file.seekg(0, file.end);
    size = file.tellg();
    file.seekg(0, file.beg);
    model_stream = new char[size];
    assert(model_stream);
    file.read(model_stream, size);
    file.close();
}

runtime = createInferRuntime(gLogger);
assert(runtime != nullptr);
engine = runtime->deserializeCudaEngine(model_stream, size);
assert(engine != nullptr);
sess = new cudaSession(engine->createExecutionContext(), input_blob_name, output_blob_names);
assert(sess != nullptr);

ERROR:
[05/07/2021-15:06:43] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin ResizeNearest_TRT version 1
[05/07/2021-15:06:43] [E] [TRT] safeDeserializationUtils.cpp (323) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plug
[05/07/2021-15:06:43] [E] [TRT] INVALID_STATE: std::exception
[05/07/2021-15:06:43] [E] [TRT] INVALID_CONFIG: Deserialize the cuda engine failed.
cudaNet_yolo_test: cudaNet.cpp:29: cudaNet::cudaNet(const char*, std::__cxx11::string, std::vector<std::__cxx11::basic_string >): Assertion `engine != nullptr’ failed.
Aborted (core dumped)

Hi,

Engine file cannot be used cross different software version.
Please reconvert the engine from TLT on JetPack 4.4 directly.

Thanks.

#It was convert the engine from TLT on JetPack 4.4 directly
#ds-tlt -c pgie_yolov3_tlt_config.txt -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -b 2 -d
#it can convert successed.

INFO: [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: [TRT]: Detected 1 inputs and 4 output network tensors.
0:09:36.958037807 23153 0x5562b75520 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInfer ContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1748> [UID = 1]: serialize cuda engine to file: /home/wangyuanwen/deepstream_tlt_apps-master/models/yolov3/yolov3_resnet18.et lt_b8_gpu0_fp16.engine successfully
INFO: [Implicit Engine Info]: layers num: 5
0 INPUT kFLOAT Input 3x544x960
1 OUTPUT kINT32 BatchedNMS 0
2 OUTPUT kFLOAT BatchedNMS_1 200x4
3 OUTPUT kFLOAT BatchedNMS_2 200
4 OUTPUT kFLOAT BatchedNMS_3 200

0:09:36.995619403 23153 0x5562b75520 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus: [UID 1]: Load new model:/home/wangyu anwen/deepstream_tlt_apps-master/configs/yolov3_tlt/pgie_yolov3_tlt_config.txt sucessfully
Running…

#when check the input or output node name:
import tensorrt as trt

def get_engine(engine_path):
# If a serialized engine exists, use it instead of building an engine.
print(“Reading engine from file {}”.format(engine_path))
with open(engine_path, “rb”) as f, trt.Runtime(TRT_LOGGER) as runtime:
return runtime.deserialize_cuda_engine(f.read())

TRT_LOGGER = trt.Logger()

engine = get_engine("./yolov3_resnet18.etlt_b8_gpu0_fp16.engine")
for binding in engine:
size = trt.volume(engine.get_binding_shape(binding)) * 1
dims = engine.get_binding_shape(binding)
# print(size)
print(dims)
print(binding)
print(“input =”, engine.binding_is_input(binding))
dtype = trt.nptype(engine.get_binding_dtype(binding))
# print("dtype = ", dtype)

#the error:
[TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin ResizeNearest_TRT version 1
[TensorRT] ERROR: safeDeserializationUtils.cpp (323) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
Traceback (most recent call last):
File “trt_input_output_name_check.py”, line 13, in
for binding in engine:
TypeError: ‘NoneType’ object is not iterable

#c++ inference:

std::ifstream file(model_path, std::ios::binary);
size_t size{0};
char* model_stream{nullptr};
if (file.good()) {
    file.seekg(0, file.end);
    size = file.tellg();
    file.seekg(0, file.beg);
    model_stream = new char[size];
    assert(model_stream);
    file.read(model_stream, size);
    file.close();
}

runtime = createInferRuntime(gLogger);
assert(runtime != nullptr);
engine = runtime->deserializeCudaEngine(model_stream, size);
assert(engine != nullptr);
sess = new cudaSession(engine->createExecutionContext(), input_blob_name, output_blob_names);
assert(sess != nullptr);

#ERROR:
[05/07/2021-15:06:43] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin ResizeNearest_TRT version 1
[05/07/2021-15:06:43] [E] [TRT] safeDeserializationUtils.cpp (323) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plug
[05/07/2021-15:06:43] [E] [TRT] INVALID_STATE: std::exception
[05/07/2021-15:06:43] [E] [TRT] INVALID_CONFIG: Deserialize the cuda engine failed.
cudaNet_yolo_test: cudaNet.cpp:29: cudaNet::cudaNet(const char*, std::__cxx11::string, std::vector<std::__cxx11::basic_string >): Assertion `engine != nullptr’ failed.
Aborted (core dumped)

Hi,

You need the ResizeNearest_TRT plugin to inference the model.

The plugin implementation is available at our OSS GitHub:
Please build it from source and replace the libnvinfer_plugin.so plugin library :

Thanks.

*libnvinfer_plugin.so was build, etlt to engine also depend on it.
*

Hi,

Do you replace the libnvinfer_plugin.so.* in /usr/lib/aarch64-linux-gnu/ with the OSS?
If yes, you should have ResizeNearest_TRT already.

Thanks.