# Falure to do inference

**URL:** <https://forums.developer.nvidia.com/t/falure-to-do-inference/197570>\
**Category:** TAO Toolkit\
**Tags:** tensorrt\
**Created:** [December 9, 2021, 8:18am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570 "2021-12-09T08:18:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![moche.chan](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@moche.chan](https://forums.developer.nvidia.com/u/moche.chan)\
**Post date:** [December 9, 2021, 8:18am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/1 "2021-12-09T08:18:03Z")

</div>

## Description

The attached c++ program is for LPRNET and it produces a failure when doing a inference.  
How to produce a good inferred result?

## Environment

**TensorRT Version** : 7.1.3.0  
**GPU Type** : jetson xavier nx  
**Nvidia Driver Version** : jetpack 4.5.1  
**CUDA Version** : 10.2.89  
**CUDNN Version** : 8.0.0.180  
**Operating System + Version** : ubuntu 18.04  
**Python Version (if applicable)**:  
**TensorFlow Version (if applicable)**:  
**PyTorch Version (if applicable)**:  
**Baremetal or Container (if container which image + tag)**:

## Relevant Files

![car1](https://global.discourse-cdn.com/nvidia/original/3X/3/8/38bda3ba2938cfbf95e2472c094a91e3a62a3255.jpeg)  
[nx4.5us\_lprnet\_baseline18\_deployable.etlt\_b16\_gpu0\_fp16.engine](https://forums.developer.nvidia.com/404) (31.6 MB)  
[trt\_lprnet.cpp](https://forums.developer.nvidia.com/uploads/short-url/ypycStKUl5XaqhrY90mHD8c8z0H.cpp) (5.6 KB)

The attached engine file is produced by the following commands:

```auto
wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/lprnet/versions/deployable_v1.0/files/us_lprnet_baseline18_deployable.etlt
./tao-converter -k nvidia_tlt -p image_input,1x3x48x96,4x3x48x96,16x3x48x96 us_lprnet_baseline18_deployable.etlt -t fp16 -e lpr_us_onnx_b16.engine

```

## Steps To Reproduce

Please include:

- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered

The file can be compiled/run by using:

```auto
g++ trt_lprnet.cpp -lnvinfer -Lcudart -pthread $(pkg-config --cflags --libs opencv4) -I/usr/local/cuda/include -L/usr/local/cuda/lib64 -lcuda -lcudart -O0 -p -g 
./a.out 

```

Error messages:

```auto
a.out: trt_lprnet.cpp:62: void doInference(nvinfer1::IExecutionContext&, float*, float*, int): Assertion `engine.getNbBindings() == 2' failed.
Aborted (core dumped)

```

---

<div class="post-metadata">

**Author:** ![NVES](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nves/32/14043_2.png) [@NVES](https://forums.developer.nvidia.com/u/NVES)\
**Post date:** [December 9, 2021, 5:11pm UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/2 "2021-12-09T17:11:18Z")

</div>

Hi,  
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist  
[https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec](https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec)

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

> <https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md>

Also, request you to share your model and script if not shared already so that we can help you better.

Meanwhile, for some common errors and queries please refer to below link:

> **[Developer Guide :: NVIDIA Deep Learning TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/#error-messaging)**
>
> This is the revision history of the NVIDIA TensorRT 8.4 Developer Guide.

> **[Developer Guide :: NVIDIA Deep Learning TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/#faq)**
>
> This is the revision history of the NVIDIA TensorRT 8.4 Developer Guide.

Thanks!

---

<div class="post-metadata">

**Author:** ![moche.chan](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@moche.chan](https://forums.developer.nvidia.com/u/moche.chan)\
**Post date:** [December 13, 2021, 4:00am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/4 "2021-12-13T04:00:03Z")

</div>

Unfortunately, I can not compile the mentioned trtexec successfully so far.  
The attached model should be good because the model can work for the python code.  
[lprnet\_simplest.py](https://forums.developer.nvidia.com/uploads/short-url/k9IrxfMdedbyzkxOwKWovO9peBM.py) (3.4 KB)

The identical model works for the attached python code, but does not work for the attached c++ code. Therefore, I just want to find out a workable c++ code.

---

<div class="post-metadata">

**Author:** ![spolisetty](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/spolisetty/32/14047_2.png) [@spolisetty](https://forums.developer.nvidia.com/u/spolisetty)\
**Post date:** [December 23, 2021, 9:16am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/5 "2021-12-23T09:16:47Z")

</div>

Hi,

Looks like you’re using TAO toolkit LPRNET. We are moving this post to TAO forum to get better help.

Thank you.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 24, 2021, 2:27am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/6 "2021-12-24T02:27:52Z")

</div>

@moche.chan  
I suggest you to take a look at [GitHub - NVIDIA-AI-IOT/deepstream\_lpr\_app: Sample app code for LPR deployment on DeepStream](https://github.com/NVIDIA-AI-IOT/deepstream_lpr_app) for running inference.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 24, 2021, 2:50pm UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/7 "2021-12-24T14:50:37Z")

</div>

More, please note that there are two outputs for lprnet. See below

> $ python -m pip install colored  
> $ python -m pip install polygraphy --index-url [https://pypi.ngc.nvidia.com](https://pypi.ngc.nvidia.com/)  
> $ polygraphy inspect model model.plan  
> [I] Loading bytes from /workspace/model.plan  
> [I] ==== TensorRT Engine ====  
> Name: Unnamed Network 0 | Explicit Batch Engine (35 layers)
> 
> ---- 1 Engine Input(s) ----  
> {image\_input [dtype=float32, shape=(-1, 3, 48, 96)]}
> 
> ---- 2 Engine Output(s) ----  
> {tf\_op\_layer\_ArgMax [dtype=int32, shape=(-1, 24)],  
> tf\_op\_layer\_Max [dtype=float32, shape=(-1, 24)]}
> 
> ---- Memory ----  
> Device Memory: 65249280 bytes
> 
> ---- 1 Profile(s) (3 Binding(s) Each) ----
> 
> - Profile: 0  
> Binding Index: 0 (Input) [Name: image\_input] | Shapes: min=(1, 3, 48, 96), opt=(4, 3, 48, 96), max=(16, 3, 48, 96)  
> Binding Index: 1 (Output) [Name: tf\_op\_layer\_ArgMax] | Shape: (-1, 24)  
> Binding Index: 2 (Output) [Name: tf\_op\_layer\_Max] | Shape: (-1, 24)

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 27, 2021, 3:29pm UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/8 "2021-12-27T15:29:02Z")

</div>

I modify something based on your code.  
It can work. Please check.

```auto
#include <string>
#include <future>
#include <deque>
#include <fstream>
#include <iostream>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>
#include <mutex>
#include <stdio.h>
#include <cassert>
 
#include <opencv2/opencv.hpp>
 
#include "cuda_runtime_api.h"
#include <cuda.h>
#include "NvInfer.h"
 
 
#define DEVICE 0 // GPU id
#define BATCH_SIZE 1
 
static const int INPUT_H = 48;
static const int INPUT_W = 96;
static const int OUTPUT_SIZE = 24;
 
const char *INPUT_BLOB_NAME = "image_input";
const char *OUTPUT_BLOB_NAME_1 = "tf_op_layer_ArgMax";
const char *OUTPUT_BLOB_NAME_2 = "tf_op_layer_Max";
 
const std::string alphabet[] = {
    "0", "1", "2", "3", "4", "5", "6", "7", "8", "9",
    "A", "B", "C", "D", "E", "F", "G", "H", "I", "J",
    "K", "L", "M", "N", "P", "Q", "R", "S", "T", "U",
    "V", "W", "X", "Y", "Z"
};
 
 
#define CHECK(status) \
    do\
    {\
        auto ret = (status);\
        if (ret != 0)\
        {\
            std::cerr << "Cuda failure: " << ret << std::endl;\
            abort();\
        }\
    } while (0)
 
class Logger : public nvinfer1::ILogger {
    void log(Severity severity, const char* msg) override {
        if (severity <= Severity::kWARNING){
            std::cout << msg << std::endl;
        }
    }
} logger;
 
 
void doInference(nvinfer1::IExecutionContext &context, float *input, int *output_1, float *output_2, int batchSize) {
    const nvinfer1::ICudaEngine &engine = context.getEngine();
 
    // Pointers to input and output device buffers to pass to engine.
    // Engine requires exactly IEngine::getNbBindings() number of buffers.
    assert(engine.getNbBindings() == 3);
    void *buffers[3];
 
    // In order to bind the buffers, we need to know the names of the input and output tensors.
    // Note that indices are guaranteed to be less than IEngine::getNbBindings()
    const int inputIndex = engine.getBindingIndex(INPUT_BLOB_NAME);
    const int outputIndex_1 = engine.getBindingIndex(OUTPUT_BLOB_NAME_1);
    const int outputIndex_2 = engine.getBindingIndex(OUTPUT_BLOB_NAME_2);
 
    // Create GPU buffers on device
    CHECK(cudaMalloc(&buffers[inputIndex], batchSize * 3 * INPUT_H * INPUT_W * sizeof(float)));
    CHECK(cudaMalloc(&buffers[outputIndex_1], batchSize * OUTPUT_SIZE * sizeof(float)));
    CHECK(cudaMalloc(&buffers[outputIndex_2], batchSize * OUTPUT_SIZE * sizeof(float)));
 
    // Create stream
    cudaStream_t stream;
    CHECK(cudaStreamCreate(&stream));
 
    // DMA input batch data to device, infer on the batch asynchronously, and DMA output back to host
    CHECK(cudaMemcpyAsync(buffers[inputIndex], input, batchSize * 3 * INPUT_H * INPUT_W * sizeof(float), cudaMemcpyHostToDevice, stream));
    context.enqueue(batchSize, buffers, stream, nullptr);
    CHECK(cudaMemcpyAsync(output_1, buffers[outputIndex_1], batchSize * OUTPUT_SIZE * sizeof(float), cudaMemcpyDeviceToHost, stream));
    CHECK(cudaMemcpyAsync(output_2, buffers[outputIndex_2], batchSize * OUTPUT_SIZE * sizeof(float), cudaMemcpyDeviceToHost, stream));
    cudaStreamSynchronize(stream);
 
    // Release stream and buffers
    cudaStreamDestroy(stream);
    CHECK(cudaFree(buffers[inputIndex]));
    CHECK(cudaFree(buffers[outputIndex_1]));
    CHECK(cudaFree(buffers[outputIndex_2]));
}
 
int main(int argc, char *argv[]){
    cudaSetDevice(DEVICE);
    char *trtModelStream{nullptr};
    size_t size{0};
 
    std::ifstream file("/workspace/demo_2.0/lprnet/lpr_us_onnx_b16.engine", std::ios::binary);
    if (file.good()) {
        file.seekg(0, file.end);
        size = file.tellg();
        file.seekg(0, file.beg);
        trtModelStream = new char[size];
        assert(trtModelStream);
        file.read(trtModelStream, size);
        file.close();
    }
    std::cout << "size:" << size <<"\n";
 
    nvinfer1::IRuntime* runtime = nvinfer1::createInferRuntime(logger);
    assert(runtime != nullptr);
 
    nvinfer1::ICudaEngine* engine = runtime->deserializeCudaEngine(trtModelStream, size);
    assert(engine != nullptr);
 
    nvinfer1::IExecutionContext *context = engine->createExecutionContext();
    assert(context != nullptr);
 
    static float data[BATCH_SIZE * 3 * INPUT_H * INPUT_W];
 
    cv::Mat img = cv::imread("/workspace/demo_2.0/lprnet/data/openalpr/train/image/wts-lg-000158.jpg");
    cv::Mat pr_img;
    cv::resize(img, pr_img, cv::Size(INPUT_W, INPUT_H), 0, 0, cv::INTER_CUBIC);
 
    int i = 0;
    for (int row = 0; row < INPUT_H; ++row) {
        uchar* uc_pixel = pr_img.data + row * pr_img.step;
        for (int col = 0; col < INPUT_W; ++col) {
            data[i + 2 * INPUT_H * INPUT_W] = ((float)uc_pixel[2] - 127.5)*0.003921568627451;
            data[i + INPUT_H * INPUT_W] = ((float)uc_pixel[1]-127.5)*0.003921568627451;
            data[i] = ((float)uc_pixel[0]-127.5)*0.003921568627451;
            uc_pixel += 3;
            ++i;
        }
    }
 
    // Run inference
    static int tf_op_layer_ArgMax[BATCH_SIZE * OUTPUT_SIZE];
    static float tf_op_layer_Max[BATCH_SIZE * OUTPUT_SIZE];
    auto start = std::chrono::system_clock::now();
    printf("running inference \n");
    doInference(*context, data, tf_op_layer_ArgMax, tf_op_layer_Max, BATCH_SIZE);
    auto end = std::chrono::system_clock::now();
    std::cout << std::chrono::duration_cast<std::chrono::microseconds>(end - start).count() << "us" << std::endl;
    std::cout << std::endl;

    std::vector<float> preds;
    for (int i = 0; i < 24; ++i) {
        preds.push_back(tf_op_layer_ArgMax[i]);
    }
 
    // remove repeat blank label
    int pre_c = preds[0];
 
    std::vector<int> no_repeat_blank_label;
    for (auto c: preds) {
        if (c == pre_c || c == 35) {
            if (c == 35) pre_c = c;
            continue;
        }
        no_repeat_blank_label.push_back(c);
        pre_c = c;
    }
 
    //print the character list
    std::string str;
    for (auto v: no_repeat_blank_label) {
        str += alphabet[v];
    }
 
    std::cout<<"result:"<<str<<std::endl;
 
    // Destroy the engine
    context->destroy();
    engine->destroy();
    runtime->destroy();
 
    return 0;
}
 
/*
time g++ trt_lprnet.cpp -lnvinfer -Lcudart -pthread $(pkg-config --cflags --libs opencv4) -I/usr/local/cuda/include -L/usr/local/cuda/lib64 -lcuda -lcudart -O0 -p -g && time ./a.out
 
nvcc -I/usr/local/cuda/include -L/usr/local/cuda/lib64 -lcutil -lcudpp -lcuda -lcudart -c -o FSPB_main.o FSPB_main.cpp
*/

```

---

<div class="post-metadata">

**Author:** ![moche.chan](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@moche.chan](https://forums.developer.nvidia.com/u/moche.chan)\
**Post date:** [December 28, 2021, 5:07am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/9 "2021-12-28T05:07:30Z")

</div>

By running your codes, I cannot get license plate number, and outputs of tf\_op\_layer\_ArgMax containing all zeros. Can you share your workable model file and the testing image? Thanks a lot.

By the way, I got the following error message:

```auto
Parameter check failed at: engine.cpp::resolveSlots::1227, condition: allInputDimensionsSpecified(routine)

```

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 28, 2021, 11:54am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/10 "2021-12-28T11:54:59Z")

</div>

Step:

```auto
$ tao lprnet run /bin/bash
# wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/lprnet/versions/deployable_v1.0/files/us_lprnet_baseline18_deployable.etlt
# tao-converter -k nvidia_tlt -p image_input,1x3x48x96,1x3x48x96,1x3x48x96 us_lprnet_baseline18_deployable.etlt -t fp16 -e lpr_us_onnx_b16.engine
# time g++ trt_lprnet.cpp -lnvinfer -Lcudart -pthread $(pkg-config --cflags --libs opencv) -I/usr/local/cuda/include -L/usr/local/cuda/lib64 -lcuda -lcudart -O0 -p -g && time ./a.out

```

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [January 11, 2022, 11:55am UTC](https://forums.developer.nvidia.com/t/falure-to-do-inference/197570/11 "2022-01-11T11:55:27Z")

</div>

Closed on January 11, 2022, 11:55am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.