After load an engine and do inference, how can know the meaning of each output?

Hi all,

Let me describe my situation.
Now I already loaded an engine and do infer function (this infer definition is going to do_inference.)

Copy from TensorRT which provided the code.
infer definition

    def infer(self, image_path):
        # Load image into CPU
        img = self._load_img(image_path)
        # Copy it into appropriate place into memory
        # (self.inputs was returned earlier by allocate_buffers())
        np.copyto(self.inputs[0].host, img.ravel())

        # Fetch output from the model
        output = do_inference(
            self.context, bindings=self.bindings, inputs=self.inputs,
            outputs=self.outputs, stream=self.stream)
        return output

Do_inference definition

def do_inference(context, bindings, inputs, outputs, stream, batch_size=1):
    # Transfer input data to the GPU.
    [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
    # Run inference.
    context.execute_async(batch_size=batch_size, bindings=bindings, stream_handle=stream.handle)
    # Transfer predictions back from the GPU.
    [cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs]
    # Synchronize the stream
    stream.synchronize()
    # Return only the host outputs.

    return [out.host for out in outputs]

After process the do_inference, I can get an output which is a big array.
For example, there are two variables in the uff_ssd sample.

        # Fetch output from the model
        [detection_out, keepCount_out] = common.do_inference(
            self.context, bindings=self.bindings, inputs=self.inputs,
            outputs=self.outputs, stream=self.stream)

And we can use both variables (i.e., detection_out and keepCount_out) to do somethings like below. (Copy from detect_objects.py)

    img_pil = Image.open(args.input_img_path)
    prediction_fields = len(TRT_PREDICTION_LAYOUT)
    for det in range(int(keep_count_out[0])):
        analyze_prediction(detection_out, det * prediction_fields, img_pil)

In my case, there were 26 lists after we did do_inference() for my model.
However, the output order is totally different with original onnx model.
So I cannot identify that the list of each output is which one in original onnx model.

Let me show the output order.

Origianl onnx model

    pathmodel = os.path.join('embedmask_simplifier.onnx')

    model = onnx.load(pathmodel)
    onnx.checker.check_model(model)

    sess = rt.InferenceSession(pathmodel)
    input1 = sess.get_inputs()[0].name
    outputs = sess.run([], {input1: img_preprocess})

Output the each of outputs shape

0 (1, 1, 100, 144)
1 (1, 1, 50, 72)
2 (1, 1, 25, 36)
3 (1, 1, 13, 18)
4 (1, 1, 7, 9)
5 (1, 4, 100, 144)
6 (1, 4, 50, 72)
7 (1, 4, 25, 36)
8 (1, 4, 13, 18)
9 (1, 4, 7, 9)
10 (1, 1, 100, 144)
11 (1, 1, 50, 72)
12 (1, 1, 25, 36)
13 (1, 1, 13, 18)
14 (1, 1, 7, 9)
15 (1, 32, 100, 144)
16 (1, 32, 50, 72)
17 (1, 32, 25, 36)
18 (1, 32, 13, 18)
19 (1, 32, 7, 9)
20 (1, 1, 100, 144)
21 (1, 1, 50, 72)
22 (1, 1, 25, 36)
23 (1, 1, 13, 18)
24 (1, 1, 7, 9)
25 (1, 32, 100, 144)

Let’s count all of numbers for each shape.
Output the total numbers of each outputs shape

0 14400
1 3600
2 900
3 234
4 63
5 57600
6 14400
7 3600
8 936
9 252
10 14400
11 3600
12 900
13 234
14 63
15 460800
16 115200
17 28800
18 7488
19 2016
20 14400
21 3600
22 900
23 234
24 63
25 460800

Output from the infer definition by the TRT engine

To specific,

        output = do_inference(
            self.context, bindings=self.bindings, inputs=self.inputs,
            outputs=self.outputs, stream=self.stream)

The results are 1 dimension.

0 (14400,)
1 (14400,)
2 (57600,)
3 (460800,)
4 (14400,)
5 (3600,)
6 (3600,)
7 (14400,)
8 (115200,)
9 (3600,)
10 (900,)
11 (900,)
12 (3600,)
13 (28800,)
14 (900,)
15 (234,)
16 (234,)
17 (936,)
18 (7488,)
19 (234,)
20 (63,)
21 (63,)
22 (252,)
23 (2016,)
24 (63,)
25 (460800,)

We can see that the output order is not same. But the total numbers are same. (I already compared between them.)
However, I don’t know which one of each variable of output is original variable of onnx output so that I cannot use them.
So I wanna confirm the all of output very specific. Is there any method? Or how did you know each one output meaning after do_inference?

Thanks a lot!!

o Linux distro ; Ubuntu 18.04
o GPU type : 1060
o Nvidia driver version : 440
o CUDA version : 10.0
o CUDNN version : 7.6.5
o Python version [if using python] : 3.6.9
o Tensorflow and PyTorch version : TF 1.14
o TensorRT version : 7.0.0.11

Hi,

Could you please share the repro script and model file so we can help better?

Thanks

Hi SunilJB,

I am sorry about that I don’t be allowed to provide the model. However, I think that I can ask some relevant questions, and probably it can help me to solve the problem or check where the problem is!

For example,
The sample is yolov3 of python case.
I tried to output the yolov3.onnx model by the code of previously what I mentioned, and the output shape is

0 (64, 255, 19, 19)
1 (64, 255, 38, 38)
2 (64, 255, 76, 76)

After it converted to trt engine, and then load it and use it, we can get the output which is

0 (92055,)
1 (368220,)
2 (1472880,)

as the builder.max_batch_size = 1, we can know the 1472880 = 255*76*76.
Also, we can notice that the output order seems like correct with original model.

So my question is :
Could I suppose that the output order of TRT engine should be same with original onnx model?

Thank you.

Hi,
Usually the output order should be same between onnx and trt model, but it might be different in some combination.
Best approach will be, just get the output name and match name instead. I think, the order cannot be controlled.

Thanks

1 Like

Thank you for your help.
I will keep trying on here.

Hi SunilJB,
Thank you for your help on the issue. However, the new issue is, we cannot find a way to get the name of each output.

The only lead that we could get so far is the output shape/value, which leads to my colleague @Chieh to issue this question. Could I know how to get the output name?

And also, even the output value is not quite the same as in the tutorial shown by tensorrt (the difference mean is 1 ~ >100.

*FYI:

  • We use onnx2trt to convert .onnx model to .trt successfully, whereas using both trtexec and onnx parser of tensorRT will lead to error:

ERROR: /home/chieh/github/TensorRT/parsers/onnx/ModelImporter.cpp:92 In function parseGraph:
[8] Assertion failed: convertOnnxWeights(initializer, &weights, ctx)
[03/20/2020-09:27:48] [E] Failed to parse onnx file
[03/20/2020-09:27:48] [E] Parsing model failed
[03/20/2020-09:27:48] [E] Engine creation failed
[03/20/2020-09:27:48] [E] Engine set up failed

I already found it.
After we decided the layer names, I printed the “bindings” then we can get the output layer name. :)
Thank you.