Hi all,
Let me describe my situation.
Now I already loaded an engine and do infer function (this infer definition is going to do_inference.)
Copy from TensorRT which provided the code.
infer definition
def infer(self, image_path):
# Load image into CPU
img = self._load_img(image_path)
# Copy it into appropriate place into memory
# (self.inputs was returned earlier by allocate_buffers())
np.copyto(self.inputs[0].host, img.ravel())
# Fetch output from the model
output = do_inference(
self.context, bindings=self.bindings, inputs=self.inputs,
outputs=self.outputs, stream=self.stream)
return output
Do_inference definition
def do_inference(context, bindings, inputs, outputs, stream, batch_size=1):
# Transfer input data to the GPU.
[cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
# Run inference.
context.execute_async(batch_size=batch_size, bindings=bindings, stream_handle=stream.handle)
# Transfer predictions back from the GPU.
[cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs]
# Synchronize the stream
stream.synchronize()
# Return only the host outputs.
return [out.host for out in outputs]
After process the do_inference, I can get an output which is a big array.
For example, there are two variables in the uff_ssd sample.
# Fetch output from the model
[detection_out, keepCount_out] = common.do_inference(
self.context, bindings=self.bindings, inputs=self.inputs,
outputs=self.outputs, stream=self.stream)
And we can use both variables (i.e., detection_out and keepCount_out) to do somethings like below. (Copy from detect_objects.py)
img_pil = Image.open(args.input_img_path)
prediction_fields = len(TRT_PREDICTION_LAYOUT)
for det in range(int(keep_count_out[0])):
analyze_prediction(detection_out, det * prediction_fields, img_pil)
In my case, there were 26 lists after we did do_inference() for my model.
However, the output order is totally different with original onnx model.
So I cannot identify that the list of each output is which one in original onnx model.
Let me show the output order.
Origianl onnx model
pathmodel = os.path.join('embedmask_simplifier.onnx')
model = onnx.load(pathmodel)
onnx.checker.check_model(model)
sess = rt.InferenceSession(pathmodel)
input1 = sess.get_inputs()[0].name
outputs = sess.run([], {input1: img_preprocess})
Output the each of outputs
shape
0 (1, 1, 100, 144)
1 (1, 1, 50, 72)
2 (1, 1, 25, 36)
3 (1, 1, 13, 18)
4 (1, 1, 7, 9)
5 (1, 4, 100, 144)
6 (1, 4, 50, 72)
7 (1, 4, 25, 36)
8 (1, 4, 13, 18)
9 (1, 4, 7, 9)
10 (1, 1, 100, 144)
11 (1, 1, 50, 72)
12 (1, 1, 25, 36)
13 (1, 1, 13, 18)
14 (1, 1, 7, 9)
15 (1, 32, 100, 144)
16 (1, 32, 50, 72)
17 (1, 32, 25, 36)
18 (1, 32, 13, 18)
19 (1, 32, 7, 9)
20 (1, 1, 100, 144)
21 (1, 1, 50, 72)
22 (1, 1, 25, 36)
23 (1, 1, 13, 18)
24 (1, 1, 7, 9)
25 (1, 32, 100, 144)
Let’s count all of numbers for each shape.
Output the total numbers of each outputs
shape
0 14400
1 3600
2 900
3 234
4 63
5 57600
6 14400
7 3600
8 936
9 252
10 14400
11 3600
12 900
13 234
14 63
15 460800
16 115200
17 28800
18 7488
19 2016
20 14400
21 3600
22 900
23 234
24 63
25 460800
Output from the infer definition by the TRT engine
To specific,
output = do_inference(
self.context, bindings=self.bindings, inputs=self.inputs,
outputs=self.outputs, stream=self.stream)
The results are 1 dimension.
0 (14400,)
1 (14400,)
2 (57600,)
3 (460800,)
4 (14400,)
5 (3600,)
6 (3600,)
7 (14400,)
8 (115200,)
9 (3600,)
10 (900,)
11 (900,)
12 (3600,)
13 (28800,)
14 (900,)
15 (234,)
16 (234,)
17 (936,)
18 (7488,)
19 (234,)
20 (63,)
21 (63,)
22 (252,)
23 (2016,)
24 (63,)
25 (460800,)
We can see that the output order is not same. But the total numbers are same. (I already compared between them.)
However, I don’t know which one of each variable of output is original variable of onnx output so that I cannot use them.
So I wanna confirm the all of output very specific. Is there any method? Or how did you know each one output meaning after do_inference?
Thanks a lot!!