MultiAIInferenceOP Error when multiple Outputs per Model (bindings[x] != nullptr )

[faridh]

I am trying to do inference on an onnx segmentation model which takes one images as input and outputs the detection boxes (output0) and the segmentation mask (output1).

model:

Inputs:
name: images
type: float32[1,3,640,640]

Outputs:
name: output0
type: float32[1,70,8400]

name: output1
type: float32[1,32,160,160]

It actually works fine when using my own inference operator written with cupy. So the onnx model and the converted trt model should be correct. When using the holoscan MultiAIInferenceOP however I get an error:

YAML:

inference:  # MultiAIInference
  backend: "trt"
  pre_processor_map: 
    "yolo_segment": ["in_frame"]
  inference_map: 
    "yolo_segment": "out_yolov8_segment"
  in_tensor_names: ["in_frame"]
  out_tensor_names: ["out_yolov8_segment"]
  enable_fp16: false
  input_on_cuda: true
  output_on_cuda: true
  transmit_on_cuda: true

error message:

[2023-05-21 09:54:10.273] [holoscan] [info] [utils.hpp:45] 3: [executionContext.cpp::enqueueInternal::330] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueueInternal::330, condition: bindings[x] != nullptr
)
[2023-05-21 09:54:10.273] [holoscan] [error] [holoinfer_constants.hpp:71]  TRT inference core: Inference failure.
[2023-05-21 09:54:10.273] [holoscan] [error] [holoinfer_constants.hpp:71] Inference manager, Inference failed in core for yolo_segment
[2023-05-21 09:54:10.274] [holoscan] [error] [gxf_wrapper.cpp:68] Exception occurred for operator: 'MultiAIInferenceOp' - Error in Multi AI Inference Codelet, Sub-module->Tick, Inference execution, Message->Error in Multi AI Inference Codelet, Sub-module->Tick, Inference execution, Inference manager, Inference failed in execution for yolo_segment
2023-05-21 09:54:10.274 ERROR gxf/std/entity_executor.cpp@525: Failed to tick codelet MultiAIInferenceOp in entity: MultiAIInferenceOp code: GXF_FAILURE
2023-05-21 09:54:10.274 ERROR gxf/std/entity_executor.cpp@556: Entity [MultiAIInferenceOp] must be in Lifecycle::kStarted or Lifecycle::kIdle stage before stopping. Current state is Ticking
2023-05-21 09:54:10.274 WARN  gxf/std/greedy_scheduler.cpp@235: Error while executing entity 20 named 'MultiAIInferenceOp': GXF_FAILURE
2023-05-21 09:54:10.274 ERROR gxf/std/entity_executor.cpp@556: Entity [MultiAIInferenceOp] must be in Lifecycle::kStarted or Lifecycle::kIdle stage before stopping. Current state is Ticking
2023-05-21 09:54:10.274 INFO  gxf/std/greedy_scheduler.cpp@367: Scheduler finished.
2023-05-21 09:54:10.274 ERROR gxf/std/program.cpp@497: wait failed. Deactivating...
2023-05-21 09:54:10.275 ERROR gxf/core/runtime.cpp@1251: Graph wait failed with error: GXF_FAILURE
2023-05-21 09:54:10.275 PANIC /workspace/holoscan-sdk/src/core/executors/gxf/gxf_executor.cpp@296: GXF operation failed: GXF_FAILURE

Is the config for the MultiAIInferenceOP correct? I did not find out how to declare two outputs for a single model and assign them by their real name (output0, output1).

Setup:
NVIDIA Clara AGX Developer Kit with Holoscan SDK v0.5.0.

[jinl]

Hello, you have accurately found the issue, which is that in v0.5 MultiAIInferenceOp/Holoinfer didn’t support multiple output tensors in the same model. For the next release v0.6, this will be added along with other features in MultiAIInferenceOp/Holoinfer. For now, if you only have one single model to run inference on, you could revert to the previously used TensorRTInferenceOp which should support multiple output tensors in one single model. Please note that TensorRTInferenceOp will be deprecated in the future once MultiAIInferenceOp/Holoinfer has the equivalent features added.

[faridh]

Okay, I see. Then we will for now revert to the TensorRTInferenceOp. Thanks for the help.