Inserting custom model into Issac ROS humble nitros

Hey :)
I successfully implemented my own unet model with image segmentation by changing a few arguments in /isaac_ros_image_segmentation repo and converting my own model.plan.

Now I’m trying to insert my own model trained with yolo v4 into the isaac_ros_object_detection repo
Mainly I want to change these layers :
OUTPUT_LAYERS=“output_cov/Sigmoid,output_bbox/BiasAdd”
into my own BatchedNMS layer.
is it possible to decode it in the same manner as the node decodes the original layers?

Another question:
in my try to insert a custom model I got these errors:
[component_container-1] E1116 13:19:44.344062 34428 model_repository_manager.cc:1355] failed to load ‘yolov4_resnet18_epoch_200’ version 1: Invalid argument: unexpected datatype TYPE_FP32 for inference input ‘Input’, expecting TYPE_FP16 for yolov4_resnet18_epoch_200
[component_container-1] ERROR: Triton: failed to load model yolov4_resnet18_epoch_200, triton_err_str:Invalid argument, err_msg:load failed for model ‘yolov4_resnet18_epoch_200’: version 1: Invalid argument: unexpected datatype TYPE_FP32 for inference input ‘Input’, expecting TYPE_FP16 for yolov4_resnet18_epoch_200;

thats weird, the datatype TYPE_FP32 is not mentioned at all in my config.ptxt:

name: "yolov4_resnet18_epoch_200"
platform: "tensorrt_plan"
max_batch_size: 16
input [
  {
    name: "Input"
    data_type: TYPE_FP16
    format: FORMAT_NCHW
    dims: [ 3, 384, 1248 ]
  }
]
output [
  {
        name: "BatchedNMS"
        data_type: TYPE_FP16
        dims: [ 3, 384, 1248]
    }
]
dynamic_batching { }
version_policy: {
  specific {
    versions: [ 1 ]
  }
}
1 Like

The isaac_ros_detectnet package expects the schema of a DetectNetV2 model which does not match with YOLOv4’s layout. You would have to write your own custom “decoder” (model inference output interpreter) for YOLOv4 to make this work. Sign up for the Isaac ROS webinar on integrating YOLOv5 to learn how to do exactly that.