april tag image [ColorCameraProto] input message from zed camera issue. (ZED camera tx discrepancy)

Autonomous Machines Robotics - Isaac Isaac SDK
1

Hello Everyone,

Currently I am attempting to feed an image coming from the Zed camera into the isaac.perception.AprilTagsDetection component. I have verified the functinality of April tag detection of tags with V4L2 prior to swithcing to the Zed image feed.

The issue I am running into is a discrepancy in the zed camera. in the documentation and in the carter example to connect the output of zed to april tags, the isaac.ZedCamera outputs the following:
left_camera_rgb : left rgb image and camera intrinsics
right_camera_rgb : right rgb image and camera intrinsics
left_camera_gray : left gray image and camera intrinsics
right_camera_gray : right gray rgb image and camera intrinsics
extrinsics [Pose3dProto]: camera pair extrinsics (right-to-left)

In the sample for the Zed camera, the camera is actually outputing these protos (also verified in sight):
left_rgb (proto unknown, I’m assuming ColorCameraProto)
left_thumbnail (proto unknown, I’m assuming ColorCameraProto)
right_thumbnail (proto unknown, I’m assuming ColorCameraProto)

going further, I tried to use the proto names from the documentation (left_camera_rgb and right_camera_rgb) and there is no data stream (verified in sight).

to bypas this i attempted to feed the isaac.perception.AprilTagsDetection component with the image stream left_rgb that i was able to view from the zed in sight, but the same april tage i verified with V4L2 prior to the test are not recognized. This leads me to beleive that left_rgb is not a ColorCameraProto.

so I have a few questions:

Has anyone read april tags with the zed camera feed? If so, how did you set up you app?

where do I find the component that producuces the tx proto steams as described in the documentation? I have looked through the entire file system and everything leds me back to //packages/zed but isaac::zed doesnt seem to be using this package because this package transmits left_camera_rgb and right_camera_rgb and currently isaac::zed is transmitting left_rgb, left_thumbnail, right_thumbnail in my samples/zed

where am I getting this zed component that is producing these strange tx’s?

What package is isaac:zed referencing?

Thank you for the help in advance!

2

Hey FedoraLabs,

To use a Zed camera with april tag detection, you need to connect either the ZedCamera left_camera_rgb or right_camera_rgb to the AprilTagDetection tag_fill and tag_boundary.

The following worked for me:

april_tags.app.json

{
  "name": "april_tags",
  "modules": [
    "//packages/perception:april_tags",
    "zed",
    "viewers"
  ],
  "graph": {
    "nodes": [
      {
        "name": "input_images",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.ZedCamera",
            "type": "isaac::ZedCamera"
          }
        ]
      },
      {
        "name": "april_tags_detection",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.perception.AprilTagsDetection",
            "type": "isaac::perception::AprilTagsDetection"
          }
        ]
      },
      {
        "name": "image_viewers",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.viewers.ColorCameraViewer",
            "type": "isaac::viewers::ColorCameraViewer"
          }
        ]
      }
    ],
    "edges": [
      {
        "source": "input_images/isaac.ZedCamera/left_camera_rgb",
        "target": "april_tags_detection/isaac.perception.AprilTagsDetection/image"
      },
      {
        "source": "input_images/isaac.ZedCamera/left_camera_rgb",
        "target": "image_viewers/isaac.viewers.ColorCameraViewer/color_listener"
      }
    ]
  },
  "config": {
    "april_tags_detection": {
      "isaac.perception.AprilTagsDetection": {
        "max_tags": 50
      }
    },
    "input_images": {
      "isaac.ZedCamera": {
        "resolution": "1280x720",
        "tick_period": "60Hz"
      }
    },
    "websight": {
      "WebsightServer": {
        "webroot": "packages/sight/webroot",
        "assetroot": "../isaac_assets",
        "port": 3000,
        "ui_config": {
          "windows": {
            "Tags": {
              "renderer": "2d",
              "channels": [
                {
                  "name": "april_tags/image_viewers/isaac.viewers.ColorCameraViewer/Color"
                },
                {
                  "name": "april_tags/april_tags_detection/isaac.perception.AprilTagsDetection/tag_fill"
                },
                {
                  "name": "april_tags/april_tags_detection/isaac.perception.AprilTagsDetection/tag_boundary"
                }
              ]
            }
          }
        }
      }
    }
  }
}

BUILD

load("//engine/build:isaac.bzl", "isaac_app")

isaac_app(
    name = "april_tags",
    modules = [
        "//packages/perception:april_tags",
        "zed",
        "viewers",
    ],
)

The left_rgb, left_thumbnail, and right_thumbnail are from show(…) method calls. You can see the origins in packages/zed/ZedCamera.cpp, on lines 221, 223 and 228.

4

does it work with the newest release of IsaaC?