Registering Protos in Python Codelet

Hi all,

I created a project, in which I can use Bazel to build Isaac resources externally. Then I created many applications with nodes separated in different packages similar to Isaac’s standard packages. My C++ codelets are working with no problem receiving and transmitting messages.
I am now trying to add a PyCodelet to my application, succeeded to build everything with a py_binary and everything is working fine until the point where I wanted to send and receive protos.
Here is my build:

load("@com_nvidia_isaac//engine/build:isaac.bzl", "isaac_app", "isaac_py_app", "isaac_pkg")

py_binary(
    name = "reident_py",
    srcs = [
        "__init__.py",
        "reident_py.py",
    ],
    data = [
        "yolo_tracker.app.json",
        "@com_nvidia_isaac//messages:detections_proto",
        "@com_nvidia_isaac//packages/deepstream:libdeepstream_module.so",
        "@com_nvidia_isaac//packages/viewers:libviewers_module.so",
        "//packages/demux:libdemux_module.so",
        "//packages/yolo_detector:libyolo_detector_module.so",
        "//packages/tracker:libtracker_module.so",
        # "//packages/reident_py:py_init",
        "coco_config.json",
        "loco_config.json",
        "cocoloco_config.json",
        "@yolov4_coco_model",
        "@yolov4_loco_model",
        "@yolov4_cocoloco_model",
    ],
    deps = ["@com_nvidia_isaac//engine/pyalice"],
    visibility = ["//visibility:public"],
)
isaac_pkg(
    name = "reident_py-pkg",
    srcs = ["reident_py"],
)

When I try to set the protos in the start function as:

class Reidentification(Codelet):
    def start(self):
        # Input and output messages for the Codelet. We'll make connections in the json file.
        self.rx_det = self.isaac_proto_rx("Detections2Proto", "det_results") # Fresh detection results without IDs
        self.rx_track = self.isaac_proto_rx("Detections2Proto", "last_track_results") # Last tracking bboxes
        self.tx = self.isaac_proto_tx("Detections2Proto", "det_r_ids")   # Matched detection results 

I get the following error:

2020-11-07 23:58:57.355 ERROR external/com_nvidia_isaac/engine/alice/components/Codelet.cpp@229: Component 'reident_py/PyCodelet' of type 'isaac::alice::PyCodelet' reported FAILURE:
        (<class 'AssertionError'>, AssertionError('proto message type "Detections2Proto" not registered',), <traceback object at 0x7f090e55ca48>)
    2020-11-07 23:58:57.355 ERROR external/com_nvidia_isaac/engine/alice/backend/event_manager.cpp@42: Stopping node 'reident_py' because it reached status 'FAILURE'

The rest of my app works just fine, but of course without my node with PyCodelet. I tried many other proto types, nothing worked, same error. Does anyone have an idea why my protos cannot be registered?
Thanks!

My running theory is that the Python runtime is not finding the proto because it is in a different directory than expected when using the ISAAC SDK as an external in Bazel if I read your setup correctly. Could you check bazel-bin/messages for the existence of libdtections_proto.a and bazel-bin/<your app>/<your app>.runfiles/com_nvidia_isaac/messages/detections.capnp?

If you have a chance, could you whiddle this down to a simplified use case, say based on //apps/tutorials/ping_python that only tries to load the Detections2Proto? Add a MessageLedger to the components under ping_node and add the line self.rx = self.isaac_proto_rx("Detections2Proto", "det_r_ids") under start(self) somewhere. This did not reproduce any issue for me (there are also tests for messages in Python in //engine/pyalice/tests/pymessage_test.py, but it could be because I am running in the Isaac workspace directly.

Hello hemals,

I confirm that this exists:
bazel-bin/<your_app>/<your_app>.runfiles/ com_nvidia_isaac/messages/detections.capnp

But coming to the libdtections_proto.a: there is no “messages” folder in the bazel-bin of my project. However the original Isaac bazel-bin has it.

I already had a working app in original Isaac workspace, where two PyCodelets were sending ping & pong to each other. I moved it to my external project and got the same error.

Strange thing is that my C++ codelets can send and receive protos in the external project. Do you know what might be missing for Python?

PS. I found libdtections_proto.a inside bazel-bin/external/com_nvidia_isaac/messages folder of my project

I suspect that it is the “external” prefix that is not in the PYTHONPATH of the final output. If you could share your new workspace’s WORKSPACE directory, I might be able to diagnose a bit more. Otherwise, you could try to add “externals” to the imports attribute of your py_binary target and see if adding that in there fixes finding within the nested directory.

Here is the new WORKSPACE:

workspace(name = "logiception")

# Point following dependency to Isaac SDK downloaded from https://developer.nvidia.com/isaac/downloads
local_repository(
    name = "com_nvidia_isaac",
    path = "/home/ga35baq/isaac",
)
load("@com_nvidia_isaac//engine/build:isaac.bzl", "isaac_http_archive", "isaac_git_repository")
load("@com_nvidia_isaac//third_party:packages.bzl", "isaac_packages_workspace")
load("@com_nvidia_isaac//third_party:engine.bzl", "isaac_engine_workspace")
isaac_engine_workspace()

isaac_packages_workspace()

# Loads before boost to override for aarch64 specific config
isaac_http_archive(
    name = "org_lzma_lzma",
    build_file = "@com_nvidia_isaac//third_party:lzma.BUILD",
    licenses = ["@org_lzma_lzma//:COPYING"],
    sha256 = "9717ae363760dedf573dad241420c5fea86256b65bc21d2cf71b2b12f0544f4b",
    strip_prefix = "xz-5.2.4",
    type = "tar.xz",
    url = "https://developer.nvidia.com/isaac/download/third_party/xz-5-2-4-tar-xz",
)

# Loads boost c++ library (https://www.boost.org/) and
# custom bazel build support (https://github.com/nelhage/rules_boost/)
# explicitly due to bazel bug: https://github.com/bazelbuild/bazel/issues/1550
isaac_git_repository(
    name = "com_github_nelhage_rules_boost",
    commit = "9f9fb8b2f0213989247c9d5c0e814a8451d18d7f",
    licenses = ["@com_github_nelhage_rules_boost//:LICENSE"],
    remote = "https://github.com/nelhage/rules_boost.git",
    shallow_since = "1570056263 -0700",
)

load("@com_github_nelhage_rules_boost//:boost/boost.bzl", "boost_deps")

boost_deps()

# Configures toolchain
load("@com_nvidia_isaac//engine/build/toolchain:toolchain.bzl", "toolchain_configure")

toolchain_configure(name = "toolchain")

####################################################################################################

# Logiception specific dependencies

load("//third_party:repositories.bzl", "logiception_workspace")

logiception_workspace()

Thanks, e_lif_kaya! I am able to reproduce the problem with a simplified example “other_workspace” (other_workspace.tar.gz (2.2 KB) ). We will dig into it here and resolve as soon as possible.

Thank you, I look forward to hear from you.

Hello hemals,

may I ask, if this problem is solved in Isaac 2020.2?

No, I’m afraid not. Not everyone was able to reproduce the issue even with the test case and we didn’t have time to diagnose the differences. The test case itself needs to be updated for the paths in 2020.2 as well. I’ll circle back as soon as I have any updates.