ORB SLAM codelet with Realsense D435

Autonomous Machines Isaac Isaac SDK + Isaac Sim
#1

Hello!

I’m experimenting with Visual odometry and V-SLAM on Jetson Nano with Realsense cameras.
I was glad to see that the stereo Visual odometry GEM had codelets for usage with the realsense D435 in addition to the ZED camera.

However, the ORB GEM has a codelet only for the ZED camera. Is a realsense D435 codelet expected to be out soon?

1 Like
#2

Hi,

We will have support of RealSense D435 in our next release

#3

Thanks for the response @atorabi

Is there an approximate timeline as to when the next release would be out?

#4

Hi Arjun,
I’m working on a similar project and was wondering if you have made any progress on this that you might be willing to share?

#5

Hey @davis

I haven’t started any work on this yet. Planning to revisit this after other tasks (was hoping we’d have an update on the new Isaac version bu then).

Best regards,
Arjun.

#6

Hey @atorabi has it been made available in the 2020.2 release? I couldn’t find it anywhere in the codebase.

#7

The Realsense camera codelet is in (2020.2) //sdk/packages/realsense and should support the D435. It should be possible to reconfigure the Orb Demo app (//sdk/packages/orb/apps/orb_demo) to use the RealSense codelet instead of the Zed codelet as a source. ExtractVisualizeOrb codelet is taking monocular RGB frames (arbitrarily the left camera on a ZED in the example), so you can connect an edge to the Realsense color endpoint instead.

#8

I’m getting this error when I changed to realsense configuration.

2021-01-12 17:59:57.905 ERROR external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@295: packages/realsense/librealsense_module.so: /home/arjun/.cache/bazel/_bazel_arjun/e69b69c2c266e89925be19b53554157a/execroot/com_nvidia_isaac_sdk/bazel-out/k8-opt/bin/packages/orb/apps/orb_demo/orb_demo.runfiles/com_nvidia_isaac_sdk//packages/realsense/librealsense_module.so: cannot open shared object file: No such file or directory
2021-01-12 17:59:57.905 PANIC external/com_nvidia_isaac_engine/engine/alice/backend/modules.cpp@297: Could not load all required modules for application

| Isaac application terminated unexpectedly |

#9

That error indicates that you are missing the RealSense shared libraries. Did you use the deploy.sh script here or rebuild after updating the app graph by any chance? Bazel should have fetched the correct libraries and put them in place for you if you’ve run install_dependencies.sh correctly already.

#10

@hemalsI installed install_dependencies.sh correctly . Sample realsense application is working. For the ORB demo, only zed codelet was available, I changed the parameter to read from realsense camera.

#11

Did you happen to add “realsense” to the list of modules in //sdk/packages/orb/apps/orb_demo/BUILD under the “orb_demo” isaac_app target and at the top of modules list in orb_demo.app.json? That might explain why the Realsense module is not getting included in the orb_demo app even though you modified the app JSON graph.