How to use nvblox with Luxonis OAK-D Pro PoE?

Robotics & Edge Computing Robotics - Isaac Isaac ROS
,
1

Hello, I have been trying to test out nvblox with the Luxonis OAK-D Pro Wide PoE stereo camera. As I am working with ROS-based systems I was able to run this tutorial using Isaac ROS and nvblox with the provided bag data.

However, when I replace the input topics from the bag with data coming from my camera, nvblox does not seem to work or provide any errors/warnings suggesting what is broken or missing. I have matched my camera outputs to be of the same encoding as the example data, and published TF messages to give the camera a pose.

I can see that my camera is not one of the officially supported ones, can anyone please provide insight into what might be missing that is causing nvblox to not work when I provide it depth, color and pose data just from a different type of stereo camera? Thank you in advance.

2

Could you post a sample of the console log indicating what the errors/warnings are? We don’t test with the OAK-D Pro but there’s no reason we’re aware of that with good depth images (in this case, from time synchronized imagers with good intrinsics for stereo rectification and disparity estimation) nvblox would have any trouble.

3

Here is one way I have tried using the launch from the tutorial:

ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py

Which repeatedly prints the following:

[controller_server-2] [INFO] [1708004714.980645889] [local_costmap.local_costmap]: Timed out waiting for transform from base_link to odom to become available, tf error: Invalid frame ID "odom" passed to canTransform argument target_frame - frame does not exist

Until I run:

ros2 run tf2_ros static_transform_publisher 1 0 0 0 0 0 odom base_link

After which it prints once:

[controller_server-2] [INFO] [1708004820.980795435] [local_costmap.local_costmap]: start 
[controller_server-2] [INFO] [1708004821.131126240] [controller_server]: Creating bond (controller_server) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.237140316] [lifecycle_manager_navigation]: Server controller_server connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.237288366] [lifecycle_manager_navigation]: Activating smoother_server
[smoother_server-3] [INFO] [1708004821.237906022] [smoother_server]: Activating 
[smoother_server-3] [INFO] [1708004821.238000453] [smoother_server]: Creating bond (smoother_server) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.345241153] [lifecycle_manager_navigation]: Server smoother_server connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.345370273] [lifecycle_manager_navigation]: Activating planner_server 
[planner_server-4] [INFO] [1708004821.346131244] [planner_server]: Activating 
[planner_server-4] [INFO] [1708004821.346196438] [global_costmap.global_costmap]: Activating 
[planner_server-4] [INFO] [1708004821.346221371] [global_costmap.global_costmap]: Checking transform 
[planner_server-4] [INFO] [1708004821.346338610] [global_costmap.global_costmap]: start 
[planner_server-4] [INFO] [1708004821.396548140] [planner_server]: Activating plugin GridBased of type NavfnPlanner 
[planner_server-4] [INFO] [1708004821.398888150] [planner_server]: Creating bond (planner_server) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.506415465] [lifecycle_manager_navigation]: Server planner_server connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.506619551] [lifecycle_manager_navigation]: Activating behavior_server 
[behavior_server-5] [INFO] [1708004821.507564649] [behavior_server]: Activating 
[behavior_server-5] [INFO] [1708004821.507679879] [behavior_server]: Activating spin 
[behavior_server-5] [INFO] [1708004821.507714728] [behavior_server]: Activating backup 
[behavior_server-5] [INFO] [1708004821.507742516] [behavior_server]: Activating drive_on_heading 
[behavior_server-5] [INFO] [1708004821.507766571] [behavior_server]: Activating wait 
[behavior_server-5] [INFO] [1708004821.507791572] [behavior_server]: Creating bond (behavior_server) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.615716973] [lifecycle_manager_navigation]: Server behavior_server connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.615844169] [lifecycle_manager_navigation]: Activating bt_navigator 
[bt_navigator-6] [INFO] [1708004821.616322425] [bt_navigator]: Activating 
[bt_navigator-6] [INFO] [1708004821.659189066] [bt_navigator]: Creating bond (bt_navigator) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.764152241] [lifecycle_manager_navigation]: Server bt_navigator connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.764221738] [lifecycle_manager_navigation]: Activating waypoint_follower 
[waypoint_follower-7] [INFO] [1708004821.764866168] [waypoint_follower]: Activating 
[waypoint_follower-7] [INFO] [1708004821.764944022] [waypoint_follower]: Creating bond (waypoint_follower) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.870946812] [lifecycle_manager_navigation]: Server waypoint_follower connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.871149571] [lifecycle_manager_navigation]: Activating velocity_smoother 
[velocity_smoother-8] [INFO] [1708004821.871916656] [velocity_smoother]: Activating 
[velocity_smoother-8] [INFO] [1708004821.872039315] [velocity_smoother]: Creating bond (velocity_smoother) to lifecycle manager. 
[lifecycle_manager-9] [INFO] [1708004821.977177398] [lifecycle_manager_navigation]: Server velocity_smoother connected with bond. 
[lifecycle_manager-9] [INFO] [1708004821.977236914] [lifecycle_manager_navigation]: Managed nodes are active 
[lifecycle_manager-9] [INFO] [1708004821.977255554] [lifecycle_manager_navigation]: Creating bond timer...

Then because we are not using the bag data I set use_sim_time to False:

ros2 param set /nvblox_node use_sim_time False

Then, when running my camera launch, I get output that suggests the messages are being received but no output topics publish anything meaningful.

[lifecycle_manager-9] [INFO] [1708012787.548474046] [lifecycle_manager_navigation]: Managed nodes are active 
[lifecycle_manager-9] [INFO] [1708012787.548500271] [lifecycle_manager_navigation]: Creating bond timer... 
[component_container_mt-1] [WARN] [1708013621.048144346] [tf2_buffer]: Detected time source change. Clearing TF buffer. 
[component_container_mt-1] [INFO] [1708013621.119221657] [nvblox_node]: Depth Statistics:  
[component_container_mt-1] avg=nan, min=nan, max=nan, std_dev=nan, count=0 
[component_container_mt-1] [INFO] [1708013621.122293758] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013622.142110232] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013623.162235484] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013624.192563258] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013625.222149530] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013626.322126894] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013627.482145718] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013628.562258029] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013629.572828892] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013630.632235549] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013631.121398977] [nvblox_node]: Depth Statistics:  
[component_container_mt-1] avg=82.663232, min=22.180779, max=241.557170, std_dev=38.503534, count=121 
[component_container_mt-1] [INFO] [1708013631.722399435] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013632.752406449] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013633.812412415] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013634.892286137] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013635.892553643] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013636.922265329] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013637.942252961] [nvblox_node]: depth queue was longer than 30 deleted 1 messages. 
[component_container_mt-1] [INFO] [1708013638.962245488] [nvblox_node]: depth queue was longer than 30 deleted 1 messages.

For example, here is a message from /nvblox_node/static_esdf_pointcloud:

header: 
  stamp: 
    sec: 1708014833 
    nanosec: 237362703 
  frame_id: odom 
height: 0 
width: 0 
fields: [] 
is_bigendian: false 
point_step: 0 
row_step: 0 
data: [] 
is_dense: false 
---
4

I have also tried running the node standalone like so:

ros2 run nvblox_ros nvblox_node

With the same TF publishing and with my camera node outputting to its 4 input topics: /color/camera_info /color/image /depth/camera_info /depth/image

And I get the same messages to /nvblox_node/static_esdf_pointcloud with empty arrays. Here is my rqt_graph

image
image2562×1442 413 KB

5

Hi @sarcosian1

Can you provide details on how you set up the OAK-D Pro node?
Looking at your error and your rqt_graph, it looks like nvblox doesn’t have any pose input that is required to build a map.

From the documentation, Nvblox takes a depth image, a color image, and a pose as input, with which it computes a 3D scene reconstruction on the GPU.

Best,
Raffaello