Foundation Pose Isaac Sim

Robotics & Edge Computing Robotics - Isaac Isaac ROS
,
1

Hi

I am trying to understand and demo foundation pose to understand the working.
The reason I started working on this is by having the understanding that it will work on novel object without retraining as indicated here

I am following this link for running foundation pose in Isaac Sim Tutorial for FoundationPose with Isaac Sim — isaac_ros_docs documentation. I have following question

  1. why does this pose tracking model running in isaac sim needs 1. Isaac ROS ESS depth estimation while the usual setup here isaac_ros_foundationpose — isaac_ros_docs documentation doesn’t need that module

Now I have a doubt in working of that

  1. According to the note in documentation I gave the object “006_mustard_bottle” which is among the object classes for DETR. I ran this using the below command
ros2 launch isaac_ros_foundationpose isaac_ros_foundationpose_isaac_sim.launch.py mesh_file_path:=/workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_foundationpose/Mustard/textured_simple.obj  texture_path:=/workspaces/isaac_ros-dev/isaac_ros_assets/isaac_ros_foundationpose/Mustard/texture_map.png

and I got below results which doesn’t seem right, can you kindly highlight what could be the problem ( I got the assets through isaac_ros_examples - isaac_ros_foundationpose — isaac_ros_docs documentation Step4 for Rosbag)

image
image1907×964 114 KB

  1. How to make it run for custom objects i.e. train the DETR on the custom objects in isaac sim I have obj file using this workflow Tutorial to create your own 3D object mesh for FoundationPose — isaac_ros_docs documentation and I have imported it in SIM now need to train the detr and replace the onnx file with the trained model one
  2. THe other objects that DETR is trained on how to generate texture.png file for those ? folowing this https://nvidia-isaac-ros.github.io/concepts/pose_estimation/foundationpose/tutorial_usd_to_obj.htmlonly generates .obj file and a .mtl file but the module needs .png for texture so how to generate that ?
2

Hi @raghav.nandwani

Thank you for writing on the Isaac ROS forum.

Can you provide more debugging information, including the visualization of depth, bounding box, and mask?
If your data is confidential, you can reach our support.

Best,
Raffaello

3

Hi @Raffaello
So there were 4 major questions in above

  1. First is more of conceptual, as in documentation here Isaac ROS Pose Estimation — isaac_ros_docs documentation it is mentioned that Foundation pose works on novel objects without retraining. But in the setup here Tutorial for FoundationPose with Isaac Sim — isaac_ros_docs documentation it is confirmed that at the backend its using ESS depth estimation as well as DETR, which doesn’t work on novel object and would require retraining, so is it true that foundation pose end to end can’t work on novel objects ?

  2. For the working on trained objects. in the detr_grasp model i.e. “006_mustard_bottle” Ref. SyntheticaDETR | NVIDIA NGC) i shared the sample picture of detected pose which I didn’t get properly. asssets to run that was obtained from ( I got the assets through isaac_ros_examples - isaac_ros_foundationpose — isaac_ros_docs documentation Step4 for Rosbag)

  3. How to make it run for custom objects i.e. train the DETR on the custom objects in isaac sim I have obj file using this workflow Tutorial to create your own 3D object mesh for FoundationPose — isaac_ros_docs documentation and I have imported it in SIM now need to train the detr and replace the onnx file with the trained model one

  4. The other objects that DETR is trained on how to generate texture.png file for those ? folowing this https://nvidia-isaac-ros.github.io/concepts/pose_estimation/foundationpose/tutorial_usd_to_obj.htmlonly generates .obj file and a .mtl file but the module needs .png for texture so how to generate that ?

I hope you have more clarity now

1 Like
4

Hi @raghav.nandwani

Thank you for your reply. I forwarded your message to the engineering team.

I keep you posted!

Raffaello

5

Hi @Raffaello

Thanks, awaiting your response.

Best
Raghav

6

I’m facing about the same problems, some info about these missing/undocumented parts of the foundationPose pipeline would be really great!