# Foundation Pose estimation and tracking

**URL:** <https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079>\
**Category:** Isaac ROS\
**Tags:** isaac-ros-pose-estimation\
**Created:** [May 13, 2025, 11:33am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079 "2025-05-13T11:33:05Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [May 13, 2025, 11:33am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/1 "2025-05-13T11:33:05Z")

</div>

Hi  
I am trying to do dry run for foundation pose estimation and tracking. I was following instructions from [isaac\_ros\_foundationpose — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_foundationpose/index.html#)

It is not showing any error, but output also is not visible when trying to visualise. I have attached the video as well as log files to clarify this further. Like the node is getting executed and terminating automatically.

> I want to do pose estimation and tracking as we go and I want to do it for custom object so any resources that could help me generate 3D assets for the custom object in the format required by Foundation Pose, that would be helpful as well.

Thanks in advance

  
[FoundationPoseLogs.txt](https://forums.developer.nvidia.com/uploads/short-url/bMnOdqKMAGdGpLa275keipCmsUm.txt) (82.4 KB)

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [May 14, 2025, 1:58am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/2 "2025-05-14T01:58:02Z")

</div>

Hi @raghav.nandwani

Thanks for your post.

From your attached log, it is failed to open file model.onnx. Please check if you download the pre-trained FoundationPose models from NGC to the required directory. And convert them sucessfully.

```auto
[component_container_mt-1] Could not open file model.onnx
[component_container_mt-1] [ERROR] [1747134377.165559090] [TRT]: TRT ERROR: ModelImporter.cpp:914: Failed to parse ONNX model from file: model.onnx!

```

Meanwhile, which setup are you using?

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [May 14, 2025, 3:39pm UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/3 "2025-05-14T15:39:48Z")

</div>

Hi @Ahung

Thanks for the swift response.  
So I did [isaac\_ros\_foundationpose — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_foundationpose/index.html#run-launch-file) Step 1 as instructed to convert .onnx file to engine file. And in Step 4 while executing the cli is providing the engine file in the run command.

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/5/f/b/5fbc0b0d9b1a0906f5a7ff5475e779ddd006d0f0.png)

The onnx file is obtained from Step 2 of this [isaac\_ros\_foundationpose — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_foundationpose/index.html#download-quickstart-assets)

> I ran the step 1 i.e. file conversion inside docker as it already had trt exec utilities. and the conversion was sucessfull but it is still not working, I got the same errors and it is not working. I have attached the log file, I am not sure it is trying to find model.onnx which doesn’t exist. But I am following everything step by step as indicated in document

GPU : RTX 4090  
Cuda : 12.6  
[FoundationPoseLogs.txt](https://forums.developer.nvidia.com/uploads/short-url/151t84VJjTurvFhB2masSALCp0S.txt) (62.9 KB)

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [May 15, 2025, 6:49am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/4 "2025-05-15T06:49:29Z")

</div>

Hi @raghav.nandwani

Have you cmpleted the [Isaac ROS RT-DETR tutorial](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_object_detection/isaac_ros_rtdetr/index.html#quickstart) ? I think you probably missed this step.

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [May 21, 2025, 12:28pm UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/5 "2025-05-21T12:28:18Z")

</div>

Hi @Ahung  
Yes I have done it.

I am not sure but can you check internally about trt version mismatch, as I tried to make the engine file outside the docker where I have TRT version 10.0.X, then I got version mismatch error. As the docker’s trt version is different. Anyway i rectified it  
I have the following confusion.

1. I ran this tutorial to run in isaac sim [Tutorial for FoundationPose with Isaac Sim — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/concepts/pose_estimation/foundationpose/tutorial_isaac_sim.html) and its working fine as expected. But it didn’t mention how to run it if we want to run using sample obj files or other assets in isaac sim. Can you point me to that

2. I ran this flow as well [isaac\_ros\_foundationpose — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_foundationpose/index.html#run-launch-file) and it ran perfectly for rosbag file. But when I am trying to run it for realsense I am getting error  
[FoundationPoseLogs.txt](https://forums.developer.nvidia.com/uploads/short-url/8kqnCyMIqdtAOkRBKBm3zjVvsVB.txt) (56.1 KB)

Best  
Raghav

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [May 21, 2025, 2:45pm UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/6 "2025-05-21T14:45:50Z")

</div>

Hi @raghav.nandwani

The instructions in [isaac\_ros\_foundationpose](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_foundationpose/index.html#) shall be sufficient to complete the demo. If you’re still observing the original issue, please let me know.

For the other problems, please open separated topics. Let’s keep one topic for one problem.

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [May 22, 2025, 11:57am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/7 "2025-05-22T11:57:32Z")

</div>

Hi @Ahung

OK I can open another topic for Isaac Sim issue, but both are related to Foundation Pose so didn’t felt need to do so.

So continuing my orignal topic of running foundation pose natively [Point 2 in my previous post] Kindly provide a workaround of how to make it work with the realsense. I have attached the logs as well.

THanks in advance

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [May 28, 2025, 1:58am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/8 "2025-05-28T01:58:51Z")

</div>

Hi @raghav.nandwani ,

We will improve Isaac 4.0 to have clear requirements for dependency versions. Thanks for your reporting!

For the realsense error,

1. please check if all the versions are correct listed from [Isaac ROS RealSense Setup — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/getting_started/hardware_setup/sensors/realsense_setup.html).  
And verify that realsense camera streams are visible when running ‘realsense-viewer’ inside the docker container.

2. There’s also realsense troubleshoot documentation [Hardware Setup — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/troubleshooting/hardware_setup.html) if user sees one of the issues listed here

3. If it’s still not working, it’d be good to try replugging realsense camera physically.

Let me know if these info can help.

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [May 30, 2025, 9:58am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/9 "2025-05-30T09:58:52Z")

</div>

Hi @Ahung

Thanks for those steps, but I have already did those. Now I am just reiterating so we dont loose the context, so I want to perform pose estimation for the Novel objects using realsense. (its mentioned in the document foundation pose is the only model that can do that without retraining). But its not working. These are the logs

[FoundationPoseLogs.txt](https://forums.developer.nvidia.com/uploads/short-url/8kqnCyMIqdtAOkRBKBm3zjVvsVB.txt) (56.1 KB)

Please provide the solution and in the same context if the model can detect novel objects why does it need DETR (Object detection) input as my worry is it might only detect those objects then.

Thanks in advance

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [June 2, 2025, 10:43am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/10 "2025-06-02T10:43:27Z")

</div>

Hi @raghav.nandwani

The usb error still presists in your new attached log. But given that you can have realsense working in another topic, I think your problem now is more about how to detect novel objects.

The fountionpose demo needs the segmentation info from DETR. If you are interested to know the details of the design, please refer to [Isaac ROS FoundationPose](https://www.youtube.com/watch?v=2F8t9L2cmXs)

For the novel objects, it’s required to create your own 3D object mesh for FoundationPose. The associated tutorials are available in [try-more-examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/isaac_ros_foundationpose/index.html#try-more-examples)

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [June 2, 2025, 2:35pm UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/11 "2025-06-02T14:35:44Z")

</div>

Hi @Ahung  
So the followup to that is so the assumption that I believed due to its presence in documentation here [Isaac ROS Pose Estimation — isaac\_ros\_docs documentation](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_pose_estimation/index.html#overview) that Foundation pose out of the box works with novel object is not right, as it uses DETR in the backend to detect the mask and we would need to train DETR for that novel object anyway to get the correct pose detection and tracking ?

Thanks for pointing to the resource of importing custom object into Isaac Sim, I saw that and followed it for the same. If the above assumption is correct can you point out any utilities or any end to end ways by which we can train DETR for the custom object generating data in Isaac Sim or any other way this could be acheived.

Best  
Raghav

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [June 3, 2025, 2:56am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/12 "2025-06-03T02:56:50Z")

</div>

Hi @raghav.nandwani

The only thing you need to do is to create 3D object mesh which is required by  
foundationpose node and tracking node. Thus, you don’t need to re-train DETR.

Please refer to [Isaac ROS Office Hours: Isaac ROS FoundationPose](https://www.youtube.com/watch?v=2F8t9L2cmXs) for details.

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [June 4, 2025, 9:28am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/13 "2025-06-04T09:28:41Z")

</div>

Hi @Ahung

I believe there are some contradicting so in below architecture if you see

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/b/4/0/b406435e849111b609a0cd30d59525330864d23b.jpeg)

It needs DETR’s input to detect the mask. In the office hour video also, at 28:38 the question on similar lines is asked. She said it might but we need to check the DETR page.I could not find any such information. Can you kindly check, as I tried and its not working on the novel objects because of DETR backend I assume.

Would love to hear your thoughts.

And instructions to how to fine tune DETR for foundation pose with out own custom objects (in Isaac Sim probably as it would be relatively easier)

Best  
Raghav

---

<div class="post-metadata">

**Author:** ![Ahung](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ahung/32/347464_2.png) [@Ahung](https://forums.developer.nvidia.com/u/Ahung)\
**Post date:** [June 9, 2025, 10:26am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/14 "2025-06-09T10:26:44Z")

</div>

Hi @raghav.nandwani

The models (sdetr\_grasp and sdetr\_amr) in [Isaac ROS RT-DETR](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_object_detection/isaac_ros_rtdetr/index.html#quickstart)  
is trained for object classes that can be grasped by a standard robot arm and object classes that are relevant to the operation of an Autonomous Mobile Robot respectively.

If you tried Isaac ROS RT-DETR and your novel object can be detected, you just need to create 3D mesh for foundationpose. But if your novel detect can’t be detected by Isaac ROS RT-DETR, then you would need to train DETR for that novel object.

Best,  
Ahung

---

<div class="post-metadata">

**Author:** ![raghav.nandwani](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@raghav.nandwani](https://forums.developer.nvidia.com/u/raghav.nandwani)\
**Post date:** [June 9, 2025, 10:51am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/15 "2025-06-09T10:51:24Z")

</div>

Hi @Ahung  
So as mentioned in the documentation of trained DETR for object grasping, the classes it is trained on is present here [SyntheticaDETR | NVIDIA NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/isaac/models/synthetica_detr) . I was trying to detect pose of normal objects like bottle, phone etc. but it is not working as the similar classes it is not present in the objects of this trained model.

So as I can see, the DETR has tao support so if we want to train synthetic DETR for our particular object. Do you have a way to do it seamlessly using TAO, i.e. synthetic data generation as well as training. or any other set of instructions, using which I can generate the synthetic data and annotation using Isaac sim and train the model for my usecase so that it can seamlessly integrate with Isaac Sim.

Thanks in advance.

Best  
Raghav

---

<div class="post-metadata">

**Author:** ![vchuang](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/vchuang/32/436710_2.png) [@vchuang](https://forums.developer.nvidia.com/u/vchuang)\
**Post date:** [January 6, 2026, 9:12am UTC](https://forums.developer.nvidia.com/t/foundation-pose-estimation-and-tracking/333079/16 "2026-01-06T09:12:42Z")

</div>

Closed on January 6, 2026, 9:12am UTC


