It is not showing any error, but output also is not visible when trying to visualise. I have attached the video as well as log files to clarify this further. Like the node is getting executed and terminating automatically.
I want to do pose estimation and tracking as we go and I want to do it for custom object so any resources that could help me generate 3D assets for the custom object in the format required by Foundation Pose, that would be helpful as well.
From your attached log, it is failed to open file model.onnx. Please check if you download the pre-trained FoundationPose models from NGC to the required directory. And convert them sucessfully.
[component_container_mt-1] Could not open file model.onnx
[component_container_mt-1] [ERROR] [1747134377.165559090] [TRT]: TRT ERROR: ModelImporter.cpp:914: Failed to parse ONNX model from file: model.onnx!
Thanks for the swift response.
So I did isaac_ros_foundationpose — isaac_ros_docs documentation Step 1 as instructed to convert .onnx file to engine file. And in Step 4 while executing the cli is providing the engine file in the run command.
I ran the step 1 i.e. file conversion inside docker as it already had trt exec utilities. and the conversion was sucessfull but it is still not working, I got the same errors and it is not working. I have attached the log file, I am not sure it is trying to find model.onnx which doesn’t exist. But I am following everything step by step as indicated in document
I am not sure but can you check internally about trt version mismatch, as I tried to make the engine file outside the docker where I have TRT version 10.0.X, then I got version mismatch error. As the docker’s trt version is different. Anyway i rectified it
I have the following confusion.
The instructions in isaac_ros_foundationpose shall be sufficient to complete the demo. If you’re still observing the original issue, please let me know.
For the other problems, please open separated topics. Let’s keep one topic for one problem.
OK I can open another topic for Isaac Sim issue, but both are related to Foundation Pose so didn’t felt need to do so.
So continuing my orignal topic of running foundation pose natively [Point 2 in my previous post] Kindly provide a workaround of how to make it work with the realsense. I have attached the logs as well.
We will improve Isaac 4.0 to have clear requirements for dependency versions. Thanks for your reporting!
For the realsense error,
please check if all the versions are correct listed from Isaac ROS RealSense Setup — isaac_ros_docs documentation.
And verify that realsense camera streams are visible when running ‘realsense-viewer’ inside the docker container.
Thanks for those steps, but I have already did those. Now I am just reiterating so we dont loose the context, so I want to perform pose estimation for the Novel objects using realsense. (its mentioned in the document foundation pose is the only model that can do that without retraining). But its not working. These are the logs
Please provide the solution and in the same context if the model can detect novel objects why does it need DETR (Object detection) input as my worry is it might only detect those objects then.
The usb error still presists in your new attached log. But given that you can have realsense working in another topic, I think your problem now is more about how to detect novel objects.
The fountionpose demo needs the segmentation info from DETR. If you are interested to know the details of the design, please refer to Isaac ROS FoundationPose
For the novel objects, it’s required to create your own 3D object mesh for FoundationPose. The associated tutorials are available in try-more-examples
Hi @Ahung
So the followup to that is so the assumption that I believed due to its presence in documentation here Isaac ROS Pose Estimation — isaac_ros_docs documentation that Foundation pose out of the box works with novel object is not right, as it uses DETR in the backend to detect the mask and we would need to train DETR for that novel object anyway to get the correct pose detection and tracking ?
Thanks for pointing to the resource of importing custom object into Isaac Sim, I saw that and followed it for the same. If the above assumption is correct can you point out any utilities or any end to end ways by which we can train DETR for the custom object generating data in Isaac Sim or any other way this could be acheived.
The only thing you need to do is to create 3D object mesh which is required by
foundationpose node and tracking node. Thus, you don’t need to re-train DETR.
It needs DETR’s input to detect the mask. In the office hour video also, at 28:38 the question on similar lines is asked. She said it might but we need to check the DETR page.I could not find any such information. Can you kindly check, as I tried and its not working on the novel objects because of DETR backend I assume.
Would love to hear your thoughts.
And instructions to how to fine tune DETR for foundation pose with out own custom objects (in Isaac Sim probably as it would be relatively easier)
The models (sdetr_grasp and sdetr_amr) in Isaac ROS RT-DETR
is trained for object classes that can be grasped by a standard robot arm and object classes that are relevant to the operation of an Autonomous Mobile Robot respectively.
If you tried Isaac ROS RT-DETR and your novel object can be detected, you just need to create 3D mesh for foundationpose. But if your novel detect can’t be detected by Isaac ROS RT-DETR, then you would need to train DETR for that novel object.
Hi @Ahung
So as mentioned in the documentation of trained DETR for object grasping, the classes it is trained on is present here SyntheticaDETR | NVIDIA NGC . I was trying to detect pose of normal objects like bottle, phone etc. but it is not working as the similar classes it is not present in the objects of this trained model.
So as I can see, the DETR has tao support so if we want to train synthetic DETR for our particular object. Do you have a way to do it seamlessly using TAO, i.e. synthetic data generation as well as training. or any other set of instructions, using which I can generate the synthetic data and annotation using Isaac sim and train the model for my usecase so that it can seamlessly integrate with Isaac Sim.
