Hi,
I’m currently using a RealSense camera to follow the tutorial at isaac_ros_rtdetr — isaac_ros_docs documentation for object detection using the RT-DETR model. The example uses the sdetr_grasp.onnx pre-trained model, and I was able to successfully reproduce the demo with the provided Rosbag.
Now, I want to test a different model. I downloaded a pre-trained model from the RT-DETR PyTorch repo (RT-DETR/rtdetr_pytorch at main · lyuwenyu/RT-DETR · GitHub) and replaced the original commands as follows:
From:
/usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan
ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,rtdetr engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/sdetr_grasp.plan
To:
/usr/src/tensorrt/bin/trtexec --onnx=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/rtdetr_r101vd_6x_coco.onnx --saveEngine=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/rtdetr_r101vd_6x_coco.plan
ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_mono_rect,rtdetr engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/synthetica_detr/rtdetr_r101vd_6x_coco.plan
There were no errors during the process, and the camera feed was displayed successfully. However, no detection results were shown. I would like to ask: besides replacing the ONNX model and engine file, are there any additional steps needed to run other RT-DETR models with this example?
Thank you very much in advance. I’ve also included my log information and model below for reference.
log.txt (26.8 KB)
model.zip
Hi @chihyungwang79
Thank you for your post.
The documentation outlines all the necessary steps to convert a model. Based on your logs, everything appears to be functioning correctly, and the converted model has loaded successfully.
Which setup are you using? It may be a long process on your device.
Best,
Raffaello
Hi @Raffaello ,
Thank you for your response. My operating system is Ubuntu 22.04, running on an x86 platform with an RTX 5000 Ada GPU.
1 Like
Hi @Raffaello
I’m still not getting any detection results. Could you please give me some suggestions?
Hello Raffaello, can you help me take a look at the problem?
When I set the confidence_threshold to 0.1, I can see detection results, but they are inaccurate, and the bounding boxes are very small.
I would like to ask: during the process of converting from a .pth file to .onnx, and then to .plan, are there any important things to watch out for?
Is it possible that some information or precision is lost during the conversion process?
Hi @997582974
Have you solved this problem?
I am still stuck with the same problem.
Hi all,
can you confirm that your model works on Python/PyTorch directly?
We want to confirm that the model is not the source of the bad detection.
Best,
Raffaello
Hi,
I’m certain that the trained model is working correctly. I followed the instructions from RT-DETR/rtdetrv2_pytorch at main · lyuwenyu/RT-DETR · GitHub to train my model and used its inference script to run predictions on images. Both the .pth and .onnx versions can successfully detect my object.
Hi @Raffaello ,
However, I still have no luck resolving this issue. May I ask if your team has any plans to address it in the near future?
I also have this issue. Using rtdetr_r101vd from lyuwenu’s RT-DETR repo, I verified that in its ONNX version it worked the same as the sdetr_grasp.onnx example using an onnxruntime inference and annotation script. However, once I converted it to a tensorRT .plan following the instructions from isaac_ros_rtdetr, all of the detection outputs are less than 10% confidence and very small bounding boxes.
Hi @chihyungwang79
Thank you for your response. In reviewing your last post, I see three potential areas where deviations might occur:
- The image may be encoded into a tensor differently than the reference.
- The model might produce a different output tensor for the same input tensor.
- The output tensor could be decoded into detections differently than the reference.
Best,
Raffaello
1 Like
I’ve identified the difference between NVIDIA’s provided model and the inference script at
RT-DETR/rtdetrv2_pytorch/references/deploy/rtdetrv2_onnxruntime.py at main · lyuwenyu/RT-DETR · GitHub.
The main distinction is whether the pixel values are normalized—specifically, whether they are scaled from the [0, 255] range to [0, 1].
1 Like
Hi @chihyungwang79
What you discover is the right point.
The following steps is normalize the node to work with the same range. You can use ImageNormalizeNode to fix the range.
Let me know if I helped you.
Best,
Raffaello
Hi @Raffaello
I modified the image_to_tensor_node to normalize the pixel values to match my model, and now I can successfully detect my object.
Thank you for your help!
1 Like
@chihyungwang79
In the provided launch file in isaac_ros_rtdetr/launch/isaac_ros_rtdetr_core.launch, I changed line 96
image_to_tensor_node from False to True. Looking at ros2 param list,
I was not able to see a parameter to define what to scale the image to. How did you go about doing that? Just setting it to True, I am still not getting any detections when I ros2 topic echo /detections_output
I am running from a ROS bag and I changed the name of the image topic to /image_rect to match the subscription
For me, simply changing False to True was enough to successfully detect the object. It’s possible that your confidence_threshold value is set too high, which might be why no objects are being detected.
1 Like
Thank you for your info , I also find the scale issue , but I change the value from False to True , still cannot fixed the issue , could you please tell me which model do you use from RT-DETR/rtdetrv2_pytorch at main · lyuwenyu/RT-DETR · GitHub
Thank you very much
