So I am trying to set up a digital twin for the UR16e. I have the Isaac Sim enviornment set up but now I am trying to create a ROS2 node using the joint_trajectory_controller from ros2_control, while at the same time still being able to use something like MoveIt2 or just another control node running to allow for positional programming. I am wondering if you guys know of any examples that I can look into, or how you have everything structured.
Or am I thinking about this all wrong? Is the robot not supposed to have that type of looping connection, because at least with ROS2, I can have the one controller going, which then causes issues when I try to have some form of control over the robot? Am I supposed to have another sensor that complete the digital twin?
Why do you need multiple control nodes for the robot arm? There can be controller conflicts if multiple entities try to command the robot at the same time.
For detailed examples of integrating MoveIt 2 with Isaac Sim and ROS 2 control, see these official documentation resources:
Sorry rereading what I posted I am realizing I didn’t give all the detail. I am using a custom node to send positional information from Isaac Sim to the real (or simulated robot usign the ur_driver) UR16e then having Isaac Sim subscribe to the position information from the real (or simulated robot usign the ur_driver) UR16e so as the UR16e moves Isaac Sim will also mirror. Another thing with that driver is that I haven’t really found a way to have the real (or simulated robot usign the ur_driver) UR16e mirror the Isaac Sim UR16e without using a controller from ros2 control. The goal is something like if the digital robot in Isaac Sim hits a wall then the real robot will not move past that point but the Isaac Sim robot is subscribed to the position information from the real (or simulated robot using the ur_driver) UR16e. This is so we can do rapid prototyping using a digital twin set up so the operators can use the teach pendent and get practice that has more than just seeing that it hit the wall in a simulation. This is sorta the stepping stone that I am trying to setup for the next person down to line rather than having a completely helpful digital twin.
EDIT: Also I am making sure that then are using the same type of controller. But what I am more wondering is how to achieve the bidirectional mirroring while still having the ability to control? And is this even an achievable goal?
Your overall goal absolutely makes sense: using Isaac Sim as a digital twin for the UR16e, where Isaac Sim can both follow the real robot and also impose virtual constraints so the real robot stops when the simulated robot hits a wall is a valid and achievable use case.
The tricky part is how “bidirectional” you make the control. In practice you generally want one side to be the authority for motion commands at any given time, and the other side to be a follower, otherwise you can end up with control loops and conflicting commands. For example, if Isaac Sim is sending trajectories through ros2_control and the UR driver is also being commanded from something else (e.g. MoveIt or the teach pendant), they can both try to drive the same joints.
A more robust way to think about it is:
One direction as “command”: Isaac Sim (or an external planner that uses the sim) computes motion with collision checks and sends commands to the UR (e.g. via joint_trajectory_controller). If the simulated robot would collide or exceed a limit, you simply stop or clamp the command so the real robot never receives unsafe motion.
One direction as “feedback”: The real UR publishes joint states through the ur_driver, and Isaac Sim subscribes and mirrors those states so the twin tracks what the real robot actually did. This gives you the operator training and visualization you want without making the real robot a second independent controller.
You can still support workflows like “teach pendant training” by making the real robot the command source in that mode: the pendant moves the real UR, UR publishes joint states, and Isaac Sim just follows. In a separate “guided” or “safety‑enforced” mode, you flip roles and let Isaac Sim / your ROS 2 node be the command authority, with the UR only following those commands. The key is to avoid having two active position controllers on the same joints at the same time; instead, define clear modes and master–follower roles.