Sim-to-Real Chess Robot: Franka Panda + ROS2 + Octo VLA/CNN (Overcoming VRAM limits)

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hi everyone,

I want to share the current progress of my Mechatronics Engineering undergraduate thesis. I’m building an autonomous chess-playing robotic arm using a Franka Panda in MuJoCo, but I hit a classic hardware bottleneck and had to re-architect the system.

The Challenge & The Architecture: Running the full pipeline locally RViz (ROS2) alongside heavy Vision-Language-Action (VLA) foundation models like Octo (leveraging its native Franka training base), RT-1, and RT-2 was immediately maxing out the 4GB VRAM on my local GTX 1650. Instead of relying on heavy cloud servers, I fully modularized the perception and control loops:

  • FSM & ROS2 Integration: A Finite State Machine manages all logic and only requests trajectories to MoveIt2 when strictly necessary, keeping the system idle when waiting for a move.

  • Low-Level Control: I implemented MPC (Model Predictive Control) and WBC (Whole Body Control) running directly within the MuJoCo physics simulation to ensure grasp stability.

  • Modular Perception: To bypass the VRAM limit during fast-loop testing, the system boots with a custom CNN I trained from scratch specifically for the chessboard state. This leaves the heavy lifting for the Octo VLA and RT-series models for separate, dedicated inference instances.

(The attached video shows a quick demo of the FSM terminal logs orchestrating MoveIt and the custom CNN in action.)

Next Steps (The RL Phase) & Seeking Feedback: Even if it increases the complexity of my thesis, my absolute focus right now is mastering and optimizing the kinematics and control loops. My immediate roadmap includes:

  • Using Reinforcement Learning (RL) to dynamically tune the Q and R matrices of the MPC and the WBC weights.

  • Improving MoveIt2 trajectory generation (smoothing paths and increasing overall execution speed).

  • Enhancing real-time trajectory correction via the control loop.

Next semester, I will be leveraging this exact same base architecture to build a robotic welding simulator for my institute. Because welding requires extreme path precision and dynamic correction, getting the MoveIt2 and MPC/WBC integration flawless right now is critical.

I would highly appreciate any feedback or pointers from the community, especially regarding:

  1. Best practices for smoothing MoveIt2 trajectories and increasing execution speed.

  2. RL approaches for dynamic trajectory correction and stability using MPC/WBC in MuJoCo.

Cheers, Matheus

2

Hello @matheuscovre05,

Welcome to the Isaac ROS forum and thanks for sharing your project!

Since you are already using MoveIt 2 with a Franka, one option worth evaluating is the Isaac ROS cuMotion MoveIt plugin ( isaac_ros_cumotion_moveit — Isaac ROS ). It generates smooth, collision-free, optimal-time trajectories, which you could use as the reference input for your existing MPC/WBC layer. It would be interesting to compare motion time, jerk, and tracking error against your current MoveIt planning pipeline.

For RL, keeping MPC/WBC as the nominal controller and using the policy to provide bounded trajectory corrections may be a good starting point. The isaac_ros_manipulation_dnn_policy Isaac ROS RL sim-to-real tutorial and Gear Assembly workflow may also be a useful reference for integrating learned-policy inference into a ROS 2 control pipeline, although it does not directly tune MPC/WBC parameters.

Good luck with your project!