Converting VR hand and arm data into robot joint angles

Omniverse Isaac Sim
1

Isaac Sim Version

4.5.0

Operating System

Ubuntu 22.04

Topic Description

Converting VR hand and arm data into robot joint angles

Detailed Description

I am using Vuer to transmit hand and arm data from a VR headset into Isaac Sim, but I am not sure how to convert this data into robot joint angles. I am looking for reference methods or examples to perform inverse kinematics calculations for the arm and hand, in order to correctly control the robot arm in Isaac Sim.

Steps to Reproduce

  1. Configure Vuer and the VR headset.
  2. Receive hand and arm data in Isaac Sim.
  3. Attempt to convert the data into robot joint angles (currently unsure how to do this).
  4. Send the resulting actions to the robot joints (already implemented)

Expected Behavior
I hope to get suggestions or example code for converting VR finger data into robot joint angles.

Actual Behavior
Currently, I don’t know how to perform the conversion from data to joint angles.

2

Hi.

I had no experience with Vuer,
but in my case, I had used Vive Pro 2 HMD and Vive Tracker, and senseglove which is dataglove with 4 encoder/finger.

My best suggestion is simply to transfer the TF of the Vive Tracker into the targetcube followtarget example– the specific requirement might be different, but I think this is a good point to synchronize VR device and IsaacSim.

3

Okay, thank you for your advice. I will give it a try.

4

Hi @qwert10318520,

As @PARKBONG suggested, the general approach is:

  1. For the arm: Use the VR controller/hand position as an IK target. Isaac Sim provides motion generation (Lula IK / RMPflow) to solve joint angles from a Cartesian end-effector target. The follow_target example demonstrates this workflow —
    feed the VR wrist pose as the target and the IK solver computes joint angles automatically.
  2. For the hand/fingers: VR hand tracking gives you finger joint angles (or fingertip positions). For a dexterous hand, you can either:
    • Direct joint mapping — if the robot hand has similar kinematics, map VR finger angles directly to robot finger joints (with scaling)
    • Fingertip IK — use the fingertip positions as IK targets for each finger chain

The typical pipeline is: VR data → Cartesian pose → IK solver → joint angles → apply_action().

Closing this out since the approach was discussed. Feel free to open a new topic if you run into specific issues during implementation.