Important: Isaac Sim support
Note: For Isaac Sim support, the community is gradually transitioning from this forum to the Isaac Sim GitHub repository so that questions and issues can be tracked, searched, and resolved more efficiently in one place. Whenever possible, please create a GitHub Discussion or Issue there instead of starting a new forum topic.
Note: For any Isaac Lab topics, please submit your topic to its GitHub repo ( GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub ) following the instructions provided on Isaac Lab’s Contributing Guidelines ( Contribution Guidelines — Isaac Lab Documentation ).
Please provide all relevant details below before submitting your post. This will help the community provide more accurate and timely assistance. After submitting, you can check the appropriate boxes. Remember, you can always edit your post later to include additional information if needed.
Isaac Sim Version
6.0.1
6.0.0
5.1.0
5.0.0
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):
Operating System
Ubuntu 24.04
Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):
GPU Information
- Model:
- Driver Version:
Topic Description
Detailed Description
(Describe the issue in detail, including what you were trying to do, what you expected to happen, and what actually happened)
Steps to Reproduce
TL;DR – I have a Unitree G1 hierarchical controller trained in Isaac Lab (PhysX): a high-level switcher → 64-D “target-frame” AMP actors (run/jump) → a FROZEN BeyondMimic/ProtoMotions tracker (157-D obs → 29 joint targets, with BeyondMimic PD scaling). Re-implementing this in MuJoCo, the tracker alone matches a reference motion fine, but the combined policy falls in ~1 s. Matched the model, gains, armature, default pose, timing, obs/action conventions exactly and ruled out all the usual suspects. Looking for anyone who has actually transferred an Isaac/PhysX-trained tracker + RL actor stack to MuJoCo (or knows the missing piece). — STACK — - PhysX 2000 Hz (dt=0.005), decimation 4 → 50 Hz control. - Actuation: implicit position PD (Isaac “built-in PD”), stiffness/damping = BeyondMimic G1 values, effort limits per joint, armature per joint. - Low-level (explicable) tracker: proprio(64) = dof_pos(29) + dof_vel(29) + root_local_ang_vel(3) + proj_gravity(3) target(64) = rel_anchor_rot6d(6) + dof_vel(29) + proj_gravity(3) prev(29) = previous processed actions (absolute joint targets) obs = concatenate(proprio, target, prev) → 157; action → q_target = default_dof_pos + (effort/stiffness)*raw - Mid-level: AMP run actor, obs 134 = proprio(64) + base_lin_vel(3) + vel_cmd(3) + prev_target_frame(64); outputs the 64-D target frame directly. - High level: 6-D switcher → gets run/jump. - Model: g1_holo_output.xml + same MJCF used for tracking; per-joint armature, frictionloss=0.1, sphere feet). — MuJoCo REPRO (what we matched) — - Loaded the same MJCF; set opt.timestep=0.005; 4 substeps @ 50 Hz. - Same default pose (hip -0.312, knee 0.669, ankle -0.363, elbow 0.6, shoulders +/-0.2), same stiffness/damping/effort, same armature. - Same obs conventions: local ang vel = pelvis-frame qvel[3:6]; proj gravity in torso frame; normalization baked into the exported .pt. - Same action scaling (offset + effort/stiffness * raw) and same prev_action processed = absolute targets. - Cross-checked with the tracker’s embedded normalization constants. — EVIDENCE — 1) Tracker alone: feeds target frames built from a real reference locomotion clip → robot stays upright for the full episode in MuJoCo (z ≈ 0.78, no fall). So the tracker checkpoint, model, gains and plumbing are OK in MuJoCo. 2) Composite (run actor + tracker): falls in ~0.9–2 s for every velocity command (including cmd=0), from every initial joint pose (zeros, default), with prev-action zero or warm-started. 3) Diagnostics: the exported run actor’s obs_mean_action block does not look like a valid target frame (non-unit rot6d, and even under a kinematic replay of a reference motion the actor saturates to a constant, out-of-distribution frame. Target-frame block-order permutations and rot6d orthonormalization don’t help. — WHAT WE TRIED (no fix) — - sim dt 0.001 vs 0.005; explicit torque PD vs implicit position PD. - friction loss on/off; contact solref/solimp; friction; foot capsules vs spheres; self-collision on/off. - EMA on actions (1.0/0.8/0.5/0.3/0.20), PD-target clamp, exact-effort clipping. - Target-frame layout permutations, identity rot6d, command=0, init-pose/warm-start sweeps. - A proven Open MuJoCo G1 deploy harness (RoboJudo) as the environment: its own ProtoMotions tracker stands and tracks, ours does too WITH REFERENCE MOTIONS, but the composite run actor still falls. — CONTEXT / PRIOR ART — - ProtoMotions issue #230 reports the same symptom for non-domain-randomized checkpoints (stand ~0.7 s then fall on MuJoCo/Newton; fine in Isaac Lab), with the maintainer noting an old DR-trained G1 is expected to transfer. - whole_body_tracking guidance: kp/kd must match training exactly (ours do). — QUESTION — 1) Has anyone successfully deployed an Isaac/PhysX-trained BeyondMimic/ProtoMotions tracker *with an RL actor on top* to MuJoCo without retraining? If so, what contact/actuator/system-ID details were decisive? 2) Is the composite’s failure expected (closed-loop fragility of the aggressive actor) or is there a known convention mismatch (e.g., target-frame parameterization, reduced-coords target built relative vs absolute, or per-joint target index order) that we’re missing? 3) Does anyone know a public, smaller run/jump motion pack (or a working MuJoCo sim2sim script) for this exact tracker so we can cross-check the run actor’s expected target distribution?
-– ARTIFACTS WE CAN SHARE — - G1 MJCF (g1_holo_compat.xml) and USD, the tracker .pt (+ metadata), the run/jump actors .pt (+ metadata), the training configs, and a standalone MuJoCo harness that reproduces the fall. Happy to attach/upload or host them.