PartyMimic

Results on held-out motions

Every method is trained on the same 10,731 clips: LAFAN1, a 10k diversity subset of BONES-SEED, 56 kneeling and crawling clips of our own, and 635 KungFuAthlete clips. It is tested on five held-out sets: Loco (80 clips), Mani (48), Dance (40) and Ground (60) from MotionDecode, a different motion-capture source, and Extreme (308): KungFuAthlete martial arts, jumps and flips, held out by whole source video. RP1 uses the same motions retargeted to RP1. PartyMimic comes with two reference encoders: an MLP (default) and a cross-attention encoder (attention, --ref_enc xattn).

PartyMimic rows and the shipped checkpoints use the prior defaults (std_scale 0.35, no Muon weight decay). The updated recipe (--muon_wd 0.1 --std_scale 0.6) matches that tracking to within seed noise and improves deployment metrics (Recipe, Sim2real). Scored by the sim2real leaderboard MuJoCo sim2sim evaluator after ONNX export: every clip played from start to end, failure when the pelvis height error exceeds 0.3 m or the gravity direction error exceeds 0.8. Error is the local body position error over the tracked part of each clip; completion is the tracked share of each clip; Mean is the mean error of the five sets. All trained methods ran 12,000 iterations (NablaMimic: its 2-hour budget) with 2 seeds; the table gives the means. Training is the wall-clock time of 12,000 iterations on an otherwise idle RTX 4090 (i9-14900K). Our cards are 48 GB editions; the MLP fits the standard 24 GB card: with only 24.1 GB left free, G1 and RP1 runs kept the same speed and peaked at 22.6 and 23.4 GB.

Training data and how new the test sets are

We describe every 1-s window by heading-invariant root, joint and hand / foot kinematics, project to 32 dimensions and cluster the 41 h into 2,000 motion clusters. The library spans 864 distinct motion types (the exponential of the cluster entropy): LAFAN1 alone 548, BONES-SEED adds 249, kneeling / crawling 11 and KungfuAthlete, with only 1.3 h, 56. A test window counts as covered when a training window lies within the median distance between windows of different training sources. Only 47% of Loco, 57% of Mani, 8% of Dance, 46% of Ground and 15% of Extreme windows are covered: most test motion is new to every method. KungfuAthlete training clips raise the Extreme coverage from 7% to 15%.

MLP or attention

The attention encoder lets a causal Transformer over the observation history query the goal frames, so the robot's own dynamics shape how it reads the reference. It lowers the G1 error on Ground by 4.4 mm and on Extreme by 1.7 mm, and the mean by 1.6 mm (RP1: 3.2 mm), at 1.6× the training time. The MLP fits a 24 GB card; the attention encoder needs more memory, or --pi_batch 1024 on 24 GB (29 min on G1), which costs G1 2.7 mm on Ground and 2.1 mm on Extreme. --pipeline splits training over two GPUs for 1.6–1.9× the speed.

Sparse keypoints: one command for any body

With --interface sparse5 the command is six poses per frame: the pelvis pose, the head orientation and the poses of both wrists and both ankles, 51 numbers and no joint angles. Any body with a pelvis, a head, two hands and two feet can be written this way: a person in a motion-capture suit, a VR tracker set, a video pose estimate, or another robot. The command does not depend on a robot's joint layout.

The same LAFAN1 motion as captured from a person and retargeted to Unitree G1 and RoboParty RP1, each written as the same six keypoint poses (spheres, with their orientation axes). The person's axes are aligned to G1's at the first frame.

Three additions let the policy follow such a command nearly as well as full joint references:

• The observation adds the robot's own keypoint errors, from forward kinematics of its joint encoders.
• A keypoint-to-joint network, fit to the motion library and frozen, estimates the reference pose for the reference-action skip (--kp_skip).
• An auxiliary head predicts the reference joint angles from the policy's reference latent (--kp_aux).

On RP1 the mean whole-body error over the five sets is 48.7 mm from keypoints against 47.8 mm from joint references with the MLP (47.8 vs 44.6 mm with attention), below every baseline that sees the full joint reference (best: 60.4 mm). The keypoints themselves are tracked within 2 mm of joint references on Loco/Mani and Dance, with about half the error of the baselines. tools/export_deploy_onnx.py exports the policy with a keypoint input (ref_kp) and a deployment YAML.

The RP1 sparse-keypoint policy (MLP, seed 0) in sim2sim. Only its command is drawn: the six target poses, placed in the robot's own pelvis heading frame as the policy reads them, and a red line from each of the robot's wrists and ankles to its target. Locomotion and Dance are the median clips of their sets; Extreme is the martial-arts clip of the videos below.

Sim2sim on held-out motions: zero-shot

Rollouts from the evaluator (MuJoCo, deterministic policy, default MLP encoder, seed 0). Loco, Mani, Dance and Ground show the clip with the median error of its set. Extreme shows two fast clips with clean references, chosen by us: a 12-second fist routine (faster than 63% of the Extreme clips; error 39 mm) and a jump (faster than 86%; 50 mm). The robot is solid, the reference a blue ghost, both at their global poses.

Why the fast Extreme clips are harder

The Extreme references are estimated from monocular video. We split the 308 clips into thirds by reference speed (95th percentile of the wrist and ankle speed) and measured every G1 rollout:

The robot is not late: it trails by one or two control steps, and shifting the reference by that much lowers the error by only 9%. Two other things grow with speed. In the fast clips the reference's feet slide over the ground at about 1 m/s while in contact, an artifact of the video estimate; the robot plants its feet, so its position drifts away from the reference and the ghost runs off. And the fastest swings exceed what the actuators deliver: the robot reaches 78% of the reference speed. Over all clips the error follows foot sliding (rank correlation 0.85) about as closely as speed (0.88). RP1 shows the same pattern (fastest third: 84% of the speed, 0.86 m/s sliding).

Speed on LAFAN: learned in 3–5 min

This comparison is on the LAFAN dataset (LAFAN-40: the 40 G1 LAFAN1 clips, used for both training and evaluation), XMimic's speed benchmark, not on the held-out test sets above. XMimic rebuilds BlinkMimic's main recipe in JAX and reports BlinkMimic's LAFAN-40 target in 0.91 min on four H100s. We ran its 1-GPU and 2-GPU (AL) trainers, BlinkMimic main and PartyMimic on our two RTX 4090s, under XMimic's protocol, 3 seeds each. PartyMimic keeps its recipe unchanged; it was not tuned for this benchmark. In the comparison above XMimic matches BlinkMimic (G1 mean error 60.2 vs 59.9 mm).

• Return 0.82. PartyMimic needs the fewest iterations (1,200–1,300; BlinkMimic 1,300–1,600; XMimic 1,400–1,500). On two GPUs it gets there as fast as XMimic (1.66 vs 1.71 min, 1.8× BlinkMimic). On one GPU XMimic's faster iterations make up for that (2.32 vs 2.45 min).
• Zero failures. PartyMimic is slower to the full target. One or two hard clips (dance1_subject1, jumps1_subject2) keep failing, so zero failures held over three checkpoints comes at 2,100–3,100 iterations (XMimic 1,900–2,400, BlinkMimic 1,600–1,800). BlinkMimic and XMimic run evaluation-guided probe sweeps that search for exactly these failures among the training clips. PartyMimic dropped them because they did not improve held-out tracking (see the recipe).
• Final quality. At the end of the 3,500-iteration budget PartyMimic's return is 0.883; the others reach 0.860–0.865.
• Where PartyMimic's speed comes from. On one GPU, the motion latents of all 32 imagined steps are computed in one batched pass, and every Adam update (the critic's and the Adam half of Muon) runs on one flat buffer. Together these cut the policy update from 86.5 to 78.4 ms and raise training from 7.65 to 8.2 it/s. With --pipeline, GPU 0 collects data and fits the world model while GPU 1 updates the policy and the critic on the previous iteration's data and world model, which gives 1.6× the 1-GPU throughput. The transfers between the GPUs run through pinned host memory on side streams and overlap the compute. The policy crosses as soon as its step is done, during the critic fit. Linear weights cross as bf16, which is exact because they only enter bf16 autocast matmuls. This matters here: the second 4090's PCIe link runs at x1 (about 1.6 GB/s measured), so a 36 MB fp32 policy alone would take over 20 ms to cross.

Protocol. LAFAN-40: the 40 G1 LAFAN1 clips, both for training and evaluation, with the same files as XMimic (checked against its SHA-256 list). Every run trains 3,500 iterations with a checkpoint every 100. Each checkpoint plays every clip once from its first frame per replica (4 replicas), without randomization or noise. A clip fails when the training termination triggers. BlinkMimic and PartyMimic checkpoints are scored by BlinkMimic's eval_batch.py, XMimic's by its JAX evaluator, which XMimic reports agrees with it to +0.001. Training minutes exclude the first log interval's excess over the steady interval for every method, by XMimic's tools/exclude_compile.py rule. That is compilation and CUDA-graph capture for the PyTorch trainers, and the first-execution overhead left in XMimic's clock, since XMimic compiles before its clock starts. The with-start-up column adds back each method's start-up with warm compilation caches (seeds 1 and 2): 0.17 min for BlinkMimic, 0.26 / 0.28 min for XMimic (its ahead-of-time compilation), 0.41 min for PartyMimic. The very first run on a machine compiles from scratch and takes longer: 3.3 / 2.0 min more for XMimic 1 GPU / AL, 1.9 min for the PartyMimic pipeline. Hardware: two RTX 4090 (48 GB), i9-14900K. The two-GPU methods share the x1 link of the second GPU. XMimic AL runs its probe process on the learner GPU under CUDA MPS, as in its README.

The PartyMimic recipe and what matters most

PartyMimic trains with BPTT: the policy gradient is the derivative of an analytic tracking reward through 32 imagined steps of a learned world model, closed by the critic's value gradient ∇_sV. The defaults of python -m partymimic.train are this recipe. Effects are the change when a factor is removed (G1, held-out Loc/Man error, ablations on the earlier 9,173-clip library).

Why SuperMimic works: full-factor ablation

Stage 1 of the study ablated SuperMimic itself, factor by factor, on our reimplementation (LAFAN-40, G1, 6,000 iterations per variant, one variant per factor). The score is the normalized evaluation return (mean of the last two evaluations, deterministic policy). The baseline over 3 seeds is 0.686 ± 0.048, and single-seed |z| < 1.5 counts as noise. Δ is the change from the baseline.

Conclusions: what dominates and why

• The multi-step gradient through an online-refit world model is the method. Keeping only the one-step model gradient (onestep, gwin1boot) gives a score of 0.00, and so does freezing the world model after 1,000 iterations (wmfreeze). With only one model step the policy follows the critic's ∇_sV, which nothing supervises. A frozen model is exploited as soon as the policy leaves the region where it is accurate.
• The horizon and the critic split the work. Without the critic (0.43), with H = 8 (0.43) or with H = 16 (0.60), results are clearly worse, while H = 48 or 64 brings no gain. Short chains rely on the inaccurate ∇_sV; longer chains only add variance. The terminal ∇V still matters (vstopgrad 0.54), but as a tail correction rather than the main signal.
• The closed-loop observation path carries most of the useful gradient (noobsgrad 0.52, gwin8 0.53). The gradient of how the next observation, and hence the next correction, depends on the state is what teaches feedback control.
• Model error becomes gradient error. A smaller world model (wm512 0.54) or less model fitting (wmb2 0.52) hurts directly; more fitting (wmb20) helps only slightly.
• The observation needs the reference relative to the robot (paperobs 0.48). Over long clips, yaw drift is otherwise unobservable.
• Optimization knobs: policy EMA (+0.09, 3 seeds) and a 2–3× higher learning rate help, and combined they give the best result (emalr6 0.83). These carry over to PartyMimic. Everything else (λ, critic epochs, action noise, UTD, replay size, reset distribution, survival weighting) is within noise.
• Caveat: turning off self-collision scores better here (+0.07, 3 seeds), but 33 of 40 LAFAN clips then fall on the leaderboard evaluator. Any factor that changes the simulator must be re-checked under the deployment evaluator. This is why PartyMimic trains on the evaluator's collision model.

Official SuperMimic choices on PartyMimic (full library)

After the official code opened, we ported its design choices as switches and tested each one on PartyMimic's full library (mix_kf, 10,731 clips, 12,000 iterations, G1). Score: mean body-position error over Loco/Mani, Ground, Dance, Extreme and XM14 (mm, lower is better). Official SuperMimic itself cannot run here (it needs the private mjlab_research); the ported row turns every official choice on and PartyMimic's extras off.

Only Muon weight decay clearly improves PartyMimic. Widening the reward kernel to 0.6 keeps the five-set mean within noise of the prior defaults and is the change that improves foot-slip and action smoothness under deployment-like perturbations (next section). Everything else from the official release is noise or worse on this library; dropping PartyMimic's failure / diversity reset shares hurts a lot.

Sim2real-oriented evaluation

G1 actuators, action scale and default pose match the public mjlab BeyondMimic model used by official SuperMimic; the gap people report on hardware is not from those. We score completion, body error, joint-target rate / jerk, torque saturation and foot slip under perturbations the training never saw (delay, PD scale, mass, push, low friction, soft foot contact). Same clips for every method.

Completion stays ≥96% under delay 20 ms, PD×0.75, push and friction 0.25 for every method. Switching the solver to the official 10 iterations changes nothing; soft-foot contact is where the prior recipe slips more than the ported official policy. The final recipe closes most of that gap while keeping tracking far ahead of the port (20.8 vs 33.8 mm nominal). Shipped checkpoints were trained with std_scale 0.35; retrain with the new defaults for the numbers above.

Use it

Instructions for data, training, export and evaluation are in the README.

• Data, from public datasets only: tools/data/build_library.sh g1 (LAFAN1, the BONES-SEED diversity subset, KungfuAthlete and the Extreme test set), then rp1 for the RP1 retargets.
• Default (MLP): bash scripts/train.sh logs/g1 --robot g1mj --motion_dir data/motions/g1_mix
• Attention encoder: add --ref_enc xattn (24 GB cards: also --pi_batch 1024).
• Sparse keypoints: add --interface sparse5; export with tools/export_deploy_onnx.py.
• Two GPUs: add --pipeline. Fine-tune on a few clips before deployment: --init <run>/last.pt.
• Trained checkpoints (MLP, joint references): checkpoints/g1_mlp.pt and checkpoints/rp1_mlp.pt.
• Diversity analysis of a library: tools/diversity/analyze.py.