UFO: A General Unsupervised Reinforcement
Learning Framework for Humanoid COntrol

Motivation

Tracking-based control remains the dominant paradigm for humanoid robots, while unsupervised RL offers a complementary path toward general skill acquisition, with strong potential in disturbance robustness, data efficiency, skill generalization, and behavior composition.

Yet progress has been slowed by high training complexity, substantial engineering overhead, and the absence of unified infrastructure, which has limited broader follow-up work. UFO aims to lower this barrier, enabling wider exploration of unsupervised RL and accelerating the development of humanoid behavioral foundation models.

Highlights

Highlight 1 - Fast Training + Better Performance

Improves the training efficiency of FB learning, reducing training time to 6-8 hours while achieving better performance.

Highlight 2 - New Unsupervised RL Algorithm

Adds TeCH, a contrastive temporal-distance representation learning algorithm, showing support for URL beyond FB.

Highlight 3 - General & Extensible Framework

Provides a general framework supporting diverse robots and multi-dataset training, enabling high-dynamic skill injection.

Highlight 4 - Robust Real-world Teleoperation

Supports real-world teleoperation and demonstrates robust policy execution under external disturbances.

Highlight 2: New Unsupervised RL Algorithm

Core idea: Learn a latent space where distances reflect temporal reachability.

TeCH jointly learns:

• Temporal representations that preserve trajectory structure.
• A goal-conditioned policy driven by unified progress rewards.

How does TeCH learn?

• Explore the environment to collect diverse transitions.
• Construct pseudo-goals via temporal rolling.
• Learn temporal representations through contrastive learning.