arXiv:2509.19573v2 Announce Type: replace Abstract: Achieving highly dynamic behaviors on humanoid robots, such as running, requires controllers that are both robust and precise, and hence difficult to design. Classical control theory offers principled tools for stabilizing such systems but struggles to synthesize real-time controllers for their nonlinear, hybrid dynamics; reinforcement learning (RL) handles these dynamics well but typically depends on extensive hand-tuning of heuristic rewards. In this work, we embed ideas from nonlinear control theory, specifically control Lyapunov functions (CLFs), along with optimized reference trajectories for the hybrid system into the reinforcement learning reward. This approach, CLF-RL, replaces the need to handcraft and tune heuristic reward terms with easy to tune stability informed rewards. We further develop a trajectory optimization routine that can handle the hybrid structure of the system to produce dynamically feasible, multi-domain, trackable references; using full-order dynamics rather than a kinematic approximation yields up to a 1.7x improvement in position tracking. The resulting policy runs on a Unitree G1 humanoid with a full flight phase, operating reliably in tight corridors such as a treadmill as well as in outdoor environments, all while remaining robust to disturbances applied to the torso and feet. Moreover, it achieves accurate global reference tracking utilizing only on-board sensors, making a critical step toward integrating these dynamic motions into a full autonomy stack.