Numbers from one CPU-only box (16 cores), PyTorch 2.15+cpu, a from-scratch ~300-line SAC, seed 0. Every number comes from the eval.csv logs shipped in the repo below.
Draft for dev.to. Numbers from one CPU-only box (16 cores), PyTorch 2.15+cpu,
my own ~300-line SAC, seed 0. Filled from runs/Pendulum-v1_s0/eval.csv.
Before you trust any RL implementation on an expensive task, run it on a cheap one
where failure is loud. On continuous control that task is still Pendulum-v1, and it
costs pocket change on CPU.
The test
- Train your algorithm on
Pendulum-v1for 100k steps with your standard hyperparameters. - Watch the deterministic-policy evaluation every 10k steps.
- Read the shape, not just the endpoint.
What healthy looks like
Measured on my implementation, seed 0, one gradient step per environment step
(runs/Pendulum-v1_s0/eval.csv):
| steps | eval return (10 episodes, deterministic) |
|---|---|
| 10k | -123 ± 57 |
| 20k | -120 ± 53 |
| 50k | -148 ± 52 |
| 70k | -197 ± 98 |
| 80k | -216 ± 71 |
| 100k | -168 ± 78 |
That is the honest shape of a healthy but noisy run: it converges into the
healthy regime early and then oscillates inside it. the fix is not more steps of
the same thing; it is the standard tweak — two gradient steps per environment
step — which trades ~2x wall clock for a better final return (-111 vs -168
here) at comparable variance. (Both runs ship in the package so you can
compare the two curves yourself.)
How to read failures
- Flat around -1,000: the policy gradient is not flowing — check that you are backpropagating through the sampling (rsample), not
.sample(). - Improves then collapses to -1,000: the entropy temperature collapsed to zero and the policy froze; check
target_entropyand the alpha loss sign. - Noisy but trending to -150 ± 20: fine. Pendulum's practical target is about -150; scores near -130 are already at the ceiling of this task.
- NaNs: tanh correction log(1 - tanh²) needs an epsilon; without it the log blows up at saturated actions.
Why Pendulum and not CartPole
CartPole is discrete; it will not test your squashed-Gaussian machinery, which is
where almost every SAC bug lives. Pendulum is continuous, badly scaled, and its
optimal return is comfortably below zero — so an implementation that "looks
trained" because numbers are increasing can still be broken, and an
implementation that converges near -150 has exercised the whole pipeline:
rsample, tanh correction, twin critics, target Polyak averaging, and learned
temperature.
The 30-minute budget
With 256x256 networks, batch 256 and one gradient step per environment step, the
full 100k-step Pendulum run took about 14 minutes on 16 CPU cores (838 s, wall
clock, measured). There is no excuse for skipping this check before a GPU run.
The SAC implementation and all logs referenced here ship in
https://gitlab.com/quilldatafix/sac-mujoco-repro; every number above is reproducible with one command.
Code, training logs and checkpoints (MIT): sac-mujoco-repro · Ready-to-run packages: Datafix Tools