Methodology post. Every claim here is implemented in the SAC package's README (its 'what is reproduced / what is not claimed' sections), linked below.
Draft. Methodology post; every claim here is implemented in the SAC package's
README (what is reproduced / what is not claimed sections).
"Reproduced" is the most abused word in ML tooling. This is the checklist a
reproduction package should satisfy before it uses the word, and the four
honesties that separate a useful package from marketing.
The checklist
- Name the environment version.
HalfCheetah-v1andHalfCheetah-v4are different simulators with different contact models. A comparison that does not name versions is not a comparison. - State the compute. Steps, seeds, wall clock, and hardware. "We trained for 1M steps" means nothing without the wall clock: it tells the reader whether they can afford to check you.
- Every number in the README is generated. The comparison table is written by a script from run logs, never typed. If the table and the logs can disagree, they will.
- Show the failure budget. A seed that crashed and was restarted, a run that diverged and was re-run — say so. A reproduction with zero failures is usually a reproduction with hidden selection.
The four honesties
- Scale vs exactness. For RL baselines, the honest claim is usually "same algorithm, same hyperparameters, comparable return within the same regime" — not "bit-identical to Table 1". Say which one you are claiming.
- Reference spread. The original paper's numbers and a modern reimplementation's numbers often differ. Cite both so the reader can see the spread your run should land inside.
- What was NOT reproduced. The most useful section of a reproduction README is the one that says what the package does not claim. It is also the section that makes the rest believable.
- One seed is a demo, three is evidence. Label which one you have. Do not present a single run as an average.
Why this sells
Buyers of reproduction packages are not paying for the code — they are paying for
not having to debug someone else's README. The checklist above is the difference
between a package that saves an afternoon and one that costs a weekend.
This post's companion implementation is https://gitlab.com/quilldatafix/sac-mujoco-repro; its README follows the
checklist above, and its tables are generated by scripts/plot.py from run CSVs.
The package this describes (code, logs, checkpoints, MIT): sac-mujoco-repro · More repro kits: Datafix Tools