I have been recording how GNU coreutils 9.12 behaves, then replaying the same cases against uutils, the Rust rewrite. The recorder is behave: a Go CLI (and MCP server) that probes flags from --help, odd stdin, missing files, and other corners, then diffs stdout, stderr and exit codes.

So far that work has led to about 30 bug reports and 37 fix PRs upstream. 27 of those PRs are merged. Here is what those merges taught me about black-box testing a rewrite, with concrete examples.

1. Match the reference release, not "whatever is on PATH"

CI on Ubuntu often ships an older GNU than the release the rewrite targets. I record from GNU 9.12 because that is what uutils aims at. A case that is correct on 9.12 can fail on 9.4 (for example how wc or expand counts a multi-byte character). Specs that depend on a version gap stay out of CI, or they teach people to ignore failures.

Lesson: pin the reference binary. Diffing against a random system coreutils produces noise that looks like product bugs.

2. Option precedence is where rewrites disagree quietly

uutils#15381 (merged): with both --check and --merge, GNU sort lets --check win. The rewrite preferred --merge. No crash, just the wrong mode.

uutils#16114 (merged): seq treated an option-looking token before the first operand as data, so seq -inf 1 could loop. GNU parses those as options.

uutils#15529 (merged): bare tail -c and obsolete +NUM under _POSIX2_VERSION did not match GNU.

These are not exotic. They show up as soon as a probe walks every flag pair and every help-listed value. Unit tests written from the author's mental model rarely hit "both flags at once" or "option-shaped operand".

3. Limits and overflows hide in "it worked for small inputs"

uutils#16181 (merged): base64 -w with a width too large to represent still tried to wrap. GNU turns wrapping off.

uutils#16191 (merged): fmt could emit a line past WIDTH when no break fitted. GNU never does.

Black-box recording does not need to invent these numbers. Help text and a few large or awkward values are enough. The failure is a wrong line of stdout, which a golden case catches on the first replay.

4. Error context is part of the contract

I wrote a longer piece on this earlier. The short version from merges that landed since:

  • uutils#15692: cp now names the source when stat fails for a reason other than "missing".
  • uutils#15714: ln names the source it cannot access.
  • uutils#15868: chown/chgrp say cannot access unless the operand is a dangling symlink.
  • uutils#15825: mv -n on the same file skips the same-file error the way GNU does.

Scripts and humans parse those messages. A rewrite that returns the right exit code with the wrong words still breaks tooling.

5. Checksums and parsers have "almost right" modes

uutils#16167 (merged): checksum tools unescaped names only on lines that start with a backslash, matching GNU. A broader unescape looked fine in happy-path tests and failed on real check files.

Golden cases from the reference tool encode the narrow rule without anyone writing a paragraph about it.

What I would repeat on the next rewrite

  1. Record from the release you claim compatibility with. Keep version-sensitive cases out of the shared CI job.
  2. Let the probe walk help text. Enum values, flag pairs, and obsolete forms find precedence bugs that hand-written tests miss.
  3. Assert stderr shape, not only exit codes. Context words are user-visible API.
  4. File a fix with a test. Upstream reviewers move faster when the PR fails without the patch and passes with it.
  5. Keep the README honest. Counts (reports, PRs, merges) change; update them when you publish.

Try the same specs 

brew install emirmrkaljevic/tap/behave # or: go install github.com/emirmrkaljevic/behave/cmd/behave@latest behave run examples/seq behave diff examples/fmt --impl /path/to/uutils/fmt

The specs under examples/ were recorded from GNU 9.12. The merged uutils PRs above are linked from the behave README findings table if you want the full list.