Your CLI golden tests run in an empty directory (and that hides bugs)

I've been building behave, a tool that probes a command-line program as a black box (its --help, every flag, flag pairs, every value the help lists, odd stdin, bad values), records what it does, and replays those cases against another implementation. The test bed is GNU coreutils vs uutils, the Rust rewrite.

Last week ls and du looked suspiciously good. All 48 generated ls --sort=…, --format=…, --time=… value cases passed against uutils. The reason was boring: every case ran in a fresh, empty temp directory. ls --sort=size on nothing prints nothing, in both implementations.

Give directory tools something to look at

behave now drops a small tree into the work dir when a tool's help text talks about "the current directory" or recursion (ls, du, rg, grep -r):

.hidden  README.md  a.txt  data.csv  file10.log  file2.log  sub/nested.txt

The names, sizes and extensions are picked so that sorting by name, size, version (file2 vs file10), extension and time all give different orders. --sample-files on|off overrides the guess.

Making ls -l and ls -t deterministic

Real files bring real timestamps, and golden tests hate those. Three things made the recordings stable:

1. Fixed mtimes. Every input file gets a fixed modification time, one hour apart in name order, so ls -t has a real order and ls -l prints the same date every run.
2. A private parent directory. ls -a lists ... If the work dir sits directly in /tmp, .. is /tmp, whose link count and mtime change whenever anything else on the machine creates a temp dir. Each case now runs in <tmp>/behave-XXXX/work, and both directories get fixed times.
3. Write files in reverse name order. You can't set ctime or birth time. Files created within the same clock tick tie, and ls -c then falls back to name order; files that straddle a tick come out newest first. Creating the files in reverse name order makes "newest first" and "name order" the same list, so ls -c and ls --time=birth print the same output either way.

After that, 500 generated ls cases replay 15 times in a row against GNU ls with zero failures. The 8 cases marked flaky are the --inode ones, which is honest: inode numbers really do change.

What it found

Against uutils 0.12.0, with stderr wording ignored, the interesting new differences were in du's grand-total line:

$ du --total --separate-dirs        # GNU
28      .
36      total
$ du --total --separate-dirs        # uutils
28      .
28      total

GNU's total still adds up every directory it measured (the sub dir's 8 plus the top's 28), uutils only counts the top level. And du --total --time prints the time column on the total line in GNU but not in uutils. Both turned out to be already listed in uutils' fuzzer tracking issue (#15254) but had no fix yet, so I sent one: #15455, currently in review.

One difference I did not report: ls --hyperlink ends its OSC 8 links with ESC \ in uutils and with BEL in GNU 9.7. Both are valid terminators and uutils chose ESC \ on purpose, so behave's ansi normalizer now strips both forms instead of flagging it.

Earlier the same approach found a sort --check --merge bug in uutils; that fix (#15381) is merged.

Try it

All of this is in v0.1.5.

go install github.com/emirmrkaljevic/behave/cmd/behave@latest
behave probe du --out behave-du
behave diff behave-du --impl /path/to/other/du --ignore-stderr

It's MIT, a single Go binary, and also runs as an MCP server if you want a coding assistant to drive the probing. Issues and ideas welcome on GitHub.

I used AI assistance while writing this post; every number above comes from runs on my machine with GNU coreutils 9.7 and uutils 0.12.0.