No tool in the repository checks that a change keeps every test name and result #227

New Issue

2026-10-16T01:56:26Z

cgalo5758 commented

2026-10-16 01:56:26 +00:00

What a person cannot do today

Confirm, from tracked files, that a change runs the same tests with the same results as its parent commit. go test passing says only that nothing failed. A change that deletes a subtest, renames a test, folds a table's named subtests into one loop, or makes a test skip leaves the run green; every database test skips itself when TEST_DATABASE_URL is unset (docs/testing.md:30-32). The refactor in #166 checked every commit by comparing the go test -json output of the parent and of the commit, with scripts that were never committed.

What they should be able to do

Run one command over two go test -json outputs, one taken at the parent commit and one at the change, and get:

every test and subtest that ran at the parent and not at the change, and every test whose result changed between pass, fail and skip; each fails the command unless the contributor lists it as an expected rename or removal;
every test new at the change, listed without failing;
every skip whose message changed, because a test that now skips for another reason no longer runs what it used to;
each package's result, and the count of each result on both sides.

The command reduces each output to one line per package and test name, subtests included, holding the last pass, fail or skip the stream reports for it. A skipped test keeps its output lines, without the === and --- headers, as its skip message.

Both runs are taken the same way: the test databases reset with test/reset-test-db.sh, test/.env loaded, the app in the same state (down, so the walkthroughs that need it skip on both sides), then go test -json -count=1 -p 1 over the same packages in the same order. The reset is part of the check: tests share one database (#192) and some sweeps read every row in it, so a run on a used database can differ for reasons the change did not cause.

Why it matters

scripts/mutation-check answers a different question. It applies breaks someone wrote down, one at a time, and reports whether the tests its list names catch each one. A test outside the list is never looked at, a list has to be written for each area, and nothing in it notices that a subtest disappeared from a table or that a test now skips. The names-and-results comparison needs no list, covers every test in the packages it runs, and was the check every refactor commit used. Without it in the repository, a contributor who refactors cannot repeat that check, and a reviewer cannot ask for its output.

Where

scripts/, beside scripts/mutation-check and scripts/sqlc-unused; docs/testing.md, next to the deliberate-break rule ("Conventions & gotchas").

Done when

A Go command under scripts/ reduces a go test -json stream as described above and compares two reductions. It exits non-zero on a missing test, a changed result or a changed skip message that is not listed as expected. Its header gives the usage, as scripts/mutation-check/main.go does.
A skip message that names something each worktree sets differently, such as the app's URL with the port test/bootstrap-stack.sh chose, compares equal across worktrees.
Its tests feed it small recorded go test -json streams: a removed subtest, a pass that became a skip, a changed skip message, and a renamed test listed as expected.
docs/testing.md says when to run it (a change that moves tests, or reorganizes the code under them without changing what it does) and how to take the two runs.

## What a person cannot do today Confirm, from tracked files, that a change runs the same tests with the same results as its parent commit. `go test` passing says only that nothing failed. A change that deletes a subtest, renames a test, folds a table's named subtests into one loop, or makes a test skip leaves the run green; every database test skips itself when `TEST_DATABASE_URL` is unset (`docs/testing.md:30-32`). The refactor in #166 checked every commit by comparing the `go test -json` output of the parent and of the commit, with scripts that were never committed. ## What they should be able to do Run one command over two `go test -json` outputs, one taken at the parent commit and one at the change, and get: - every test and subtest that ran at the parent and not at the change, and every test whose result changed between pass, fail and skip; each fails the command unless the contributor lists it as an expected rename or removal; - every test new at the change, listed without failing; - every skip whose message changed, because a test that now skips for another reason no longer runs what it used to; - each package's result, and the count of each result on both sides. The command reduces each output to one line per package and test name, subtests included, holding the last `pass`, `fail` or `skip` the stream reports for it. A skipped test keeps its output lines, without the `===` and `---` headers, as its skip message. Both runs are taken the same way: the test databases reset with `test/reset-test-db.sh`, `test/.env` loaded, the app in the same state (down, so the walkthroughs that need it skip on both sides), then `go test -json -count=1 -p 1` over the same packages in the same order. The reset is part of the check: tests share one database (#192) and some sweeps read every row in it, so a run on a used database can differ for reasons the change did not cause. ## Why it matters `scripts/mutation-check` answers a different question. It applies breaks someone wrote down, one at a time, and reports whether the tests its list names catch each one. A test outside the list is never looked at, a list has to be written for each area, and nothing in it notices that a subtest disappeared from a table or that a test now skips. The names-and-results comparison needs no list, covers every test in the packages it runs, and was the check every refactor commit used. Without it in the repository, a contributor who refactors cannot repeat that check, and a reviewer cannot ask for its output. ## Where `scripts/`, beside `scripts/mutation-check` and `scripts/sqlc-unused`; `docs/testing.md`, next to the deliberate-break rule ("Conventions & gotchas"). ## Done when - A Go command under `scripts/` reduces a `go test -json` stream as described above and compares two reductions. It exits non-zero on a missing test, a changed result or a changed skip message that is not listed as expected. Its header gives the usage, as `scripts/mutation-check/main.go` does. - A skip message that names something each worktree sets differently, such as the app's URL with the port `test/bootstrap-stack.sh` chose, compares equal across worktrees. - Its tests feed it small recorded `go test -json` streams: a removed subtest, a pass that became a skip, a changed skip message, and a renamed test listed as expected. - `docs/testing.md` says when to run it (a change that moves tests, or reorganizes the code under them without changing what it does) and how to take the two runs.

cgalo5758 added the

kind

enhancement

area/testing area/meta

priority

medium

labels 2026-10-16 01:56:26 +00:00

Sign in to join this conversation.