Should a generated-mutant report run over the packages whose tests need no database? #228

New Issue

2026-10-16T01:56:27Z

cgalo5758 commented

2026-10-16 01:56:27 +00:00

The question

scripts/mutation-check applies breaks a contributor writes by hand, so it measures only the facts someone thought to break, and only for the tests its list names. A mutation tool generates the breaks itself (a flipped comparison, a removed statement, a changed constant) and lists every one that no test catches. Would a report from such a tool, run now and then and never failing a build, find weak tests sooner than hand-written lists do?

The candidates are the packages whose tests need no database, so a run needs no test stack: internal/auth, internal/stripetest, internal/lint, internal/forms, internal/config and internal/workflows/domains. Two test files in internal/auth (callback_db_test.go, callback_edges_test.go) use the database and skip without one, so a run there would report the mutants only they catch as missed.

What depends on the answer

Whether weak tests in those packages are found by a machine or only when someone writes a break list for them. How much triage a report costs, since some generated mutants change nothing a caller can observe and no test can catch them. Whether the report waits for automated checks (#14) or is run by hand. The hand-written breaks run so far found weak assertions and code no caller could observe, and never a production defect, which bounds what to expect.

Options known so far

Measure once: run a Go mutation tool (gremlins and go-mutesting are two) over the six packages, count the missed mutants, and sort them into weak tests and changes no caller can observe. Decide from the numbers.
A periodic, non-gating report over the same packages, with a list of accepted survivors and the reason for each, so each report shows only what is new.
Nothing: keep the hand-written breaks docs/testing.md asks for ("Prove a test with deliberate breaks").

Decided when

The project records which tool runs over which packages, how often, where the report goes, and how a missed mutant becomes a test or an accepted entry; or it records that it runs none, and why.

## The question `scripts/mutation-check` applies breaks a contributor writes by hand, so it measures only the facts someone thought to break, and only for the tests its list names. A mutation tool generates the breaks itself (a flipped comparison, a removed statement, a changed constant) and lists every one that no test catches. Would a report from such a tool, run now and then and never failing a build, find weak tests sooner than hand-written lists do? The candidates are the packages whose tests need no database, so a run needs no test stack: `internal/auth`, `internal/stripetest`, `internal/lint`, `internal/forms`, `internal/config` and `internal/workflows/domains`. Two test files in `internal/auth` (`callback_db_test.go`, `callback_edges_test.go`) use the database and skip without one, so a run there would report the mutants only they catch as missed. ## What depends on the answer Whether weak tests in those packages are found by a machine or only when someone writes a break list for them. How much triage a report costs, since some generated mutants change nothing a caller can observe and no test can catch them. Whether the report waits for automated checks (#14) or is run by hand. The hand-written breaks run so far found weak assertions and code no caller could observe, and never a production defect, which bounds what to expect. ## Options known so far - Measure once: run a Go mutation tool (gremlins and go-mutesting are two) over the six packages, count the missed mutants, and sort them into weak tests and changes no caller can observe. Decide from the numbers. - A periodic, non-gating report over the same packages, with a list of accepted survivors and the reason for each, so each report shows only what is new. - Nothing: keep the hand-written breaks `docs/testing.md` asks for ("Prove a test with deliberate breaks"). ## Decided when The project records which tool runs over which packages, how often, where the report goes, and how a missed mutant becomes a test or an accepted entry; or it records that it runs none, and why.

cgalo5758 added the

labels 2026-10-16 01:56:27 +00:00

cgalo5758 referenced this issue

2026-10-16 04:59:29 +00:00

Should pure functions get property-based tests beside their examples? #235

Sign in to join this conversation.