Test quality standards

A test that cannot fail is worse than no test. It costs review time, claims coverage it does not provide, and turns “the suite passes” from evidence into noise. On a car that can hurt someone, that is a safety problem. The full standard, with examples from the workspace, is TEST_QUALITY_STANDARDS.mdGitHub.

Rule 1: prove every new test can fail

Before calling a new or changed test done, break the code it covers and watch it fail.

1. Replace the body of the unit under test with pass, with return <the exact value the test expects>, or with raise NotImplementedError.
2. Run only that test.
3. It must fail. If it still passes, it is detecting nothing. Rewrite it.
4. Restore the code, run the test again, and confirm it passes. Never commit the break.

return <the expected value> is the case most often skipped, and the one that exposes a test that has memorized an answer instead of checking a computation. For mathematics, geometry and control code, each of these changes must also make at least one test fail:

Change	Shows that the test checks
Flip a comparison, < to <=	The boundary, not only the interior
Negate a sign	Direction: left against right
Change a constant by 10%	That the value matters
Swap two adjacent arguments of the same type	Argument order, a common pose bug
Return an input unchanged	That a computation happens
Delete a clamp	That a limit is enforced

Line coverage is not evidence: a test can run every line of racing_math.py and assert nothing.

Rule 2: never weaken a test to pass the suite

A failing test is a finding. Fix the code. Do not widen a tolerance, weaken an equality to a bound, drop the case that failed, add skip or xfail, or delete the test, and never do any of these in the same change as an edit to the code under test. If a test seems wrong after a change of specification, say what it asserted, what the new behaviour is, where that behaviour is documented, and why the old behaviour was wrong.

Reject a test that

Code	Pattern
A1	Asserts on a mock’s own return value. Mock only a real process boundary, never the workspace’s own ROS-free modules.
A2	Would still pass against a stub: checks only a type, a shape, or that nothing raised.
A3	Hard-codes today’s output with no independent source for the expected value
A4	Covers only the normal case. Required edges include zero, negative, the boundary, NaN and inf, and empty input; scan consumers also need all-inf, all-zero and wrong beam counts.
A5	Asserts something always true, swallows exceptions, or uses pytest.approx without a stated tolerance and unit
A6	Was loosened, skipped or deleted because it failed
A7	Takes its verdict from the thing under test, such as F1TENTH Gym’s own collision flag
A8	Tests only the permissive side of a safety check: “LB held, the car drives” is the half that fails safely
A9	Is not run by the command people use, or is hidden by a collection error

Every expected value traces to a closed form recomputed in the test, an invariant, a cited specification, or a recorded measurement, and the test says which. A value with no such source is labelled # change-detector (not an oracle), and the same unit gets a real test too.

The deadman requirement

Every node that can publish /drive, /ackermann_cmd or /commands/* needs its own four refusal tests: LB held, LB released, /joy never received, and /joy stale. They are listed in Write a driving node.