8 posts tagged with "Testing"

A failing test gives you a stack trace and a line number. It does not give you the twelve steps that ran before it, the value two of them passed along, or the branch that was taken three calls earlier. So the first debugging session is usually a re-run with print statements — the same information, paid for twice.

Blong's Test runs console starts the run from inside the portal and shows it while it happens. The steps are a timeline, the dependencies are a graph, and every assertion sits beside the value the step returned. The console is a page in the application, not a separate report.

A Storybook story is the cheapest place to look at a screen and the most expensive place to trust one. The model page renders perfectly in the story, and in the application it fails — because the story's rows came from a fixture file and the gateway method the page calls does not exist. The toolbar that used to solve this in the previous, webpack-based setup has been rebuilt for Blong's Vite Storybook, and a realm now gets it in one line:

export default defineBlongStorybookPreview(browser, {backend: true});

That line brings five toolbar items — Backend, Role, Theme, Language, Direction — and the machinery behind them. This post is about the two interesting ones.

A failing test that says expected true, got false is a puzzle. It tells you the assertion, not the journey: which step of twelve ran, which branch was taken, what the values were when the decision was made. So the first thing anybody does with a red build is re-run it locally with print statements — paying for the same information twice.

Blong's handlers can announce where they are. A checkpoint is a named moment with data; a decision is a branch with its candidates. Both travel with the work, both reach the report, and both are free in production — which is what makes them worth writing into the code rather than into a test.

Read a hundred browser tests from a system that has been running for a few years and most of them are not about the product. They are about the plumbing: a helper that signs in, a helper that navigates to the right page, a helper that waits for the table to fill, and — the one that hurts — a copy of the field names, so that the day a form is redesigned, forty tests fail for a reason unrelated to the feature each one was protecting.

Blong's Playwright layer takes that plumbing away and puts a floor under it, so a test file is close to a statement of intent. The fixtures log in; the CRUD helpers find the page, its widgets and its fields from the model the server already declares; and the primary assertion is a picture rather than a selector.

Assertions are cheap to write and expensive to keep. A test that checks four fields of a response is four lines, and the fifth field — the one nobody thought to check — is the one that breaks in production. Multiply that by every handler in a framework and the test suite becomes a description of what the code used to do, maintained by hand, field by field.

Snapshot testing inverts it: capture the output once, commit it, and let a diff be the review. The discipline is in what you capture and how narrowly, and that is what this post is about.

The oldest argument in testing is whether the specification or the test is the source of truth. When they are two artifacts, the answer is neither: the specification describes what the system should do, the test asserts what it does, and the difference between them is discovered in production. When they are one artifact, the question disappears.

In Blong that artifact is a Gherkin scenario, parsed by the real @cucumber/gherkin parser and compiled into the same chain steps every other test uses. A business rule and its automated check are one file that a domain expert can read.

Every test suite decides how much of reality it is willing to pay for, and most suites make that decision by accident: the first test reaches for a real database because it was there, the next one stubs a client because the real one was slow, and two years later nobody can say what the suite actually proves. The tests pass; the question "would this catch it?" has no answer.

Blong has three levels of backend fidelity, and the difference between them is what each one can prove. What matters is that the level is chosen by the environment — the same test, unchanged, runs at all three.

A test is a story, and stories are linear. So test code is linear: set up a user, then fetch the account, then build the report, then assert. The runner then executes that story one line at a time, and a suite of such stories takes as long as the sum of everything in it — even though the four lines that set up four unrelated fixtures had no reason to wait for one another.

The usual fixes are both unsatisfying. You can declare the dependencies by hand and watch the declarations rot as the test changes, or you can run everything concurrently and watch the suite fail in ways that depend on timing. Blong took the third option: the executor discovers the dependencies from the test's own data flow and parallelises everything the data flow permits.