Cloud Parts

Automated testing

Code that checks other code: a runner finds the tests, runs them and reports what failed, so every change is checked in seconds instead of by hand.

4 technologies on this page ↓

How it works

A test is a small function that sets up an input, calls the code under test and asserts on the result. A test runner finds tests by naming convention (pytest collects test_*.py files and test-prefixed functions; Vitest collects *.test.ts and *.spec.ts files), runs them, and exits with a non-zero code if any assertion fails, so a CI job can block the merge. Runners usually run test files in parallel, in separate processes or worker threads so one file can’t leak state into another. Around that core sit helpers: mocks that stand in for a mailer or an API, snapshots that save an output to a file and fail when it changes, coverage reports, and a watch mode that reruns only the tests a saved file affects. Two layers build on the runner. A test environment manager such as tox or nox runs the same suite once per Python version or dependency set, each in a fresh virtual environment. A component workshop such as Storybook renders each UI component in isolation, one state per “story”, and turns those stories into tests in a real browser.

Think of it as

A pilot’s pre-flight checklist. The same items, in the same order, every time, quick enough that nobody skips it. Most checks are on single parts (unit tests); only a few are full test flights (end-to-end tests), because those are slow and expensive.

Reach for it when

  • Code that several people change, or that you’ll change again in six months
  • Libraries that promise to work on several language or dependency versions: run the suite once per version
  • A bug you just fixed: reproduce it in a low-level test first, so it stays fixed
  • UI components with many states (empty, loading, error, 1,000 rows) that are slow to reach by clicking through the app

Think twice when

  • Throwaway scripts and spikes you’ll delete this week
  • Leaning mostly on end-to-end tests through the UI: they are brittle, expensive to write and slow to run
  • Snapshot tests nobody reviews: accepting a new snapshot accepts whatever changed
  • Living with flaky tests: once people learn to dismiss red builds as noise, they dismiss real failures too

A real example

A team maintains a Python client library that supports Python 3.10, 3.11 and 3.12, plus a React dashboard. For the library, one tox.toml lists the three versions and a lint environment. tox run builds the package, installs it with pytest into a fresh virtual environment per version and runs the same tests in each: about 12 seconds on a laptop for a tiny suite, almost all of it environment setup. For the dashboard, Vitest runs the logic tests in about 100 ms and, in watch mode, reruns only the files a save affects. Each component state is a Storybook story, and the Vitest addon runs every story as a test in headless Chromium. CI runs all three and blocks the merge if any exits non-zero.

Kinds of automated testing

Specific products, and what each does differently.

Also used for automated testing

Products of other kinds that are often used this way, grouped by what they are.

The test pyramid#

For a long time, test automation meant tools that recorded clicks through an application’s user interface and played them back. Martin Fowler describes where that leads: tests that run end-to-end through the UI are brittle, expensive to write and slow to run, and a small change to the app can break many of them at once. A suite made mostly of such tests is the “ice-cream cone”.

The test pyramid turns it the other way up: many fast, focused unit tests at the bottom, fewer tests through a service or API layer in the middle, and only a few broad UI tests on top. Mike Cohn made the idea known as the “Test Automation Pyramid” in the 2009 book Succeeding with Agile, after first drawing it around 2003–2004. Fowler adds two caveats. The shape assumes broad tests are slow and brittle, so if yours are fast and reliable you need fewer low-level ones. And when a high-level test finds a bug, write a unit test that reproduces it before fixing it, so the bug stays dead.

Component tests sit between the layers. Storybook pitches them as the middle ground: the in-browser fidelity of an end-to-end test, with the scope of a unit test, since a component can still be given mocked data.

Flaky tests#

A flaky test passes and fails on the same code. Common causes are concurrency, reliance on undefined or non-deterministic behaviour, flaky third-party code and infrastructure problems. Google reported in 2016 that about 1.5% of all its test runs gave a flaky result, that almost 16% of its tests showed some flakiness, and that about 84% of the pass-to-fail transitions its CI saw involved a flaky test.

The cost is trust. With 1,000 tests and a 1.5% flaky rate, about 15 fail on any run for no reason, and people start dismissing real failures as noise. Automatic retries cut the noise but encourage people to ignore flakiness, and they delay real breaks: Google’s option to report a failure only after three failures in a row means a broken 15-minute test isn’t noticed for 45 minutes.

In code · python

# test_pricing.py: run with `pytest -q`
from unittest.mock import Mock

import pytest


def total(items, coupon=None):
    s = sum(price * qty for price, qty in items)
    return round(s * 0.9, 2) if coupon == "TEN" else s


# one test function, three cases: arrange the input, act, assert the result
@pytest.mark.parametrize(
    ("items", "coupon", "expected"),
    [
        ([(2.5, 4), (1, 3)], None, 13),
        ([(20, 1)], "TEN", 18),
        ([], None, 0),
    ],
)
def test_total(items, coupon, expected):
    assert total(items, coupon) == expected


def test_receipt_is_sent():
    send = Mock()  # a test double stands in for the real mailer
    send(total([(5, 2)]))
    send.assert_called_once_with(10)

# $ pytest -q
# ....                                       [100%]
# 4 passed in 0.02s
# change the last case to expect 1 and the runner names it and exits non-zero:
# FAILED test_pricing.py::test_total[items2-None-1] - assert 0 == 1

Often confused with

CI/CD pipelinesLinters, formatters and type checkersLocal cloud emulators
Where it came from1997–2025

Automated unit testing as most programmers know it began in Smalltalk. Kent Beck built small frameworks so that Beck and Beck’s clients could write tests in their normal Smalltalk environment and run a subset or the whole set quickly after every change. Beck preferred each team to build its own, which took a couple of hours. In 1997, on a flight from Zurich to the OOPSLA conference in Atlanta, Beck and Erich Gamma pair-programmed the first version of JUnit, test first. JUnit spread the idea with Extreme Programming and test-driven development, and ports followed for almost every language, CppUnit perhaps first; the family became known as xUnit.

As suites grew, the question became which tests to write. Mike Cohn sketched the test automation pyramid with Lisa Crispin around 2003–2004 and published it in 2009: many unit tests, fewer service tests, very few slow UI tests.

Later tools took the same loop in new directions. tox (2010) ran a Python package’s tests across several interpreters in virtualenvs. React Storybook (2016) rendered UI components in isolation, and from 2025 ran those stories as tests through Vitest, a runner first committed in December 2021 that reuses the Vite build pipeline.

Why it’s built this way

Tests written in the same language as the code
Beck’s focus was making it easy for programmers to define tests in their regular environment and run a subset or the full set quickly, so Beck’s teams ran the unit tests after every change.
Many small tests, few broad ones
Tests driven through the UI were brittle, slow and expensive to write. Pushing most checks down to unit tests keeps the suite fast and stable enough to run on every change.
A fresh environment per version
A library has to install and work on every Python version it claims. tox’s first release described itself as checking that a package installs correctly with different Python versions and interpreters.
Stories as test cases
A component workshop already lists every state of the UI, so adding assertions to those states lets the suite cover thousands of states with little extra code.

Timeline

  1. 1997

    Beck and Gamma write the first JUnit on a flight to OOPSLA

  2. 2004

    Mike Cohn describes the test automation pyramid at a Scrum gathering

  3. 2009

    Cohn’s Succeeding with Agile publishes the pyramid

  4. 2010

    tox 0.5 runs a package’s tests across Python interpreters in virtualenvs

  5. 2016

    Kadira launches React Storybook

  6. 2021

    First commit of Vitest, a test runner built on Vite

  7. 2025

    Storybook 9 runs every story as a Vitest test in a real browser

Sources14

Improve this page on GitHub