Running Tests in CI: What Runs Where

Tests that don't run automatically don't protect you — someone forgets, or skips them under deadline, and the coverage you built is worth nothing that week. This part is about wiring testing into CI so it catches regressions without becoming the thing everyone waits on.

The principle: run each check close to the change that breaks it

The sooner a failure surfaces after the change that caused it, the cheaper it is to fix. A unit test that fails on your machine 30 seconds after you introduce the bug costs a minute. The same bug caught by a nightly end-to-end run costs an hour of bisecting. The same bug in production costs an incident.

That gives you a natural ordering — fast, cheap, precise checks early and often; slow, broad checks later and less often.

The per-commit stage is the one to protect most fiercely. If it creeps past ~10 minutes, developers start context-switching away while they wait, batching changes to amortise the wait, or pushing straight to a branch and hoping — all of which undermine the point.

Keeping the pipeline fast

• Parallelise. Split the test run across multiple machines or containers so wall-clock time stays flat even as the suite grows. Most CI systems and test runners support sharding out of the box.
• Fail fast, in the right order. Run the quick, high-signal checks first — lint and type checks in seconds, then unit tests. Don't make someone wait 15 minutes for the end-to-end suite only to be told a formatting rule failed. Some teams gate the expensive stages behind the cheap ones passing.
• Cache aggressively. Dependencies, build artifacts, compiled assets, browser binaries. A cold npm install on every run is pure waste.
• Only run what's affected, where you can. For large monorepos, running the tests touched by a change rather than the whole suite on every PR keeps the inner loop fast — with a full run on merge as the backstop.
• Quarantine flaky tests immediately. The day a test is identified as flaky, move it out of the blocking path. A flaky test that can fail a green build trains the whole team to rerun instead of investigate, and that habit is expensive to reverse. See false positives vs false negatives.

Where autonomous testing fits

Broad end-to-end coverage is slow, and when it's scripted it's high-maintenance — so it doesn't belong on every commit. It belongs on merge to main and on a schedule, where a 20–30 minute run is acceptable and its breadth pays off.

Autonomous runs slot into exactly that position without the maintenance overhead of a scripted E2E suite: after a deploy to staging, trigger a run, and review the findings alongside the release. There's no selector suite to keep in sync with the pipeline. Scheduled runs and API-triggered runs that fire directly from CI on a deploy event are on Manta's roadmap; until then, teams trigger them manually — after a deploy to staging and before releases — see continuous testing.

A note on required vs. advisory checks

Not every check should block a merge. Type checks, unit tests, and the core integration suite are usually required. A slow visual-regression job or a broad autonomous run might be advisory — it posts results a human reviews rather than hard-blocking — especially while you're calibrating its signal-to-noise. Move a check from advisory to required once you trust it.

The takeaway

Static and unit checks on every commit (keep this stage under ~10 minutes), the full integration suite and a thin scripted E2E set on merge, broad end-to-end and autonomous coverage on deploy or on a schedule, smoke checks and synthetic monitoring in production. Keep the pipeline fast with parallelism, caching, and fail-fast ordering, and quarantine flaky tests the day you spot them rather than tolerating them. Part 12 looks at the metric everyone quotes about all of this — test coverage — and what it actually measures.