Test Environments and Test Data

A test result is only as trustworthy as the environment it ran in and the data it ran against. A green suite on a toy environment tells you the code works on a toy environment. This part is about getting both close enough to reality that the results transfer.

The environment ladder

Most teams have some version of these, in increasing order of "production-like" and cost:

Local — a developer's machine. Fastest feedback, least production-like. Databases are small, integrations are usually stubbed, and configuration drifts from person to person. Good for unit and integration tests while writing code; not a place to trust an end-to-end result.
CI — ephemeral, spun up fresh per pipeline run and torn down after. Deterministic and clean, which is exactly what unit and integration tests need. Not suitable for anything that needs persistent state or real third-party services.
Staging / pre-production — a shared, persistent environment built to mirror production: same infrastructure shape, same deploy path, same integrations in sandbox mode, realistic data volumes. This is where end-to-end and autonomous testing belong.
Production — the real thing. A few checks run here deliberately: synthetic monitoring hitting the critical journeys on a schedule, and smoke checks right after a deploy. These catch problems that only appear with real infrastructure, real traffic, and real data.

The higher you go, the more a passing result means — and the more expensive and slower it is to produce. Match the layer to the question: "is this function correct" is a CI question; "does the whole checkout work against the real payment sandbox" is a staging question.

What "production-like" actually requires

Staging is only worth having if it's genuinely similar to production. The ways it usually isn't:

A different build or config path. If staging is built differently from production — different bundler flags, different environment variables, a manual deploy instead of the pipeline — you're testing an artifact you won't ship.
Stubbed integrations where production uses real ones. Payment, email, auth providers, and search should run in the vendor's sandbox/test mode, not be replaced with hand-rolled fakes — unless you're deliberately isolating a test from a flaky dependency.
Unrealistic data. Production has long names, right-to-left text, emoji, users with 10,000 records and users with zero, accounts in every subscription state. Staging seeded with three tidy example rows won't surface the bugs that live in the messy cases. The data doesn't have to be real, but it has to have the same shape.
The wrong scale. Anything performance-sensitive — a report query, a list view, a background job — behaves differently against 100 rows than against 10 million. If you'll test performance on staging, staging needs production-comparable volume.

A staging environment that's a scale model of production produces results that look reassuring and don't transfer.

Test data strategy

Known starting state. Tests need fixtures — seeded data they can rely on being there. Either seed it fresh before each run, or reset it on a schedule (nightly is common). A test that depends on data left over from a previous run will pass until it doesn't.
Isolation. One test's actions shouldn't change another test's outcome. Give each test (or each run) its own account, tenant, or namespaced data. Shared mutable state is one of the top causes of order-dependent test flakiness.
Dedicated accounts, one per role. A standard user, an admin, an unverified user, a user on each plan. Never a real customer account — tests create, edit, and delete data, and they'll do it to whatever account you point them at.
Privacy. If you copy production data down to staging to get realistic volume, it must be anonymised — names, emails, payment details, anything personal scrubbed or synthesised. Real user data sitting in a lower environment with looser access controls is a breach waiting to be reported.

In practice, for an autonomous run

The practical setup for running an autonomous test:

A staging environment built the same way as production, with real integrations in sandbox mode.
A dedicated test account per role, so the agent can cover admin-only areas as well as the regular user experience.
Resettable data, so successive runs and comparisons start from a comparable state.

If staging isn't reachable from the public internet, the Manta Runner executes from inside your network so the environment never needs an inbound hole.

The takeaway

Test results transfer to production only if the environment is genuinely production-like — same build, real integrations, realistic data shape, comparable scale — and the data is known, isolated, and safe to destroy. Anonymise anything copied down from production. Match each kind of test to the right rung of the ladder. Part 11 covers wiring all of this into CI so it runs without anyone asking.