Tests as Contracts for AI Coding Agents in Production

A failed import must not move the saved cursor, which was the rule in a Rails system importing a UK property inventory agency's history from a third-party API.

The importer selects a page of root records, fetches dependent data, stores source payloads as snapshots, and projects local records, but it can stop halfway through, get a 429, or meet a bad attachment or unexpected response, so if it records progress after selecting a root but before importing it, a failure creates a hole: the job looks healthy and the cursor is newer, but one property and perhaps its reports, contacts, images, and invoices are absent, while a green job log will not find that gap.

The failed-root test therefore says that the persisted cursor does not advance, which is a better instruction for an AI coding agent than “make the import resumable” because it makes an observable proposition true, so when you give it a failing test the right part of the system is less open to interpretation.

That is how I use tests with AI agents: as the specification before generation rather than a ceremonial gate afterwards.

The agent only knows what the contract makes observable

A coding agent sees the repository, prompt, and available tools, so it can infer conventions and write code quickly, but it cannot know which reasonable behaviour the business relies on unless that behaviour is explicit, which is ordinary requirements work made easier to miss because generated code reads confidently.

A vague ticket has the same problem for a new engineer, but while a human may ask what “correct” means, an AI agent fills every blank in seconds.

“Resume imports after a failure” hides decisions about whether to resume at the next page or the same root, write the cursor when selecting a root, when the response arrives, or after projection commits, fail the root when an optional endpoint fails, skip and advance when a root exists locally, and respond when a webhook updates a record after a bulk importer has read an older version, since all of those choices can look tidy and pass a shallow happy-path test, but only one may preserve the data.

I put important constraints in a prompt, but prose gets interpreted, whereas a test is executable: it gives the agent a pass condition, makes policy reviewable, and remains for the next person or model, so I do not ask an agent to “make this safer” or “handle edge cases” - I ask it to make a named rule pass, then inspect the diff and adjacent tests, because the quality ceiling is the contract: a weak suite produces weak generated code, while a precise suite narrows wrong answers.

A persisted cursor is a small piece of business policy

The platform holds over 25,000 properties, more than 60,000 inspections, nearly 7 million observations, over 17 million attachments, and more than 4 TB of media, so at that scale “we can run it again” is not recovery; imports must be restartable, inspectable, and slow enough not to disrupt the operations team using the same remote API.

The root selector pages properties and inspections in batches of 50, while its cursor retains current page, offset, and exhaustion state: the record of what the system believes it has safely completed.

Its tests establish the following recovery rules:

• a persisted cursor resumes selection at the correct root after a process restart
• roots already synchronised locally are skipped without blocking later work
• a failed root leaves the persisted cursor where it was
• a paged import selects roots in the expected order rather than repeatedly taking page one

Those are recovery semantics rather than method tests, because when asked to “add concurrency”, an agent might persist the cursor in an ensure block to avoid repeated work and smooth progress; the common case may be faster, but the failed-root test makes it wrong, since persistence after selection is not persistence after a completed import.

The importer retries a 429 and transient 5xx responses up to four times, honours Retry-After when present, otherwise uses bounded exponential backoff, and records retry audit events, because “the request eventually succeeds” permits immediate retries, infinite retries, or a missing audit record; the contract also asserts that after a 429 the retry count changes and the retry event exists, so operations can distinguish a quiet import from one continually pushed back, while the shared production ceiling was about 60 requests a minute and the historical actions backfill was held to 40 so staff activity retained room, a deployment choice that rejects “retry aggressively” in favour of preserving headroom and still needs an engineer who understands the operating context.

Snapshot first, then make the data useful

The importer persists snapshots of fetched payloads, then projects them into local clients, properties, jobs, reports, invoices, and payments, rather than discarding a remote response after creating canonical rows, which makes an import debuggable after an API change or projection bug.

A root is not complete because a property row appears in PostgreSQL: the source payload must also exist locally so someone can inspect what the source said then, because a refactor that folds fetch, transform, and persistence can accidentally remove snapshots while the screen and import count still look right, which removes the evidence needed to investigate a bad record; the contract therefore asserts both that the local property has the correct external identifier and that the source snapshot remains, so the agent may choose a service boundary or reduce duplication, but it cannot redefine a successful import.

An inspection can have detail, report data, metadata, contacts, attachments, and a PDF generation path, but some endpoints are temporarily unavailable while others are absent by design, so treating every missing optional response as fatal stalls the backfill for harmless gaps, while treating every endpoint failure as optional lets the platform accept incomplete imports; tests exercise the cases where a root can still import because absence is tolerable, without weakening required-detail failures, so I would ask an agent to add one endpoint, preserve that behaviour, and make the named test pass without weakening the required-detail failure test, because separate rule tests make cursor advancement, snapshot retention, optionality, and retries legible when a model changes them together.

Freshness rules stop old imports overwriting live changes

The strongest rule is about time: the platform receives source-system webhooks while a bulk importer walks historical pages, so a webhook may write the newest version at 10:05 and the importer then processes a snapshot fetched at 09:58, while a blind upsert overwrites the webhook value with old data without throwing an exception or retry even as every job reports success; the tests assert that incoming timestamps do not overwrite newer webhook values with older import data, because “imports update records” describes an implementation, while “older import data must not replace newer webhook data” states policy and tells an agent which write is forbidden, and the test builds the race in miniature by creating a local record with the newer webhook timestamp and value, feeding the importer an older payload, and asserting the protected value remains, then providing a newer import payload and asserting the expected update, because without that second half a model can pass by refusing all imports.

Every negative rule needs its permitted neighbour: the failed-cursor rule sits beside the successful-root case that advances it, and the stale-data rule beside the newer-data case that updates it, because otherwise an agent can satisfy a contract by doing nothing.

For more on verifying a system rather than trusting a clean-looking repository, I have written about technical due diligence. What evidence proves a claim when the happy path is no longer representative?

Tests should tell the agent where its authority ends

Tests make delegation safer by making the task smaller rather than by permitting wider changes, so my loop proceeds as follows:

1. Find behaviour that can cause harm if guessed wrong.
2. Write or tighten a test that fails for it and passes for the intended neighbour.
3. Ask the agent for the smallest change that makes the specified tests pass.
4. Review the diff as if the tests did not exist.
5. Run the related suite and keep the test for future changes.

A passing test does not make generated code acceptable, because it may add an unnecessary abstraction, change an unrelated path, create a query per record, or make failures harder to diagnose; tests decide behaviour, while review decides whether the implementation is reasonable, so the prompt needs a boundary rather than a novel: “Modify the InventoryBase importer only. Preserve snapshot persistence. Do not change cursor representation. Add the smallest code needed for the failing cursor test.” A named test gives review a shared reference, because “this persists progress before the root is projected, violating the failed-root contract” is specific enough for the model and another engineer to act on.

I treat a test an agent writes as a proposal rather than evidence: it must fail before the code change, fail for a plausible broken implementation, and assert an outcome rather than a private method call, while the contract belongs where damage appears - persisted integration metadata after the root outcome for a cursor, the canonical upsert for stale data, or the request boundary before a signed webhook enters the queue - because a helper test three layers away can help, but not when policy lives at the system edge.

The small audio app shows the same rule at a different scale

A smaller iOS sound meter makes the same point: it has 34 unit-test methods across six test files and seven UI tests, reads microphone buffers, applies A-weighting, calculates a dBFS value, exposes a configurable estimated SPL value, and shows session state in SwiftUI, but does not record or transmit audio.

Its DSP tests fix facts an agent should not casually rewrite: invalid FFT sizing is rejected, a 1 kHz full-scale sine has the expected level, silence floors at -120 dBFS rather than negative infinity, 100 Hz is attenuated by A-weighting, and the curve is normalised at 1 kHz, because “make the meter more accurate” is meaningless without those anchors, while the silence floor remains a product decision because a mathematically pure implementation might return negative infinity for zero mean square, leaving the interface with a value it cannot sensibly display, so the test says silence is -120 dBFS.

The view-model suite clamps calibration to ±10 dB, resets session statistics when a session resets, and bounds the displayed range, while a fake NoiseMeter supplies readings, avoiding a physical microphone and timing-sensitive audio engine so feedback stays fast and repeatable; the seven UI tests launch with UI_TESTING, skipping live microphone capture and its permission prompt, and cover controls, settings and guide screens, the expired-trial paywall, onboarding disclosure, and dismissing an upsell, proving screen flow and accessibility identifiers rather than that AVAudioEngine works on a particular device or that physical acoustic accuracy is real.

Anything absent from the suite is still guesswork

Tests preserve the system knowledge we managed to state, but they do not replace knowing the system: the property platform's tests cannot choose 40 requests per minute rather than 35 next month, detect a source provider semantics change, or prove every historical image arrived, so the migration needed read-only production checks, a per-property manifest, live API comparisons, a durable missing-download worklist, and an explicit decision not to delete the source while the actions backfill was incomplete.

That is the difference between code contracts and operational evidence: tests describe represented cases, while integrity work checks whether the real estate, including old data shapes and missing UUIDs, made it across; the audio app can verify A-weighting at published points and view-model state transitions, but it cannot certify a phone microphone as a laboratory instrument, so it produces an estimate, applies a documented 120 dB reference offset, and lets the user add a persisted adjustment after comparison with a reference meter, since no generated test can manufacture device-by-device calibration evidence.

Green tests do not mean a repository describes the full problem, because unobserved assumptions can produce polished code correct only inside a small, accidental box, so before delegating I ask what behaviour must not change and what important behaviour is unobserved: the first becomes a test or existing contract, while the second becomes review, a manual check, a production measurement, or a request for a decision, because a passing suite is evidence, never a blank cheque.

If you are adding AI features rather than using an agent inside the engineering workflow, my OpenAI API checklist applies the same discipline to structured output, failure handling, and checking what a model actually returned.

I build Rails, iOS, and integration-heavy products with tests that make dangerous rules explicit. My CV and rates are here.