Building with AI agents in the loop: spec, plan, adversarial review
A one-person team shipping a 19-tool SaaS by putting Claude agents on rails — and the defects the rails caught that I would have shipped.
- commits
- 394
- on main in about three months
- design specs
- 50
- committed before the code
- harness pages
- 54
- one HTML file per feature
- backend tests
- 332
- before one destructive release
Context
MemberHub is a multi-tenant SaaS with nineteen tools, built from nothing in about three months: first commit on 2026-04-09, 394 commits on main by mid-July. There is one engineer. Most of the code was written by AI agents; I decided, reviewed and operated.
The rails come from two facts about the environment rather than a theory about agents. The first is that the development box is production: the working tree serves live traffic against the shared production database, with no staging tier to absorb a mistake. The second is that the project does genuinely destructive things — one release deleted a 760-line rendering component and dropped around forty columns from a live table.
So the process is not about trusting the model. It is about what has to be written down for a capable worker with no memory of yesterday, who cannot ask a question halfway through and will do exactly what the instructions say.
What I built
The pipeline has a fixed shape: brainstorm, committed design spec, implementation plan, multi-agent build, adversarial review, verification in a real browser, deploy. Its artifacts live in the repository next to the code — 50 dated design specs, 25 implementation plans, 2 operational runbooks and 54 browser-harness pages. Seventy-four of those 394 commits are documentation commits, and most of them land before the code they describe.
- Brainstorm
- Design spec
- Implementation plan
- Multi-agent build
- Adversarial review
- Browser verification
- Deploy
Specs record decisions together with the alternatives that lost, and the checks made while designing — one justifies "no data migration" by noting production was queried first and had nothing to migrate.
Plans are written for agents rather than for humans skimming. They name exact files with line anchors, carry the code to insert, end each task with "commit only the files this task touched", and declare Produces/Consumes contracts between sequential tasks. Every plan opens with a Global Constraints block encoding the hazards: this box is production with a shared database; code tasks must not migrate, collect static files, restart anything or deploy; never stage everything at once, because my own unfinished work sits in the same tree.
Independent work splits into file-disjoint units — backend, components, locale files — that run in parallel, with the API contract frozen in the spec before fan-out: parallel agents are only safe when the interface between them is decided in advance, never discovered during the build.
There is no bundler and no JavaScript test framework, so verification is a set of harness pages: one HTML file each, importing the real modules and asserting into a results array a real browser hands back. The agent drives that browser itself, so the result is first-hand evidence rather than a claim: the results get pasted before anything may be called green.
Then a separate agent with fresh context gets the diff and the spec, and is told to find where the change is wrong.
Decisions
Cross-cutting steps get exactly one owner. With no bundler, a changed frontend module needs its cache-busting version bumped, and when five agents each change a component, every one of them assumes somebody else will do it. Nobody does, the tests pass, and returning visitors run stale code against a new API. The fix was an ownership rule rather than a reminder, written into the plans with visible irritation: the deploy task owns the cache busters, standing lesson, implementers forget them.
The orchestrator keeps the irreversible steps. Implementers may generate a migration file but never apply it; they never restart a service, deploy, or edit a template, since templates render from source and editing one is a production change in itself. Destructive work is staged so no running code meets a schema it does not expect, and the human gets a heads-up immediately before the irreversible command.
Full ceremony even for a one-line change. A change that turned out to be a single style passthrough still got a committed spec with its rejected alternatives. A lighter fast path was considered and refused: the cost model is asymmetric — the pipeline costs minutes of agent time, and an unreviewed small change on a production box has no ceiling on its downside.
What went wrong
The model retired and nothing said so. There is exactly one call site for a language model: a 54-line helper behind the "translate" button in four tools. It was pinned to a dated model snapshot; the snapshot was retired, and every AI translation began returning an upstream 404, with nothing deployed. Nine days passed unnoticed — translation is a human-triggered assist no health check exercises — and it surfaced only when the store's bilingual editor, the fourth caller, made the first translate call since the retirement. One call site meant a one-commit fix: a newer model, parsing that takes the first text block rather than assuming it, and an SDK pin corrected to what production had long run. A pin that lies is worse than no pin: it misinforms you exactly when you debug.
A test suite that verified code from days ago. Cache-busting query strings reach only the entry modules named in templates; everything imported bare is cached under its plain URL, and a parameter on the harness URL busts the page but not its imports. So a browser can happily confirm a copy of a module from days ago — stale red after a fix, or, worse, stale green before one. The first countermeasure written into the plans, a fresh port every run, works locally and does nothing against deployed files. The rule that holds is fussier: fetch each changed module with the cache bypassed, and only then re-navigate, because priming after the page has loaded does not fix that page's imports. It now goes verbatim into every agent prompt, because every agent not told rediscovered it the hard way. Plan text that is merely almost right produces confident green tests on old code.
The fix that was the next bug. One review round found two ways an authenticated user could get the backend to sign storage keys outside their own namespace. The remediation added walkers for shape data — which validated against the global key allowlist instead of the per-tool prefix, reintroducing the same leak on a public, unauthenticated path. The next round caught that. It is why fixes cite the finding they close: a later reviewer can check whether an earlier fix worked.
Outcome
The receipts are in the history. Twenty-one fix commits across four review rounds on one storage migration, each tagged with the finding it closes. Twelve tests added by the custom-domains review, taking the suite to 299. A destructive release verified by 332 backend tests and 255 browser assertions before it ran. Eleven commit subjects referencing a review outcome, including the ones recording a clean review.
More usefully, the reviews caught the classes of bug a second engineer is for: cross-tenant leaks, Host-header trust, a duplicate send when two scheduled ticks overlapped, an order number that collides under concurrency, and a migration that narrowed a column below the length of the data already in it — a failure that would only ever have appeared in production.
None of this is a guarantee, and the documents say so. It is discipline, written down where the next agent — and the next me — will read it.