The doc-intelligence demo on this site went from a written spec to production in 29 days: spec dated May 29, 2026, live on Fly.io June 27, 74 commits, about $20 a month to run. One engineer built it by running coding agents in parallel, and the system is one loop: a document goes in, a grounded answer comes out about 9 seconds after you ask.
The first article in this series argued that AI now solves almost every problem for one reason: the narrow, measurable loop got cheap to close. This post is the operational sequel. Seven steps, in the order we ran them, each with what we actually did on the demo, what it cost, and the specific way it fails when skipped.
1. Pick one loop with a measurable output
Choose one process with a defined input, a defined output, and a number that says whether it worked. Then refuse everything outside it.
The demo's entire surface is one loop: upload a document, the pipeline parses and indexes it, a live model answers questions about it. There is no second use case attached. The output is a grounded answer; the number behind it is retrieval recall, which step 4 turns into a merge gate.
Cost: zero dollars. The price is discipline, because narrow scope feels like underambition while you hold it.
Skipped: you build the general agent, and general agents are how you get silent failure at scale. Without the number, you cannot claim the system works: if you cannot measure it, you cannot claim it.
2. Write the spec and the eval set before code
The spec states what the loop does and what number proves it. The eval set makes the quality bar concrete: twenty or so real questions paired with the chunks and answers a correct system should produce.
Our spec is dated May 29, 2026, four weeks before the deploy. The demo's golden dataset lives in the repo, versioned with the code it judges. Writing the eval set this early is the general practice I hold to, because a quality bar written after the system exists bends toward whatever the system already does.
Cost: days for the spec, a weekend for the evals: twenty golden questions, a scoring script, a CI job that posts the delta, and a threshold that blocks the merge.
Skipped: your first regression has no baseline. Retrieval recall drops from 0.93 to 0.87 and nobody can prove it, because "good" was never written down.
3. Ship the thinnest complete pipeline to production early
Complete means every stage exists end to end. Thin means each stage is the smallest version that closes the loop. Then put it on real infrastructure, in front of strangers, before it feels finished.
The demo went live June 27 on three shared-CPU Fly machines: an API at 1 vCPU and 512 MB, a worker at 2 vCPU and 2 GB, and Postgres at 1 vCPU and 1 GB. Fixed infrastructure runs about $20 a month at published rates. The model meter is separate and per-use: $0.15 per million tokens embedded, answers at $0.30 in and $2.50 out per million.
Cost: the price of a coffee subscription, plus the meter.
Skipped: you learn nothing. Every failure that mattered in week one appeared only because the system was live. The demo that runs for two days in a screenshot tells you nothing about the system that has to run for two years under load.
4. Put the quality checks in CI, where they run without you
A quality gate that depends on human attention is a gate that gets skipped. The demo's eval suite runs on every pull request: it scores the golden set against recorded judge cassettes, posts the score delta against main as a PR comment, and blocks the merge when the number drops. The cassettes keep it deterministic and free: the LLM judge is recorded once and replayed offline, zero API spend per run. The full machinery is in evals as CI.
The gate has paid off twice. The embeddings migration to gemini-embedding-001 at 768 dimensions shipped only after an eval spike showed retrieval held up. When the routing policy moved the demo to Gemini, the suite's assertions moved in the same commit.
Cost: a weekend to stand up, nothing per run.
Skipped: a manual quality check gets skipped exactly once, on the change where it mattered.
5. Instrument before launch
Put a timeout on every model call and key rate limits to the actual client, before strangers arrive.
Our last hardening pass, a day before launch, caught the rate limiter keyed on the connecting peer's address. Behind Fly's proxy, that address is the proxy, so every visitor shared one rate-limit bucket, and one curious person hammering the API would have throttled everyone. The fix reads the real client IP from the Fly-Client-IP header.
Instrumentation has a known limit: the worst failure in week one kept every metric green, a chat answer rendered as white text on a white background while tokens were billed for output nobody could see. Instrument so the loud failures reach you fast. Step 6 exists for the quiet ones.
Cost: hours in a hardening pass.
Skipped: the infrastructure between you and your user lies about who the user is, and your system acts on the lie.
6. Watch week one and turn every incident into a playbook entry
Week one of real traffic is where the system tells you what it actually is. Read the logs daily, and fix the class, never just the instance.
The demo needed five human interventions in its first seven days, and the full incident log is published. On June 30, three days after launch, Google retired the model snapshot the demo was pinned to; queries returned 404s with zero changes on our side. The instance fix was moving to the current model; the class fix was repointing to the gemini-flash-latest alias so the next retirement resolves instead of erroring. The same week produced the white-on-white render from step 5's caveat, plus cold starts, fixed by keeping one machine always warm. Another intervention raised the time-to-first-token budget to 20 seconds for a model that spends around 9 seconds thinking before its first token. The fifth was the rate limiter. Every fix took minutes to hours, because a human noticed quickly.
Cost: a week of attention, plus the warm machine's meter running 24/7.
Skipped: none of these failures repair themselves. Unattended, the 404s continue until every user has left.
7. Only then widen the loop
Widening means adding the next loop, and the next loop gets the same seven steps, starting with its own spec and its own golden set. Making the existing agent more general is the move the first article warns against.
I have no doc-intel receipt for this step, and that is the honest point: the demo is still one loop, on purpose. It has not yet earned a second. The general practice is the arithmetic in the first article: run this play ten times and you have ten systems, each with an eval in front and an accountable human behind. The unattended part never arrives.
Cost: every widened loop is a new system someone watches and answers for.
Rushed: widening before the first loop runs itself multiplies failure modes faster than it multiplies value, and week one showed how many failure modes one narrow loop generates on its own.
Monday morning
Pick the loop and write its number down. Write the spec and twenty golden questions before the first commit. Ship thin to real infrastructure inside a month, gate merges on the eval delta, instrument before strangers arrive, then spend week one reading logs and writing the playbook entries the incidents dictate. Widen only when the current loop runs without you.
This sequence is also the exact shape of the sprint we run for clients: one loop, closed and instrumented, handed over with its playbook. The runbooks and eval reports behind that work are summarized in the operator brief, and the system behind every date in this article is still answering questions at the live demo.