--- title: 'The narrow-loop playbook, step by step' excerpt: 'The narrow loop works when the steps run in order. Seven of them, each anchored to real dates from the doc-intel demo: 29 days, 74 commits, $20 a month.' date: '2026-07-04' lastModified: '2026-07-04' author: 'Teo Deleanu' authorAvatar: '/team/teo.jpg' tags: ['Production AI', 'AI Engineering', 'Agentic AI', 'Playbooks'] keywords: ['narrow loop AI playbook', 'production AI checklist', 'shipping AI to production', 'LLM evals in CI'] featured: false tldr: 'The narrow loop from the first article in this series, turned into an ordered runbook: seven steps, each anchored to a dated receipt from the doc-intel demo. Spec May 29, live on Fly.io June 27, 74 commits, about $20 a month. Quality checks run in CI against recorded cassettes, so they cost nothing per run and block merges without human attention. Week one produced five incidents and five playbook entries. Widening comes last, and only after the loop runs itself.' keyTakeaways: - 'Scope to one loop with one measurable output. The demo shipped in 29 days because its entire surface is document in, grounded answer out.' - 'Write the spec and the golden eval set before code, then make the eval a merge gate in CI: deterministic via recorded cassettes, zero API spend per run.' - 'Ship the thinnest complete pipeline to real infrastructure early. Every week-one failure appeared only because the system was live.' - 'Watch week one and convert each incident into a class-level fix. Widen to the next loop only after the current one runs itself.' --- The doc-intelligence demo on this site went from a written spec to production in 29 days: spec dated May 29, 2026, live on Fly.io June 27, 74 commits, about $20 a month to run. One engineer built it by running coding agents in parallel, and the system is one loop: a document goes in, a grounded answer comes out about 9 seconds after you ask. [The first article in this series](/blog/solve-almost-every-problem) argued that AI now solves almost every problem for one reason: the narrow, measurable loop got cheap to close. This post is the operational sequel. Seven steps, in the order we ran them, each with what we actually did on the demo, what it cost, and the specific way it fails when skipped. ![Seven steps of the narrow-loop playbook drawn as a loop: pick one loop, spec and evals first, ship thin, gates in CI, instrument, watch week one, widen, with an arrow from widen back to pick](/blog/narrow-loop-steps.svg) ## 1. Pick one loop with a measurable output Choose one process with a defined input, a defined output, and a number that says whether it worked. Then refuse everything outside it. The demo's entire surface is one loop: upload a document, the pipeline parses and indexes it, a live model answers questions about it. There is no second use case attached. The output is a grounded answer; the number behind it is retrieval recall, which step 4 turns into a merge gate. Cost: zero dollars. The price is discipline, because narrow scope feels like underambition while you hold it. Skipped: you build the general agent, and general agents are how you get silent failure at scale. Without the number, you cannot claim the system works: if you cannot measure it, you cannot claim it. ## 2. Write the spec and the eval set before code The spec states what the loop does and what number proves it. The eval set makes the quality bar concrete: twenty or so real questions paired with the chunks and answers a correct system should produce. Our spec is dated May 29, 2026, four weeks before the deploy. The demo's golden dataset lives in the repo, versioned with the code it judges. Writing the eval set this early is the general practice I hold to, because a quality bar written after the system exists bends toward whatever the system already does. Cost: days for the spec, a weekend for the evals: twenty golden questions, a scoring script, a CI job that posts the delta, and a threshold that blocks the merge. Skipped: your first regression has no baseline. Retrieval recall drops from 0.93 to 0.87 and nobody can prove it, because "good" was never written down. ## 3. Ship the thinnest complete pipeline to production early Complete means every stage exists end to end. Thin means each stage is the smallest version that closes the loop. Then put it on real infrastructure, in front of strangers, before it feels finished. The demo went live June 27 on three shared-CPU Fly machines: an API at 1 vCPU and 512 MB, a worker at 2 vCPU and 2 GB, and Postgres at 1 vCPU and 1 GB. Fixed infrastructure runs about $20 a month at published rates. The model meter is separate and per-use: $0.15 per million tokens embedded, answers at $0.30 in and $2.50 out per million. Cost: the price of a coffee subscription, plus the meter. Skipped: you learn nothing. Every failure that mattered in week one appeared only because the system was live. The demo that runs for two days in a screenshot tells you nothing about the system that has to run for two years under load. ## 4. Put the quality checks in CI, where they run without you A quality gate that depends on human attention is a gate that gets skipped. The demo's eval suite runs on every pull request: it scores the golden set against recorded judge cassettes, posts the score delta against main as a PR comment, and blocks the merge when the number drops. The cassettes keep it deterministic and free: the LLM judge is recorded once and replayed offline, zero API spend per run. The full machinery is in [evals as CI](/blog/evals-as-ci). The gate has paid off twice. The embeddings migration to gemini-embedding-001 at 768 dimensions shipped only after an eval spike showed retrieval held up. When the routing policy moved the demo to Gemini, the suite's assertions moved in the same commit. Cost: a weekend to stand up, nothing per run. Skipped: a manual quality check gets skipped exactly once, on the change where it mattered. ## 5. Instrument before launch Put a timeout on every model call and key rate limits to the actual client, before strangers arrive. Our last hardening pass, a day before launch, caught the rate limiter keyed on the connecting peer's address. Behind Fly's proxy, that address is the proxy, so every visitor shared one rate-limit bucket, and one curious person hammering the API would have throttled everyone. The fix reads the real client IP from the Fly-Client-IP header. Instrumentation has a known limit: the worst failure in week one kept every metric green, a chat answer rendered as white text on a white background while tokens were billed for output nobody could see. Instrument so the loud failures reach you fast. Step 6 exists for the quiet ones. Cost: hours in a hardening pass. Skipped: the infrastructure between you and your user lies about who the user is, and your system acts on the lie. ## 6. Watch week one and turn every incident into a playbook entry Week one of real traffic is where the system tells you what it actually is. Read the logs daily, and fix the class, never just the instance. The demo needed five human interventions in its first seven days, and [the full incident log is published](/blog/it-runs-itself-first-week). On June 30, three days after launch, Google retired the model snapshot the demo was pinned to; queries returned 404s with zero changes on our side. The instance fix was moving to the current model; the class fix was repointing to the gemini-flash-latest alias so the next retirement resolves instead of erroring. The same week produced the white-on-white render from step 5's caveat, plus cold starts, fixed by keeping one machine always warm. Another intervention raised the time-to-first-token budget to 20 seconds for a model that spends around 9 seconds thinking before its first token. The fifth was the rate limiter. Every fix took minutes to hours, because a human noticed quickly. Cost: a week of attention, plus the warm machine's meter running 24/7. Skipped: none of these failures repair themselves. Unattended, the 404s continue until every user has left. ## 7. Only then widen the loop Widening means adding the next loop, and the next loop gets the same seven steps, starting with its own spec and its own golden set. Making the existing agent more general is the move the first article warns against. I have no doc-intel receipt for this step, and that is the honest point: the demo is still one loop, on purpose. It has not yet earned a second. The general practice is the arithmetic in the first article: run this play ten times and you have ten systems, each with an eval in front and an accountable human behind. The unattended part never arrives. Cost: every widened loop is a new system someone watches and answers for. Rushed: widening before the first loop runs itself multiplies failure modes faster than it multiplies value, and week one showed how many failure modes one narrow loop generates on its own. ## Monday morning Pick the loop and write its number down. Write the spec and twenty golden questions before the first commit. Ship thin to real infrastructure inside a month, gate merges on the eval delta, instrument before strangers arrive, then spend week one reading logs and writing the playbook entries the incidents dictate. Widen only when the current loop runs without you. This sequence is also the exact shape of [the sprint we run for clients](/dfy): one loop, closed and instrumented, handed over with its playbook. The runbooks and eval reports behind that work are summarized in [the operator brief](/about), and the system behind every date in this article is still answering questions at the [live demo](/demo).