The Factory Kitchen for AI Agents

TL;DR:

If you're putting AI agents to work, you need more than good prompts. You need an environment where agents can pick up context, work independently, learn from previous mistakes, and prove their results. We built one using team chat, isolated threads, shared Markdown files, and automated quality gates. It supported 2,389 agent sessions and more than 1,300 merged pull requests on our connectivity quoting platform. The most important lesson is that the agents don't need to remember everything. The system does.

Introduction

When I wrote about the dark software factory, I used manufacturing as the model: stations on a line, agents doing the work, humans in the control room.

Lauren Tan challenges that framing in her talk, “2,000 PRs in a Month.” She prefers a Michelin kitchen to a software factory. Agents may cook the individual components, but you still own the final plate and the kitchen setup: the line cooks, the sous chefs, their equipment, their training.

I think she's right. But there's another kitchen that helps explain what happens when you try to make this approach work across an engineering organization.

For the past six months, we've been building a connectivity quoting platform for telecom advisors, largely with AI agents. Since May, we've run 2,389 agent sessions and made about 175,000 tool calls. From the project's first commit in April through the end of September, we've merged 1,373 pull requests.

What I've learned is that getting agents to write software isn't particularly difficult anymore. The harder problem is building an environment where each new agent can pick up where the last one left off, follow the same standards, and produce work we can trust.

That's what brought me back to Atul Gawande's 2012 New Yorker essay, “Big Med.”

Gawande, a surgeon, ate at one of The Cheesecake Factory restaurants and wondered how a chain serving more than 80 million people a year, with over 300 dinner items on the menu, made food that tasted like it came from a good restaurant. A line cook told him nearly everything was made from scratch.

So he went into the kitchen.

Every restaurant had the same layout, organized almost like a manufacturing plant: prep in the back, two lines of cooks in the middle, finished plates out the front. A screen at each station showed the recipe, a photo of the finished dish, and a timer that changed from green to yellow to red. The recipes specified ingredients and outcomes but left the technique to the cook.

The broiler cook that Gawande watched never looked at the screen. He had the recipes in his head.

Then, before the plate left the kitchen, a manager at the pass checked the steak with a clean fork, declared it perfect, and told the cooks the mashed potato tower was too tight. Every plate received a rating from one to ten. Anything lower than a seven went back to be redone.

When Gawande asked the regional manager how he'd run a hospital, the answer was straightforward: study what the best people do, standardize it, and give everyone the means to execute.

That's the kitchen I think about when building systems for AI agents.

A Michelin kitchen works because a relatively stable brigade develops shared knowledge and carries the chef's standards in their heads. Agents don't work that way. Every new session starts without reliably inheriting the hard-won, project-specific knowledge of the previous one.

Whatever the best cook knows has to be captured in the kitchen itself: the layout, the ticket, the recipe, and the inspection at the pass.

And unlike a conventional kitchen, ours must get better every time a cook makes a mistake.

Building the kitchen

Here's what six months of working this way looks like, by the numbers I can measure:

• 2,389 agent sessions touched the quoting platform's code or notes between May and September. Of those, 617 were subagents spawned by other agents, and 263 were scheduled jobs running without anyone watching.

• Those sessions made approximately 175,000 tool calls.

• The repository went from its first commit in April to 1,373 merged pull requests and 1,672 test files.

• Our shared project notes contain 1,090 numbered work briefs and 41 runbooks.

For context, that's roughly 100 agent sessions a week, sustained for months. For comparison, one of our larger product engineering teams merges around 40 pull requests a week.

Of course, pull requests and tool calls aren't measures of product value. The actual work is a connectivity quoting platform that we're building to let telecom advisors quote, compare, and order services across carriers. The numbers tell you how much engineering activity this operating model has supported, not whether the resulting software is useful.

And there's still plenty of human work involved. We decide what to build, resolve ambiguous requirements, review plans, and make the final decisions about what gets merged. The agents do much of the work between those decisions.

So how do we manage that many sessions without constantly starting over?

We've ended up with five pieces: a connection to where the team communicates, channels that organize work by topic, threads that isolate individual tasks, shared files that preserve knowledge, and tests that enforce our standards.

We use Mattermost, Obsidian, and plain Markdown files, but those particular tools aren't the point. What's important is where agents get context, how work stays isolated, what knowledge survives a session, and how results get verified.

Each piece has an equivalent in Gawande's kitchen.

Connectivity: put the agent where the orders come in

We started with agents in a terminal, the text window developers use to interact with their machines.

That works for writing code, but the terminal is where the work gets done. Much of the important context lives where the work gets discussed.

A bug report arrives as an offhand message. A decision gets made between meetings. Somebody notices something odd while reviewing a demo.

If the agent only knows what someone remembers to paste into a prompt, it's working from a summary of a summary.

So we connected our agent runtime to team chat. First Telegram, then a self-hosted Mattermost server.

Now an agent can participate directly in the conversation. I can mention a broken page while walking between meetings, and the agent can find it in the code, investigate the root cause, open a branch, write a failing test, and come back with a pull request. All in the thread where the question started.

Self-hosting turned out to matter more than I expected. Our agent runtime executes on our hardware, reads our repositories, drives a real browser against staging, and posts its findings back to a server we control.

There's no manual handoff where somebody has to decide which parts of the conversation to copy into a terminal prompt.

The agent is connected to the place where the work originates.

Chat channels: the same layout, one station per topic

Every Cheesecake Factory kitchen has the same floor plan. A cook can walk into any location and know where the equipment is, where the ingredients are kept, and how work moves through the kitchen.

Each station has what it needs within reach.

We organized our chat channels the same way.

Several specific channels for our connectivity quoting platform. A separate channel for engineering metrics. Tokenomics for AI spending. Other channels for Software Quality, Organization, Talent, and Presentations.

When an agent picks up work in the Provider APIs channel, it starts at the right station. The channel establishes which data sources, tools, conventions, and instructions apply. Each request gets its own thread, preserving the investigation and the outcome. Sometimes, that ends in a pull request. Other times, the investigation identifies additional work that other agents pick up in other threads.

One platform engineering channel alone has accumulated 6,780 posts since we moved to Mattermost. In one undifferentiated conversation, that volume would be unmanageable. Organized by topic and thread, and fully searchable, the context stays relevant.

This is an important distinction from putting everything into a single giant prompt, a single conversation, or an ephemeral agent session history. We don't want every agent to know everything. We want it to have the right context for the work in front of it.

Threads: one ticket per plate

The screens in the Cheesecake Factory kitchen give each order its own ticket and its own timer.

Our chat threads serve much the same purpose.

Each thread represents a distinct piece of work. The pricing bug lives in one thread, the contract-PDF fix in another. The investigation, decisions, tests, and pull requests for each stay together.

The thread is a durable record of the work, not necessarily one uninterrupted agent execution.

That's important because long agent sessions degrade. Context windows fill with previous problems, the agent starts confusing details, and the runtime compresses older history into summaries that inevitably lose information.

By keeping work in separate threads, we give each task a clear boundary. When I return two days or two weeks later, a new agent execution can pick up the relevant conversation without inheriting unrelated problems from other threads.

Threads also make the work legible to people.

Anyone on the team can open a thread and follow the entire sequence: the original question, the investigation, the test that reproduced the problem, the proposed fix, and the merged pull request.

Our quoting platform has thousands of these threads. Each one reads like a ticket with the actual conversation attached.

This solves one problem: keeping individual pieces of work organized.

But it doesn't solve another, more important one.

How does something learned in one thread become useful to the next hundred agents?

Obsidian vaults: the recipe book and the test kitchen

Lauren argues in her talk that you don't need a sophisticated "company brain" because agents are already good at using tools.

I agree about the sophistication.

For this project, ours was a folder of plain Markdown files in an Obsidian vault. People and agents both read and write them.

Every substantial piece of work starts as a numbered brief: what's broken, the mechanism, the evidence, the decisions that need to be made, and the tests that should fail before a fix and pass afterward.

Decisions made in conversation get written back into the brief. Runbooks for major parts of the platform contain step-by-step acceptance scripts that agents can execute against staging, along with a record of what passed and what didn't.

The vault is our recipe book. It's also our test kitchen.

Gawande describes how the Cheesecake Factory developed new dishes in a dedicated test kitchen in Calabasas. Chefs worked out not just how to make something good, but how to make it reproducible across restaurants. We do something similar.

When an agent figures out how to do something difficult, such as pricing a multi-site connectivity quote or diagnosing why a vendor API rejected a request, the procedure gets written down as a skill or runbook.

The next agent starts with that knowledge instead of rediscovering it.

A scheduled job running overnight reads the same brief I read in the morning. An agent investigating a problem can follow the same acceptance procedure another agent used last week.

And when an agent makes a mistake, we don't want the correction to exist only in the conversation where it happened. We write it down.

That's where the model starts to get interesting. Individual agents don't reliably accumulate experience across sessions, but the engineering system can.

A difficult investigation becomes a repeatable procedure. A decision becomes a convention. A recurring mistake becomes a rule.

The knowledge survives the cook.

But documentation alone isn't enough. We also need to know whether the finished work is any good.

Testing the agents: the kitchen manager at the pass

The kitchen manager is the part of Gawande's story I keep returning to.

The cook was skilled. The recipe was on the screen. The steak was cooked.

And the plate still got inspected before it went out.

Our equivalent is a collection of rules that every agent has to follow. We've developed them through experience, and we tighten them each time an agent finds a way around one.

A test has to be able to fail.

Before a fix counts, the agent has to demonstrate that its new test catches the original problem. We have it revert the fix and confirm the test goes red, then restore the fix and confirm it passes.

Agents are very good at writing tests that pass. Unfortunately, they're also quite good at writing tests that would pass regardless of what the code did.

A green test doesn't mean much until you've proved it can turn red.

Green in one lane isn't green.

On one change, 12,731 unit tests passed while the endpoint returned a 500.

The problem was that the only test lane exercising that route was the integration suite running against a real database.

The agent had plenty of evidence that the unit tests passed. It didn't have evidence that the feature worked.

Now agents have to identify which test lanes actually cover the surface they've changed and run those lanes before pushing.

Prove it on the real system.

Agents drive a real browser against staging, execute the written acceptance scripts, and record what they observed.

A successful HTTP response isn't proof that something worked.

We've encountered vendor APIs that accepted requests and quietly ignored part of the submitted data. The response looked fine. The expected result wasn't there.

The agent has to verify the outcome, not just the request.

One observation is a hypothesis.

Root causes recorded in briefs are treated as hypotheses until a probe confirms them.

A surprising number of investigations that began with "the vendor's API is broken" turned out to be mistakes in how we constructed the request.

We've learned to require evidence before accepting an explanation, even when the explanation sounds entirely plausible.

Critics review the plan before anyone builds it.

For larger changes, a separate agent examines the specification and proposed implementation against the actual code before work begins.

Automated reviewers check every pull request. An agent addresses what they find, and a human makes the merge decision.

The idea isn't to eliminate human judgment. It's to make sure that judgment is exercised with enough evidence.

From corrections to guardrails

There's a progression here that I think matters more than any individual testing practice.

Lauren makes a related observation in her talk: if you repeatedly correct an agent for the same mistake, the strongest solution is to make that mistake impossible. A lint rule or an architectural constraint is better than a reminder in a prompt.

We've found the same thing.

Many of our rules began as corrections in a chat thread. The agent did something wrong, somebody noticed, and we documented what should have happened.

But documentation isn't the end of the process.

Where possible, the correction becomes a guard test in CI, an automated check, or a constraint that prevents the mistake from recurring.

Each one exists because an agent got past the previous version.

That's the difference between a collection of agents doing work and a system that improves through experience.

An agent makes a mistake. We capture the lesson. We turn it into something enforceable. The next agent benefits without ever knowing what happened.

The next time around, the kitchen is a little better.

What this adds up to

Six months into this, I don't think the factory and kitchen metaphors are really in conflict.

The factory describes how work can move through a system with increasing autonomy. The kitchen describes what it takes to make that work consistent, adaptable, and trustworthy.

The agents are capable of doing a remarkable amount of the actual engineering. But they're not carrying years of institutional knowledge in their heads, and we shouldn't expect them to.

That knowledge needs to live in the system around them.

Channels establish the context. Threads organize the work. Shared files preserve what we've learned. Tests and automated guardrails turn experience into standards that every subsequent agent has to meet.

The most valuable output of an agent session isn't always the pull request. Sometimes it's the test, the runbook, or the rule that prevents the next agent from making the same mistake.

That's what makes the kitchen work.

The cooks change constantly, but the recipes improve, the equipment gets better, and the standards get tighter.

We're not trying to build agents that remember everything. We're building an engineering system that doesn't have to.

Join the conversation on my original LinkedIn article. If you're setting up your own kitchen for agents, I’d like to compare floor plans.

Mathew Spolin leads global engineering at AppDirect, where managing organizational debt across multiple countries, acquisitions, and restructurings has been as critical as managing our technical debt. If this framework resonates, I'd love to hear your examples.