How to build a software factory from zero

What the people building them are saying, and what we learned running ours.

Eleven days after we switched on our software factory, it had opened 332 of the 675 PRs our team merged.

That's about half of everything our four-person engineering team shipped in that time.

This guide is everything we've learned building it, combined with the best thinking from the people doing this in public: Matt Pocock (external site), Lauren Tan (external site), Dex Horthy (external site), Zach Lloyd (external site), Ben Holmes (external site), and the OpenAI engineers who described their factory to Gergely Orosz (external site).

It goes in the order we'd do it again.

TL;DR#

• Step 0: Make your codebase hard for agents to get wrong.
• Step 1: Give an agent a way to prove its own work.
• Step 2: Agree on the plan before the agent writes code.
• Step 3: Automate one trigger and one job, then widen.
• Step 4: Sandbox agents and keep the posting keys away from them.
• Step 5: Let agents review first, then read the evidence and the code that matters.
• Step 6: Make the agent that opens a PR own it to merge.
• Step 7: Route production back into the factory, and close the loop with the user.
• Step 8: Have the factory improve its own jobs.
• Step 9: Measure what shipped, not tokens spent.
• Step 10: Roll it out so the whole team can use it and improve it.

So, what's a software factory?#

The term is everywhere, and people are (rightfully) making fun of it.

Here's how we think about it:

Matt Pocock's AI Coding Dictionary (external site) separates three things people mix up:

• an AFK agent: one agent you leave running while you're away,
• a software factory: a system of agents with triggers, gates and feedback loops,
• a dark factory: a factory where no human reads or writes the code.

Ben Holmes (Warp) adds the part most people miss (external site): a factory is one shared system for the whole team, not custom setups on everyone's laptop.

One source of truth for automations, repo access, secrets, sandboxes and model choices, agreed the way you agree on CI and lint rules.

Our working definition:

work enters on its own triggers, agents carry it all the way to proof, people make the decisions, and the system gets better every time it runs.

And one warning up front, from Dex Horthy's Why Software Factories Fail (external site): models are great at passing tests and bad at keeping a codebase easy to change.

So this guide builds a factory with the lights on.

The real risk isn't that agents write bad code. It's that they write more code than anyone can understand. Every change you merge without understanding it is comprehension debt, and at agent speed it compounds fast.

Every step below is about paying that debt down: agents carry the work all the way to proof, and people review the proof instead of the diff.

Step 0: Make your codebase easy for agents to get right#

Before automating anything, make the easy thing the right thing.

Lauren Tan (@poteto) has this idea we love, a trust pyramid (external site).

You put most of your effort at the top:

1. Codebase improvements: better architecture, stronger types, clear boundaries. Agents copy what exists and take the shortest path, so remove whole classes of mistakes in code.
2. Static analysis: lint rules, types, tests, CI. Cheap, deterministic, token-efficient.
3. Prose: AGENTS.md, rules, skills. Only for what can't be enforced.
4. Human review: the hardest to scale, so spend it designing the environment, not reading every line.

Two ideas from her talk, summarized by Matt Pocock (external site) are worth stealing early:

• Lock agents down. Agents do better with abstractions they can't misuse, enforced by lint, than with sharp tools.
• Feature maps. A maintained map of every feature and how it's supposed to behave, kept in sync by the factory's own automation, so an agent can turn a vague bug report into a precise fix.

OpenAI went further and moved its documentation into the source tree (external site), so agents read it as code.

What we did: we kept every job's instructions as skills in the repo, so a change to the factory is a reviewed PR like any other code.

Step 1: Give one agent a way to prove its own work#

This is the entry point, and most people skip it.

If you can't trust one agent while you're watching it, you definitely can't trust tens or hundreds of them while you sleep.

Lauren calls verification "the entrypoint into your most basic agentic loop." (external site)

The agent should run and use what it built the way a user would, and come back with an artifact that proves:

1. the work is correct,
2. it matches what you intended,
3. it's faster to review than the diff.

For UI, that's a recording.

For a CLI, the real command + its output.

For gnarly state machines... formal methods like Lauren's /create-formal-verification-skill in pstack (external site), and GitHub Next's PDD Kit (external site), which proves agent code against a spec.

A quick story:

We use TLA+ (external site) on Traycer's sync code. When an agent wrote the collaborative spec document, a model checker explored the ways edits, deletes, and crashes would interleave in a small two-machine model. Then we replayed each new failure it found against the real code. It found ways of forcing it: it could silently vanish, or a deleted file could come back in the code that had already been through six rounds of review.

Six!

Our rule: tests prove the test; recordings prove the product.

No fix lands without a test that fails on the old code, and the agent reproduces the bug in the real build before it touches anything.

Step 2: Align before the agent writes a single line#

Faster coding moved the bottleneck to review, and the cheapest review is of a plan you already agreed on.

Matt Pocock's day shift looks like this (the skills are in mattpocock/skills (external site)):

1. /grill-me (or /grill-with-docs) to get the agent to interview you until you're aligned,
2. /to-prd for the destination,
3. /to-tickets for the journey, as small tickets agents can take in parallel.

The specs are meant to be thrown away after, not kept as documentation.

Dex Horthy's version, from part 2 (external site), goes one level deeper for anything risky:

1. Product review: the problem, what success looks like, and a rough HTML mockup instead of paragraphs.
2. System architecture: services, contracts, data models; sequence diagrams.
3. Program design: call-stack trees, file-tree diffs, types and signatures.
4. Vertical slices: build one thin path end to end you can touch, not database first and UI last.

30 minutes of planning saves hours of review.

Step 3: Start tiny: one trigger, one job#

Now automate.

But small, don't point your factory at your whole backlog.

Have it triage one issue a day.

Check its work.

Adjust.

Then two, then five, then ten.

Zach Lloyd's crawl, walk, run (external site) lists good first jobs, each a simple trigger → agent:

• reproduce and label new issues,
• review PRs as they open,
• respond to a Sentry alert with a fix,
• fix red CI,
• update docs and changelogs,
• verify changes with browser or computer use,
• fix simple user-reported bugs.

Pick chores.

Zach's rule for picking: if the agent gets it right, great; if not, ignoring the output costs nothing.

Our first jobs were exactly these: Sentry errors and flaky tests on day one, then user reports, a red trunk and the factory's own upkeep on day two, and provider updates, releases and infra alerts by day four.

Step 4: Sandbox it, centralize it, and lock the keys#

A factory runs agents nobody's watching.

So assume one of them will mess up.

Sandbox every agent. Here's how to:

• Sandboxes. Matt Pocock's open-source Sandcastle (external site) is the simplest place to start: one run() call starts an agent in Docker, Podman or a cloud microVM on its own branch, and merges the commits back.
• One shared setup. Run the factory as team infrastructure, not on someone's laptop.
• Scoped access. Give agents only what the job needs. As one reply to Zach (external site) put it: scoped permissions and rollback before autonomy, not after the first incident.

What we did: agents only hold read tokens.

Messages to people in Slack and Discord, and approved replies to user reports, go through a separate gateway.

The gateway holds the posting keys, checks every message against rules + daily budgets, and keeps quiet hours.

Anything from users, Slack or Sentry is untrusted data.

Never instructions.

A mistake we made:

We tried using an LLM to filter noisy messages before they reached us. It silently held 40 messages the jobs had been told to send. Including real approval requests and release updates.

Oops.

Plain rules + budgets work way better.

Step 5: Agents review first, people review proof#

You'll have more PRs than anyone can read.

As Dex puts it:

You don't have too many PRs. You have too many bad PRs.

What works:

• Several reviewer agents with different jobs. OpenAI (external site) runs domain-specialist reviewers, the equivalent of an infra expert and a security expert on every change.
• A blind reviewer. Ours reads the diff and the original problem, never the author agent's reasoning, so it can't be talked into agreeing.
• Risk tiers. OpenAI routes high-risk changes through stricter review and lets some areas auto-approve low-risk ones.
• Human attention on proof. Not on every line.

The clearest example we have is releases.

Before every release our release agent posts one proposal in Slack: 179 PRs grouped into 19 areas, the 5 high-risk areas and why, what QA ran on the staging build, what it couldn't check ("not covered: live desktop repros, mobile builds, the signer"), and who needs to approve what.

It attaches a two-minute narrated walkthrough.

We read for five minutes and decide.

Step 6: The agent isn't done when the PR opens#

Opening a PR is the easy part.

The work is getting it merged: answering reviews, fixing failing checks, rebasing.

At OpenAI the agent "babysits the PR until it's green." (external site)

Our version: the agent that opens a PR owns it all the way to merge.

Every 15 minutes, a check looks at the factory's open PRs and nudges anything stuck. Every time a person has to relay a message between two agents, we treat it as a bug.

Another mistake we made:

One job drafted 45 messages to us in three days. 36 of them said "no action needed."

Thankfully, none got sent. We had to teach it when to stay quiet.

Now an agent either lands its work or asks one clear question.

Step 7: Close the loop from production back to the factory#

The whole point of a factory is that work finds the agent.

• OpenAI's Perf Factory (via Gergely (external site)) watches production, removes duplicate alerts, finds real latency regressions and proposes fixes.
• Their incident agent gathers context and suggests mitigations, but a person decides.
• Their deploy agent "handholds" each change to production and builds its own monitoring dashboard for it.
• Ben Holmes routes Sentry alerts to an agent through a webhook (external site), with the fix PR as the output.

Here's one of ours, end to end:

A user couldn't install Traycer because their Windows was set to Spanish.

They reported it at 4:24am our time.

The agent matched it to 200+ Sentry errors from non-English Windows, traced it to one check that only understood English error messages, wrote a failing test and a small fix.

PR open at 8 minutes.

Edge-case tests at 22.

Passed automated review and merged at 23.

We didn't have a Windows machine for a full end-to-end install run, and the PR said so.

But, the loop doesn't end at the merge.

It ends when the person who reported it hears back.

Someone told us on Discord that long answer options in an agent's question were getting cut off.

Overnight, our triage agent found the cause, wrote a failing test, and fixed it (with one decision from us along the way).

Then it went back to the same Discord thread and told them the fix ships in the next release.

It was 2:47am for us.

Step 8: Make it improve itself#

This is the step most guides skip, and it's the one that compounds.

Ben Holmes' self-improvement loop (external site):

1. collect every agent session,
2. score each one (efficiency, code quality) with rubrics,
3. have an agent find the common causes in the low scores,
4. have it propose changes to skills, AGENTS.md or the codebase.

Matt Pocock runs /retro (external site) on old sessions for the same reason, and on a weekly schedule asks an agent to cull his global steering files.

Lauren calls this work "gardening."

Zach and Ben call it "factory engineering" (external site): product engineering improves the product, factory engineering improves the agents that improve the product.

Our numbers: 43 of the factory's first 332 merged PRs were it improving its own jobs.

Every job audits itself weekly, and any improvement it names becomes a PR the same day.

A question we answer the same way three times becomes a standing rule.

Step 9: Measure it like a system#

Not tokens spent.

Zach Lloyd's advice (external site) is to track adoption first, then throughput, cycle time and cost per PR.

Ben Holmes says (external site) most of their 67% drop in cost per PR came from replaying past tasks against different model choices.

Our scoreboard after 11 days:

• 49% of merged PRs opened by the factory,
• 92% of factory PRs merged, 28 closed unmerged,
• a median of 1.6 hours from PR opened to merged,
• 23 minutes for our fastest user report to merged fix.

Whatever you measure, measure what shipped, not tokens spent.

Step 10: Roll it out to the team#

Zach's org playbook (external site), condensed:

• start with chores so value shows immediately,
• make it as easy as tagging an agent in Slack, Linear or GitHub; don't build a new portal,
• treat the factory as shared infrastructure everyone can improve, not a product only admins configure,
• encourage it, don't mandate it, and celebrate shipped work, not token spend.

The trap to avoid, in his words: a full backlog-fixing loop on day one creates PRs nobody reviews, and the team decides "this thing is just a slop cannon."

What stays human#

Agents are great at tactical stuff.

They have no appetite for strategic change.

Matt says it best (external site):

You can have a software factory, but you have to keep the lights on.

In our factory, people own product calls, copy, auth, billing, releases, public replies, and anything irreversible.

Each one shows up as a decision with a recommendation.

Routine fixes move through the factory's gates.

People decide the exceptions, and read the code that matters.

FAQ#

Resources#

Start here#

• Dex Horthy, Why Software Factories Fail (external site), part 2 (external site), part 3 (external site)
• Zach Lloyd, Adopting the software factory model: crawl, walk, run (external site)
• Lauren Tan, how to trust your agents' work more (external site)

Skills and tools#

• Matt Pocock, mattpocock/skills (external site): /grill-me, /to-spec, /to-tickets, /implement, /retro, /codebase-design, /writing-for-agents
• Matt Pocock, Sandcastle (external site): sandboxed AFK agents with one run()
• Lauren Tan, pstack (external site): /poteto-mode, /create-verification-skill, /create-formal-verification-skill, /recall
• GitHub Next, PDD Kit (external site): proof-driven development for agent code

Real factories, described#

• Gergely Orosz, Inside OpenAI's agentic software factory (external site)
• Ben Holmes, the multi-agent system driving our factory (external site) and The road to self-improving software (external site)
• Matt Pocock, day shift and night shift (external site)

More voices worth following#

• Matt Pocock, AI Coding Dictionary: software factory vs dark factory (external site)
• Geoffrey Huntley, "your product is the factory" (external site)
• Ben Holmes, product engineering vs factory engineering (external site) and scoring and benchmarking agents (external site)
• Lauren Tan, verification is the entrypoint (external site) and volume does matter (external site)
• Theo, the audit prompt for your "babysit PR" scripts (external site)

Rolling it out#

• Zach Lloyd, Tactics for rolling out software factories (external site)
• Matt Pocock, mistakes you might be making (external site)

We're turning our factory into a product, so any team can run one. Join the waitlist.

And if you want to try the best agent workspace out there, check out traycer.ai.