Inside OpenAI's Agentic Software Factory
Owain Lewis walks through a Pragmatic Engineer article describing how OpenAI runs an agent-driven "software factory" — Codex writing code, CI looping until it passes, multiple specialist review agents, a risk classifier that decides what can auto-merge and auto-deploy, and a severity bot that triages production incidents. He then builds two live demos of his own: a Codex automation that labels open pull requests by risk and auto-merges the safe ones, and a Google ADK agent that consumes production alerts via pub/sub, investigates logs and metrics, and recommends a fix. His conclusion is that risk classification and automated on-call triage are the two ideas most worth copying today.
- OpenAI's pipeline starts with a "software builder" (engineer or PM) describing a task to Codex, which loops with CI to build and test until the change is ready.
- Code review is split across multiple specialist agents — data, infrastructure, cloud, and security — rather than a single reviewer agent.
- A classification step labels each change low or high risk, and low-risk changes can be auto-merged and auto-deployed without human review.
- Risk is relative to context: a solo developer's bad merge is cheap, while an infrastructure or database change at a company like Amazon can take down a region, so those always need a human.
- A "severity bot" monitors production, investigates errors automatically, and assists on-call engineers — a strong fit for agents because they can scan logs and correlate metrics far faster than a tired human at 2am.
- A performance factory runs tests in production and feeds regressions back into Codex, closing the loop so bugs and incidents become new tickets in the pipeline.
- In the PR demo, a low-risk change was still not merged because CI was failing, and the speaker warns against auto-merging anything that needs a rebase since the code can change.
- The incident agent correctly diagnosed 500 errors and recommended rolling traffic back to the previously healthy revision, and the speaker says it didn't take long to build.
This is the core idea of the whole system and the one the speaker says he'll adopt himself.
What's really clever or interesting about the approach from OpenAI is they do a classification on the change. They will look at whether a change is low risk or high risk and then decide whether an agent should auto merge and auto deploy that change.
A concrete demo showing the on-call agent correctly diagnosing an outage and proposing the right fix.
The system is returning 500 errors... the recommended action is to restore traffic to the previously healthy revision. So essentially a rollback is the suggestion from the agent. And this is exactly the right thing to do.
Risk is relative. The level of risk or what you consider to be a low risk or a high risk change is going to look very different depending on what you're doing or which company you're working in.
It's very difficult for humans when they're paged at 2:00 a.m. in the morning, they're tired, to remember to find all of the information... but machines are very, very good at doing this.
12 months ago, people would have said that that was crazy or reckless to use AI agents for the deployment process, but I think it makes a lot of sense right now.