How We Review Code When Agents Write Most of It

This week the Claude Code team at Anthropic published how they do code review. We read it twice. They are a large team shipping one of the busiest codebases in the world. We are a small consulting shop. But the problem they describe showed up on our desks too, just at a smaller scale.

Here is our version of the same problem. In June we merged 16 pull requests into the Periscoped website. In July, 5. In August, 31. In September, 45. In the first 18 days of October, 167.

Most of that code was written by AI agents that we direct. Nobody on our team reads 167 diffs line by line. So does anyone read the code? And if something breaks, who catches it?

This post is our answer. We did not remove review. We moved most of the reading to machines, saved people for the decisions only people can make, and made mistakes cheap to undo. Our main branch has no revert commits since April.

What code review is for

Code review has two jobs:

1. Find problems before a customer does.
2. Spread knowledge of the code across the team.

Most of this post is about the first job. We come back to the second near the end.

The question "did a person read every line?" is a weak stand-in for the question that matters: "what could go wrong with this change, and which check is the right one to catch it?" Some problems are best caught by a test. Some by a type checker. Some by an agent that reads the change with fresh eyes. And some only by a person who knows what the client actually asked for.

Asking it that way also changes when review happens. Review used to be a gate at the end: write the code, open a pull request, wait for a colleague. Now most of it happens while the work is being done, more like spellcheck than an exam.

The scarce thing is attention

For a consultancy, two kinds of time matter. Our team's time, and our client's time.

Review costs both in two ways. There is the time spent reviewing, and there is the time spent cleaning up what a bad change breaks. Cutting the first does nothing if the cost just moves to the second. We have learned that lesson the hard way: a rushed pipeline once cost a founder a full weekend.

So our goal is simple. Spend as little human attention per change as we can, without more reverts, more broken deploys, or more surprised clients. If those numbers go up, we did not save time. We only moved it.

The life of a change

A typical change on our website starts on a developer's machine, often as an agent session, and passes through several layers of review before it reaches the live site.

Figure A. Most of the path is automated. A person's attention goes where the risk is.

Layers, not one gate

No single check catches everything. Each layer catches some problems and misses others. Stack enough of them, with different blind spots, and very little gets all the way through. A fast rollback handles the rest.

Figure B. Each layer has holes. Few problems line up with a hole in every layer.

1. The work is checked before anyone else looks

Most review now happens before a pull request exists. We do not accept "it's done" from an agent. We ask it to show us: the tests it ran, the command and what it returned, a screenshot of the page.

A few habits make this work:

• Short project rules. Every repository has a CLAUDE.md, and an AGENTS.md for other AI tools, that says what the project is, how to run it, and the mistakes not to repeat. When an agent makes the same mistake twice, the fix goes into that file, not into another review comment.
• A fresh pair of eyes. We run review agents that read a change cold, with no memory of writing it and no power to edit it. One of them grades a new file as ship, fix, or bin. An author, human or agent, is the worst judge of their own work.
• Fast local checks. A commit lints the files you changed. A push runs a dependency audit and stops on anything critical.

2. Automated checks on every pull request

Every pull request into our dev or main branch runs the same set of checks:

• Unit tests that need no database.
• A type check of the whole project.
• Every database migration replayed from an empty database, then a check that the migrations and the schema still agree, then a second deploy to prove it does nothing.
• Integration tests against a real Postgres database and a stand-in for file storage.
• A check that our published API description is still valid.

Red blocks the merge.

3. Security review

A dependency audit runs on every pull request into main, on main itself, and once a week, because new problems are found in code that has not changed. We also keep review agents for the two areas where one missed bug is most expensive: sign-in and permissions. They read the whole implementation end to end, report every problem in order of severity, and change nothing.

We will be honest about one gap. We wrote a deeper code scan, and it is switched off until we buy the add-on it needs. We keep it off rather than failing, because a security check that goes red on every run teaches people to ignore red.

4. A person reviews, scaled by risk

A copy fix and a change to how people sign in do not carry the same risk, so they do not get the same review.

Risk	Examples	Human review
Low	Copy, docs, small dependency updates	None once every check is green. Dependency updates merge on their own; a failed one becomes a ticket
Medium	Ordinary features and page changes	One person, who also checks it on the dev site
High	Sign-in, permissions, payments, database migrations, production settings	A person reads the change and the evidence, plus a dedicated review agent

The change sets the tier, not the person who wrote it. And whoever directed the agent owns the change, exactly as if they had typed it themselves.

5. The dev site comes before the live site

Feature work merges into a dev branch first. That branch deploys on its own to a password-protected copy of the site, with its own database and test payments. We click through the change there. Only then does dev merge into main, and a merge into main is a release.

6. The release is gated on the whole suite

When dev merges into main, the full test suite runs against a fresh database: every migration, the integration tests, and three groups of browser tests. If it fails, there is no release, and our Slack channel says why. If it passes, the release notes write themselves from the commits, and every change carries one line, written for the person who did not do the work.

7. Fast detection, fast rollback

Some mistakes will still get through, so the last layer is about catching them fast. A build that fails never goes live; the last good version keeps serving. After every deploy, a smoke test visits the live site and checks that the key pages load. An uptime monitor keeps watching after that. If something is wrong, we roll back to the last good build in minutes.

Slowing down does lower the error rate, but mostly because you are doing less of everything. We would rather get faster at noticing and fixing problems.

How the reviewer gets better

Our review process improves in a loop. Every time something gets through, we ask one question: which layer should have caught this? Then we fix that layer, not just the bug.

Figure C. Every problem that gets through makes the layers stronger.

Some real examples from the last few months:

• Tests that could not fail. We found eleven tests whose whole body was the equivalent of "true is true." They were green, and they protected nothing. Now the rule is that an assertion must be able to fail. New route tests are checked by breaking the code on purpose and confirming the test goes red. In one batch we made 53 such breaks. The tests caught all 53.
• Skipped tests that looked like passes. When we cleaned up our browser tests, 65 were failing and 8 were quietly skipped. Every cluster was a real defect or real drift, not a flaky test. Now every release prints passed, failed, and skipped for each group of tests. A run where tests never ran must not look like a clean pass.
• A push that changed production. An old git hook copied whatever was on a developer's machine into production settings on every push. We removed it. A push now never changes settings or secrets.
• Docs that lied. Our own project notes once said the contact form sent an email. It did not. When we find a note like that, we correct it in place, with the date we checked, so the next reader and the next agent are not misled.

Noise matters too. A person can wave off a nitpick in two seconds. An agent will fix it, which means another commit and another round of review. A reviewer that flags everything is not thorough. It is expensive.

The numbers

Figure D. Merged pull requests per month. October is the first 18 days only. Revert commits on main over the same months: zero.

These are small numbers next to Anthropic's. But the shape is the same: the volume of change climbed by about ten times in a few months, and the error rate did not climb with it. That is only possible because a person is no longer the first and only line of defense.

Where people stay in the loop

Pulling someone out of deep work to catch a typo costs far more than an automated check. So our jobs have moved from reading every diff to the decisions machines are bad at.

Figure E. Six decisions stay with people. Automated checks run everywhere else.

• Should we build this at all? Direction, taste, and scope.
• What the client sees. Does this match what the client actually asked for, in their words? No test knows that.
• Risky code. Sign-in, permissions, payments, migrations, and production settings.
• Architecture. A pattern that other code will copy.
• Reading the evidence. The test output, the command that ran, the screenshot, the reviewer's findings. If you cannot say what the checks would miss, do a real review instead of a quick approval.
• Owning the outcome. A person is accountable for every change that ships, whoever typed it.

Sharing what the code does

Review's second job has always been to spread knowledge. Reading a teammate's diff is how you learn what changed. If machines do most of the reading, how does anyone learn the code?

Figure F. Ask for an explainer of exactly the part you are about to change.

Nobody keeps the whole codebase in their head, and we do not think anyone needs to. When someone needs to understand one part, they ask an agent to explain exactly that part, as the code stands today. It takes minutes. Sometimes we go further and have it draw a diagram or build a small interactive page.

A one-line copy change teaches you almost nothing. An explainer for the module you are about to change teaches you a lot.

We do the same for our clients. They should not have to read code to know what we built. So we write them a plain-language report of what changed, why, and how to try it, with links they can click.

You land it, you own it

If you land a change and something goes wrong, you usually fix it. A few things keep that from being harsh:

• Changes are small, so they are easy to review, debug, and revert.
• Everything is reversible. Rollbacks and a dev site undo most mistakes in minutes.
• Risk sets the bar. Expensive code gets more people and more checks.
• It is not blame. The person who landed the change has the most context. The question afterward is about the system: which layer should have caught it?

Because agents make code cheap to write, closing a pull request is not a loss. The trick is to raise a different direction early, in conversation, not as a blocking comment on finished work.

How you can do this with a small team

You do not need a big platform team. Most of this is a few files and a few settings.

Before the pull request

• Write a short CLAUDE.md with the commands an agent can run to check its own work.
• Ask your agent to show the output, not to say it is done.
• Have a second agent review the change cold, with no edit access.

On every pull request

• Require CI: tests, type check, and a migration check if you have a database.
• Run a dependency audit on pull requests and on a weekly schedule.
• Write down your risk tiers, and keep sign-in, payments, and production settings with a person.

Merging and after

• Put a password-protected dev site in front of production.
• Gate every release on the full test suite against a fresh database.
• Smoke test the live site after every deploy, and know how to roll back in one step.
• When something gets through, ask which layer should have caught it, and fix that layer.

Where this is going

We have more to do. Our browser tests are green, but CI still runs only part of them on each pull request, and widening that is next. The deeper code scan turns on when we buy the add-on. And we took one idea straight from Anthropic's post: cap the number of review rounds, so an agent author and an agent reviewer cannot keep each other busy all night.

The goal stays the same. Less human time per change. No more reverts than before. And more of our attention left for the part of the job that matters most: building the right thing for the people who hired us.

If your team lets agents open pull requests, we would like to hear what you have automated and what still needs a person.