Ask the old code to review the new one

An AI writes more code in an hour than you can read in a day. Theo, who runs a programming YouTube channel, said, "If you're still reading all of the code, you're not generating enough of it." I think he is right. But when I stop reading the code, something else has to check it. On a server, the best checker I have found is the old code, the code the rewrite replaces.

The old code keeps answering as before, the new code runs beside it on the same real requests, and the two answers get compared. To see what that catches, take a small app.

The old function

The app has projects, and some of them are private. All day long, the server answers whether one person can see one project. One function holds that answer, and it has grown for years.

function canSee(user: User, project: Project): boolean {
  if (project.isPublic) return true;
  if (user.bannedFrom.includes(project.workspace)) return false;
  if (project.owner === user.id) return true;
  if (teamIncludes(project.team, user)) return true;
  const link = project.guestLinks.find((l) => l.user === user.id);
  if (link && link.expires >= startOfToday()) return true;
  return false;
}

If you look at the function, there are six rules, and each one answers yes or no. Anyone gets a yes on a public project. A banned person gets a no, whatever else is true. The owner gets a yes. A member of the project's team gets a yes, and teams can sit inside teams, so teamIncludes walks down into them. A guest with a link gets a yes until the link expires, and everyone else gets a no.

Five people use the app, Ana, Bo, Cy, Di, and Eve. It holds a private project called Kite that belongs to the Product team, a public project called Moss, and a private project called Fern with no team at all.

Two of the six rules are easy to miss in a review. Bo works in Design, Design sits inside Product, and Product owns Kite, so Bo gets a yes through a team inside a team. Eve has a guest link to Kite that runs out today, so Eve gets a yes until it does. The function handles both cases, and the people who wrote those two rules left long ago.

The rewrite

Now an AI rewrites canSee. The new version is half the length, it passes the five tests the function has had for years, and I did not read it.

Those five tests check the cases their authors thought of when they wrote them. There is a test for a public project, a banned person, an owner, a direct team member, and a stranger. No test covers a team inside a team, and none covers a guest link on the day it runs out. So a green test run only means the old code and the new code agree on five inputs. The server sees thousands of inputs an hour, and some of them appear in no test.

function canSeeNew(u: User, p: Project): boolean {
  if (p.isPublic || p.owner === u.id) return true;
  if (p.team.members.includes(u.id)) return true;
  if (u.bannedFrom.includes(p.workspace)) return false;
  const g = p.guestLinks.find((l) => l.user === u.id);
  return !!g && g.expires >= now();
}

Run both on the same request

So the server checks the rest. The old code keeps answering every request, as it always has. The new code runs beside it on the same request, and its answer goes to a log instead of to the person asking. A wrapper of ten lines does the whole job.

function canSeeShadowed(user: User, project: Project): boolean {
  const old = canSee(user, project);
  try {
    const fresh = canSeeNew(user, project);
    log({ user: user.id, project: project.id,
          old, fresh, same: old === fresh });
  } catch (err) {
    log({ user: user.id, project: project.id, old, error: String(err) });
  }
  return old;
}

The person asking always gets old. When the new code throws, the wrapper catches the error, and the person still gets old. The new code can be wrong all morning, and the only place that shows it is the log.

This is called a shadow read. The new code runs in the shadow of the old one, on real requests, and only the old answer goes out. GitHub wrote a library for it in 2014, called it Scientist, and used it to rewrite the code that decides who can see which repository, a job that took years.

Read the log

A hundred requests in, the log holds three differences and one crash, and those four lines are the whole review of the rewrite. Each line has a person, a project, and two answers, and for each line the question is which answer is right. A difference only says that the two versions disagree. It does not say which one is wrong.

Two of the three differences came from the rewrite. The third came from the old code, which had been wrong for years, for a few hours a day, without anyone noticing. A shadow finds bugs on both sides, because it only reports disagreement. GitHub saw the same thing when it moved its merge button onto a new git library. That shadow run found five real bugs, and two of them were in Git itself. The run ended after 24 hours at full traffic with zero differences across tens of millions of merges, and the whole job took a working week of part-time effort.

Fix, run again, switch

The AI fixes the three bugs and the crash, and the fix goes unread too. The shadow runs again, for a day this time, and the log stays empty. Then the switch is one line at the call site, and the old code and the wrapper leave in the same commit.

-  const allowed = canSeeShadowed(user, project);
+  const allowed = canSeeNew(user, project);

From now on the new code answers. GitHub's rule for this step is the same. When the differences reach zero, delete the wrapper and the old code together, because a shadow that stays on becomes a second copy of the logic that nobody maintains.

Only for reads

A shadow is safe for a read, because a read can run twice without changing anything. A write is different. If both versions charge a card, the customer pays twice. Scientist's own warning says the same, that it is only safe for code that does not change data. For writes there is a different method, the dual write, where the new code writes to a new place beside the old one and the reads compare the two places.

In front of the server

The shadow does not have to live inside one function. It can sit in front of the whole server. A load balancer sends every request to the old server and a copy of it to the new server, and everyone still gets only the old server's answer. The copy's response is dropped, so the compare runs over the two servers' logs, or through a tool that holds both answers. Twitter built Diffy for that in 2015 and gave it a third copy, a second old server, so the noise from timestamps and random ids could be measured and taken out.

location /api/ {
  mirror /shadow;
  proxy_pass http://old;
}
location = /shadow {
  internal;
  proxy_pass http://new$request_uri;
}

Zalando, the European fashion retailer, rebuilt its returns service behind the same idea. The old monolith answered the person first, then sent a copy of each request, with its own answer attached, to the new service, which ran the compare on its own time. Sam Newman's microservices book calls this a parallel run. Google calls it a dark launch, ramps the copied traffic from a small share up to all of it, and holds it there for a full day before the switch. When Zalando's log went quiet, the cleanup removed about seven hundred lines of old code and thirteen hundred lines of tests.

Why this matters more now

Three things changed at once. The first is volume. Theo reads about a thousand lines a day and generates more than two thousand, and that gap keeps growing. The second is the tests. When the same model writes the code and the tests, and it misreads the task, the code and the tests share the misreading, and the tests still pass. The third is the cost of the check. A shadow does not need the rewrite to be readable, or small, or in a language you know. It needs the old code, real requests, and a log, and all three already exist. Theo's own advice for AI work is to give the model "some method it can use to know things work." For a refactor, the old code already is that method. It has applied every rule on every request for years.

The real requests do the other half of the work. A test suite holds the inputs someone thought of. The morning's traffic holds every shape the data has actually taken, the nested team, the link that ran out this morning, the project with no team. GitHub's write-up on Scientist calls production data "the only true test of its correctness."

What to do

1. Pick one read, an old one that nobody fully understands anymore.
2. Let the AI rewrite it, and leave the rewrite unread.
3. Wrap the call so the old code still answers, the new code runs beside it, every error in the new code is caught, and every difference is logged with its input.
4. Let it run on real traffic for a day, and read every difference, because each one is either a bug in the rewrite or a bug you already had.
5. When the log stays empty, switch, and delete the old code and the wrapper in the same commit.

In the example, the old code reviewed the rewrite on every request, and on a real server that review runs thousands of times an hour. Nobody read the new code, and the method never needed anyone to.

Sources

• Scientist: Measure Twice, Cut Over Once
• Move Fast and Fix Things
• github/scientist project page
• Diffy: Testing services without writing tests
• Theo, Stop Reading Every Line of Code
• Theo, How I code with AI right now
• Zalando, Parallel Run Pattern
• Google CRE, What is a dark launch?