A refactor I did not read

I let an AI coding agent write the code, and I read less of it every week, on purpose. Theo says it in one line, "If you're still reading all of the code, you're not generating enough of it."

That leaves one question open, whether the rewrite still gives every answer the old code gave. I answer it with production traffic instead of with my eyes.

One function, one answer

The rewrite I did not read starts in one function that reads one user's subscription status. It takes a user id, runs one query for that user, and returns the status.

async function getSubscriptionStatus(userId: string) {
  const user = await getUserById(userId);   // one query for one user
  return subscriptionStatusOf(user);
}

The refactor may change anything inside it. Every answer it gave before must stay the same.

The reason to touch it was repeated work. In one window of live traffic I counted about 230,000 extra reads of user rows, because helper functions five or six deep each fetched their own copy. The agent fetched shared values once near the top and passed them down as parameters, and replaced loops that read one user at a time with one query for all of them.

const users = await getUsersByIds(memberIds); // where userId in (...)
const userById = new Map(users.map((u) => [u.userId, u]));

That query is the code I never read. In one test set of 1,099 rewrites that had to keep every answer, an agent got only 39.4 percent right when the rewrite touched several places at once. So instead of reading the query, I arranged to see its answer next to the old one on every request.

Two versions, same input

Seeing both answers means both versions run in the same request. The old version stays in place and keeps answering users. The new version's value reaches the function through an optional parameter, which the caller fills from the map it already holds.

async function getSubscriptionStatus(
  userId: string,
  preloadedUser?: User,
) {
  const fetchedUser = await getUserById(userId); // old read, served
  return subscriptionStatusOf(fetchedUser); // preloadedUser unused
}

// the caller passes down the row it already holds from the batch query
const row = userById.get(userId);
const status = await getSubscriptionStatus(userId, row);

Callers that pass nothing are unchanged. Running the new version beside the old one while the old one keeps answering is called a shadow read. A library that a large code host published for this takes both versions in one call, where the old code is called the control and the new code the candidate. Either way there are now two answers for one input, and nothing has compared them.

Compare the two answers

Comparing them starts with a decision about what the same means, because two objects with the same values are not the same string unless their fields come out in the same order. My helper settles that first. Both values become text with a fixed key order, and the two strings are compared.

const toText = configure({ deterministic: true }); // fixed key order

function emitShadowSpan(
  site: string, preloaded: unknown, fetched: unknown,
) {
  const a = toText(preloaded) ?? 'undefined';
  const b = toText(fetched) ?? 'undefined';
  const match = a === b;
  tracer.startSpan('request-read-dedupe-shadow', {
    attributes: {
      'dedupe.site': site,
      'dedupe.match': match,
      ...(match
        ? {}
        : { 'dedupe.preloaded': a, 'dedupe.fetched': b }),
    },
  }).end();
}

The function calls it right after the old read, and only when a caller passed the new value in. Every call that reaches it adds one span and one verdict, match or mismatch, and only a mismatch carries both values.

  if (preloadedUser !== undefined) {
    emitShadowSpan('viewerUser', preloadedUser, fetchedUser);
  }

Some differences come from chance rather than from a fault. Timestamps, random values and ordering differ between two runs of the same code. One published proxy measures that noise by running two copies of the known-good code against each other, and one true-or-false value that flipped at random produced 25 percent false errors by itself. The cheap form of the same check is a baseline window where both sides call the old code, so every difference it shows is noise.

Real requests, old answer served

A baseline window only means something on real requests, and the same is true of the shadow read itself. The input is production traffic, and the user still gets the old answer.

That setup needs one guard. Failures in the new code are recorded and dropped instead of reaching the user.

let userById = new Map<string, User>();
try {
  const users = await getUsersByIds(memberIds); // the new version
  userById = new Map(users.map((u) => [u.userId, u]));
} catch (error) {
  recordShadowError('getUsersByIds', error); // traced, never thrown
}

A missing guard shows up in production. Mine did. A shadow on another code path asked Redis for two keys in one command. In Redis Cluster one command may touch only one slot, the two keys hashed to different slots, and every such command failed with CROSSSLOT. Each failure tripped the circuit breaker on the Redis read path and sent requests to the database instead, so the shadow damaged the path it watched. I put two single-key reads in place the same hour, and the shadow and the fix were both undone that evening.

- const [version, payload] = await redis.mget(versionKey, payloadKey);
+ const version = await redis.get(versionKey);
+ const payload = await redis.get(payloadKey);

Count the matches per call site

With the guard in place, every compared request leaves one span behind, and you count the spans. One query over a window of production traffic groups them by call site and by outcome.

select attributes['dedupe.site'] as site,
       attributes['dedupe.match'] as match,
       count() as n
from spans
where name = 'request-read-dedupe-shadow'
  and timestamp > now() - interval 40 minute
group by site, match

A site with no differences gives one row, a site with differences gives two, and the rows of a site add up to the pairs compared there. The first round gave 180,730 comparisons across all five call sites in 45 minutes with 0 mismatches. One round that week gave mismatches, all at one of its two sites, and its 40-minute window gives these rows.

site                           match   n
participantSubscriptionStatus  true    12023
viewerUser                     true     6994
viewerUser                     false       9

The method rests on one idea, that two versions with the same answers across a large and varied set of real requests do the same job. The differences row is the one to read.

Read the mismatches, not the code

A difference has three possible causes. The new code is wrong, the old code is wrong, or the data changed between the two reads. The nine came from the round where the new code read whole user rows in one query, and that site still matched 99.87 percent.

All nine were writes landing between the two reads, milliseconds apart. Four differed only in the version counter, the number optimistic locking raises on every write, two in a rating update, and three in a finished-game count. One of the finished-game pairs has this shape, with example values and the unchanged fields cut.

dedupe.site       viewerUser
dedupe.match      false
dedupe.preloaded  { ..., "numberOfGamesPlayed": 208, "version": 91 }
dedupe.fetched    { ..., "numberOfGamesPlayed": 209, "version": 92 }

So only the third cause was present. In another round, one place matched 93.79 percent, and every difference sat in two fields the function never reads, so both callers still got the same star total, the one field they use.

The match count in production was the test that decided whether that query shipped. Tests written by the same agent would not replace that check, because a model can misread a requirement and write tests that encode the same misreading. The nine pairs were the whole reading job, and each one had a cause.

Switch, then delete

Once every mismatch has a cause, the round ends in one commit. The new version serves from then on, and the comparison code goes.

 async function getSubscriptionStatus(
  userId: string,
  preloadedUser?: User,
) {
-  const fetchedUser = await getUserById(userId);
-  if (preloadedUser !== undefined) {
-    emitShadowSpan('viewerUser', preloadedUser, fetchedUser);
-  }
-  return subscriptionStatusOf(fetchedUser);
+  const user = preloadedUser ?? (await getUserById(userId));
+  return subscriptionStatusOf(user);
 }

The ?? keeps the old one-user query for one case, a user missing from the set query results, so the old behavior for a missing row is preserved. In one round a feature flag picked the new version first and was deleted in that same commit. The team that wrote the library ran the same order on the merge path of its own site and ended with 24 hours of zero mismatches, after which the comparison code and the old code were deleted.

Where it stops

Running both versions was safe only because the function writes nothing, and that is where the method stops. The library's own rule is reads only, because the new version is not guaranteed to run. A function that takes a life from a player after a loss is the other kind, and two versions of it on one request take two lives.

async function consumeLifeAfterLoss(userId: string) {
  await db.updateTable('lives')
    .set((eb) => ({ count: eb('count', '-', 1) }))
    .where('userId', '=', userId).execute(); // twice, two lives gone
}

No comparison may run around a function like that. Even among reads, some are excluded and always go to the database again. These are a read that holds other work back while it runs, a read taken in order to write, a deliberate read straight after a write, and reads that already run in parallel. The rule behind those exclusions is that a speed fix must not add an invariant, an assumption other code has to keep true from then on. Holding a viewer's subscription status for the life of one open connection, about 141 status reads, was rejected on that rule, since a purchase during a game has to show up.

A change that moves data to a different store needs another pattern, a dual write. You write to both stores for a while, compare the two when you read, and move the read paths before the write paths.

What to do

1. Pick one read function and leave it serving users exactly as it does today.
2. Add the new version behind an optional parameter.
3. Compare the two answers as text and record one result per comparison.
4. Run it on real traffic with the old answer served and the new version's errors dropped.
5. Open every mismatch one at a time and give each one a cause before you switch.
6. Let the new version answer, delete the helper and any flag in the same commit, and keep the old read only where the new version can miss an input.

Count how many times one read function runs inside a single request, and pick the worst one. The team that wrote the library calls production data the only true test of a rewrite, and that is why the diff was never the review.

Sources

• Diffy: Testing services without writing tests
• Move Fast and Fix Things
• Scientist: Measure Twice, Cut Over Once
• github/scientist project page
• Online migrations at scale
• pt-upgrade
• Shadow Testing (Microsoft Engineering Playbook)
• Evidence-Carrying Changes
• SWE-Refactor
• Theo (t3.gg), Stop Reading Every Line of Code