How I check a refactor I did not read

I let the AI write the code, and I read less of it every week, on purpose. Theo puts the stance in one line in a video about reading code, "If you're still reading all of the code, you're not generating enough of it." Every commit I describe was written by an AI coding session. My part was to set the bar, read the evidence, and decide each switch.

Not reading the code leaves one question open, which is how I know the rewrite returns the same answer as the old code for every input a real user sends. I answer it with production traffic instead of with my eyes.

One function, one answer

The rewrite I did not read starts in one function that reads one user's subscription status. It takes a user id, runs one query for that user, and returns the status.

async function getSubscriptionStatus(userId: string) {
  const user = await getUserById(userId);   // one query for one user
  return subscriptionStatusOf(user);
}

The refactor may change anything inside it, as long as every input that got an answer before gets the same answer after. That is the whole rule, and the rest of the method exists to check it.

The reason to touch the function was repeated work inside single requests. One window of 10 to 15 minutes of production traffic held about 230,000 extra reads of user rows, because helpers five or six deep in a request each fetched their own copy of the row. The AI session removed the repeats in two ways. It fetched the shared values once near the top of a procedure and passed them down as parameters, and where a loop called the function once per user it fetched all of the users in one query first.

// before, inside a loop over the league members
const status = await getSubscriptionStatus(member.userId);

// after, one query for the whole league, then a map
const users = await getUsersByIds(memberIds); // where userId in (...)
const userById = new Map(users.map((u) => [u.userId, u]));

That query is the code I never read line by line, and the record of agents on exactly this kind of change is the reason I could not take it on trust. On a benchmark of 1,099 behavior-preserving refactorings from real projects, an agent kept behavior in 39.4 percent of the cases that ask for more than one change at once. So instead of reading the query, I arranged to see its answer next to the old one, on every request, for as long as it took to trust it.

Two versions, same input

Seeing the new answer next to the old one means keeping both versions alive in the same request. The old version stays where it is, and its own read stays the source of the answer the caller gets. The new version's value reaches the function through an optional parameter, which the caller fills from the map it already holds.

async function getSubscriptionStatus(
  userId: string,
  preloadedUser?: User,
) {
  const fetchedUser = await getUserById(userId); // old read, served
  return subscriptionStatusOf(fetchedUser); // preloadedUser unused
}

// the caller passes down the row it already holds from the batch query
const row = userById.get(userId);
const status = await getSubscriptionStatus(userId, row);

When the argument is left out, the function returns the same bytes as before, so every untouched caller is exactly as it was. A library that a large code host published for this purpose wraps both bodies in one call. There the old code is called the control and the new code the candidate. Running the new version beside the old one while the old one keeps answering is called a shadow read. There are now two answers for one input, and nothing yet that compares them.

Compare the two answers

Comparing them starts with a decision about what the same means, because two objects that hold the same values are not the same string unless their fields come out in the same order. The helper in my code settles that first. Both values become text with a fixed key order, and the two strings are compared.

import { configure } from 'safe-stable-stringify';

const toText = configure({ deterministic: true }); // fixed key order

function emitShadowSpan(
  site: string, preloaded: unknown, fetched: unknown,
) {
  const a = toText(preloaded) ?? 'undefined';
  const b = toText(fetched) ?? 'undefined';
  const match = a === b;
  const span = tracer.startSpan('request-read-dedupe-shadow', {
    attributes: {
      'dedupe.site': site,
      'dedupe.match': match,
      ...(match
        ? {}
        : { 'dedupe.preloaded': a, 'dedupe.fetched': b }),
    },
  });
  span.end();
}

The function calls it right after its own read, and only when a caller passed the new value in. Every call that reaches it adds one span and one verdict, match or mismatch, and only a mismatch carries both values.

  const fetchedUser = await getUserById(userId);
  if (preloadedUser !== undefined) {
    emitShadowSpan('viewerUser', preloadedUser, fetchedUser);
  }
  return subscriptionStatusOf(fetchedUser);

That one call is all the comparison costs the function. Text equality is the strictest rule it can apply, and not the only one. The library compares with plain equality or with a block of code you supply, treats one side raising an error while the other returns as a mismatch, and when both raise, compares the kind of error and its message. Whatever the rule, some differences come from chance rather than from a fault. Timestamps, random values and ordering differ between two runs of the same code. A proxy that one team published for comparing two versions of a whole service measures that noise by running two copies of the known-good code against each other, and a single true-or-false value that flipped at random in a response produced 25 percent false errors on its own. The cheap form of the same measurement is a baseline window where both sides call the old code, run before any count is trusted.

// baseline: both sides run the old code, every mismatch is noise
const fetchedUser = await getUserById(userId);
emitShadowSpan('viewerUser', await getUserById(userId), fetchedUser);

Real requests, old answer served

A baseline window only means something if the requests in it are real, and the same is true of the shadow read itself, so the input is production traffic. Around a whole service the same arrangement is called shadow testing or shadow deployment. During that window the system does more work, because the new query runs on top of the reads it may later replace.

The extra work is allowed only because the user must not notice any of it, and every guard follows from that rule. Errors thrown by the new code are recorded and dropped instead of reaching the user.

let userById = new Map<string, User>();
try {
  const users = await getUsersByIds(memberIds); // the new version
  userById = new Map(users.map((u) => [u.userId, u]));
} catch (error) {
  recordShadowError('getUsersByIds', error); // traced, never thrown
}

When both versions throw the same error, that counts as a match, since the old behavior for that input was the error. The team that wrote the library first used it to move the merge button on its own site onto a new code path, and the guards on that merge rewrite went further. Candidates slower than 5,000 ms were logged and reviewed each morning, and the comparison ran for 1 percent of requests before it was raised to all of them.

A missing guard shows up in production. Mine did. A shadow on a different code path read two keys from Redis in one command. In Redis Cluster one command may touch only one slot, the two keys hashed to different slots, and every one of those commands failed with CROSSSLOT. Each failure tripped the circuit breaker on the Redis read path, requests skipped the cache in front of the database, and the shadow damaged the path it was watching. The fix was two single-key reads, in place within the hour, and both changes were reverted that evening.

- const [version, payload] = await redis.mget(versionKey, payloadKey);
+ const version = await redis.get(versionKey);
+ const payload = await redis.get(payloadKey);

Count the matches per call site

With the guards in place, every compared request leaves one span behind, and the spans are where the review happens. My server already traces every call with its full arguments and result, with no sampling, so the comparison spans sit beside everything else. One query over a window of production traffic groups them by call site and by outcome.

select attributes['dedupe.site'] as site,
       attributes['dedupe.match'] as match,
       count() as n
from spans
where name = 'request-read-dedupe-shadow'
  and timestamp > now() - interval 40 minute
group by site, match

Each call site comes back as one row per outcome, so a site with no differences has one row, a site with differences has two, and the rows of a site add up to the pairs compared there. The team behind the proxy stated the reason to count. Two versions are equivalent if their responses are similar over a large and diverse enough set of requests. The idea needs many requests, which is why mine ran on all traffic rather than on a sample, and the differences row is the one to watch.

The first round produced 180,730 comparisons across all five call sites in 45 minutes with 0 mismatches, after which the comparison code was deleted. A later round produced 77,051 comparisons in 30 minutes with 0 mismatches, split across three sites at 27,825, 27,828 and 21,398. One round that week did produce mismatches, all of them at one of its two sites, and its 40-minute window gives these rows.

site                           match   n
participantSubscriptionStatus  true    12023
viewerUser                     true     6994
viewerUser                     false       9

Each per-site count sits below the number of requests in the window, because not every request reaches every call site. The nine in the last row are the ones that had to be read.

Read the mismatches, not the code

A mismatch can mean one of three things. The new code is wrong, the old code is wrong, or the data changed between the two reads. The nine came from the round where the new code read whole user rows, many users in one query, and the site still matched 99.87 percent. All nine were writes landing between the two reads, milliseconds apart. Four differed only in the version counter, the number optimistic locking raises on every write, two in a rating update, and three in a count of finished games. One of the finished-game pairs has this shape, with example values and the unchanged fields cut.

dedupe.site       viewerUser
dedupe.match      false
dedupe.preloaded  { ..., "numberOfGamesPlayed": 208, "version": 91 }
dedupe.fetched    { ..., "numberOfGamesPlayed": 209, "version": 92 }

So only the third cause was present. In another round one site matched 93.79 percent, and every difference sat in two fields the function never reads, so both callers still got the same star total, the one field they use.

The other two causes are real bugs, on either side. The merge rewrite found one in the time it wrote on each merge, because the two versions ran more than a second apart, and passing the time in as an argument fixed it. The same comparison found five real bugs in total, two of them in code the rewrite depended on rather than in the rewrite itself.

So I read nine pairs instead of the set query the AI session wrote, and the match count in production was the test that decided whether it shipped. Tests written by the same agent would not have replaced that, because a model can misread a requirement and write tests that encode the same misreading. The bar for letting the new version answer users is a match effectively at 100 percent with every mismatch explained, and until a round gets there the loop is to investigate, fix the cause, and rerun.

Switch, then delete

Once every mismatch has a cause, the round ends in one commit. The new version answers from then on, and the comparison code goes.

 async function getSubscriptionStatus(
  userId: string,
  preloadedUser?: User,
) {
-  const fetchedUser = await getUserById(userId);
-  if (preloadedUser !== undefined) {
-    emitShadowSpan('viewerUser', preloadedUser, fetchedUser);
-  }
-  return subscriptionStatusOf(fetchedUser);
+  const user = preloadedUser ?? (await getUserById(userId));
+  return subscriptionStatusOf(user);
 }

The ?? keeps the old single-user read for exactly one case, a user missing from the new query's results, so the old behavior for a missing row is preserved to the byte. A fallback like that stays only where the new version can miss an input. In most rounds the switch was the commit itself, and in one it was a feature flag that picked the new version first and was deleted in the same commit as the comparison helper. That helper file was created, deleted, restored and deleted again across four rounds, because no comparison code stays after a switch. In the round with the nine mismatches, the switch came half an hour after the commit that added the comparison.

The merge rewrite ended the same way at a larger scale. Its end state was 24 hours with zero mismatches over tens of millions of merges, after about five days of part-time work, and then the comparison code was removed and the old code deleted.

Where it stops

Every round so far compared a function that only reads, and the method does not reach further than that. The library's own rule is reads only and no side effects, because the candidate is not guaranteed to run. A function that takes a life from a player after a loss is the other kind, and two versions of it running on one request take two lives.

async function consumeLifeAfterLoss(userId: string) {
  await db.updateTable('lives')
    .set((eb) => ({ count: eb('count', '-', 1) }))
    .where('userId', '=', userId).execute(); // twice, two lives gone
}

No comparison may run around a function like that. Even among reads, some are excluded and always go to the database again. These are a read that holds other work back while it runs, a read taken in order to write, a deliberate read straight after a write, and reads that already run in parallel. They stay fresh because performance work must not add an invariant, which is an assumption other code must keep true from then on, such as that a value cannot change between two points in time. Holding a viewer's subscription status for the life of one open connection, about 141 status reads, was rejected for that reason, since a purchase during a game has to show up.

A change that moves data to a different store needs another pattern again, a dual write, where both the old and the new store are written for a period. The old store stays the one that is read.

async function saveRating(userId: string, rating: number) {
  await oldStore.saveRating(userId, rating); // source of truth
  try {
    await newStore.saveRating(userId, rating); // second write
  } catch (error) {
    recordShadowError('saveRating.newStore', error);
  }
}

The comparison then happens when the data is read, the differences are fixed, the read paths move to the new store before the write paths, and the old data moves last. The second write stays well past the switch so that a rollback is still possible.

What to do

1. Pick one read function and leave it serving users exactly as it does today.
2. Add the new version behind an optional parameter.
3. Compare the two answers as text and record one result per comparison.
4. Run it on real traffic with the old answer served and the new version's errors dropped.
5. Open every mismatch one at a time and give each one a cause before you switch.
6. Let the new version answer, delete the helper and any flag in the same commit, and keep the old read only where the new version can miss an input.

The place to start is your traces. Count how many times one read function runs inside a single request, and if the number is far above one, that function is the one to start with. The team behind the merge rewrite calls production data the only true test of a rewrite's correctness, and that is why the diff was never the review.

Sources

• Diffy: Testing services without writing tests
• Move Fast and Fix Things
• Scientist: Measure Twice, Cut Over Once
• github/scientist project page
• Online migrations at scale
• pt-upgrade
• Shadow Testing (Microsoft Engineering Playbook)
• Evidence-Carrying Changes
• SWE-Refactor
• Theo (t3.gg), Stop Reading Every Line of Code