AI refactors
The code I did not read
August 2026
An AI rewrote a piece of my server, and I haven't read the rewrite. Theo, who runs t3.gg, put it in one line. "If you're still reading all of the code, you're not generating enough of it." I think he's right. But somebody still has to check the code, and on my server that job goes to the real users, through a shadow read. The users keep getting their answers from the old code while the new code answers the same requests on the side, and I compare the pairs.
One function, asked too often
The function takes a user and says whether that user pays for the app. The answer lives in the database, and each call is one read, which is fine on its own. The trouble started inside one request, where small helpers five or six layers down each asked the question again for themselves, and one request ended up reading the same row over and over. In one window of 10 to 15 minutes the user row alone took about 230,000 reads it didn't need.
The answer doesn't change inside a request, so the fix the AI wrote reads the row once near the top of the request and passes the answer down through the helpers. After that nothing below asks the database again.
Run both, serve the old one
Because I wasn't going to put unread code in front of users, the old readStatus function stayed where it was and kept running on every request. The new code ran beside it, with the same request and the same userId, and made its own answer. Only the old answer went to the user. The new one went into a comparison instead, and if the new code is wrong nobody using the app finds out.
Compare the two answers
The comparison turns each answer into text and checks whether the two texts are the same. Each check leaves one line in the trace store, where my server already records every call, and the line names the place in the code and says match or mismatch. On a mismatch the line also has both answers in it for me to read later, while a match has nothing else.
At the call site the whole thing is a few lines. The old read still runs, as fresh, and fresh is still what gets returned. The new answer comes in as preloaded, and when it's there the two get compared, but when it isn't the function does what it always did.
const fresh = await readStatus(userId);
if (preloaded !== undefined) {
emitShadowSpan({ site, preloaded, fetched: fresh });
}
return fresh;Let real users do the testing
There's no test input in those lines, because the comparison runs on every real request that reaches the function, with whichever user that request came from. A test I write covers the cases I can think of, and real traffic brings the ones I can't, like a user whose row is being written at the same moment the function reads it. After 45 minutes the first round had 180,730 comparisons across five places in the code, with 0 mismatches. That count was enough for me, and I still haven't looked at the new code.
Nine pairs that differed
A second shadow went live on two other places in the code, and that round wasn't clean, because one of the two places had 7,003 comparisons and 9 of them were mismatches. A mismatch line has the two answers but not the reason they differ, and because nine is not many I read all nine.
They all had the same shape. The old code and the new code had read the same user a few milliseconds apart, and a write had landed in between. In 4 of the 9 pairs only the version counter had moved. The database bumps that counter on every save of a row, and a bump on its own tells me a save landed between the two reads and nothing else in the row changed. In 2 of them a rating update had gone through as well, and in the other 3 the count of games played had gone up because a game had just ended.
The new code's read came a moment earlier, so both answers were right at their own moment and the difference came from the save between them. The other place in that round had 12,023 comparisons and no differences at all, and I switched 30 minutes after that shadow went live.
Switch, then delete
When the count says the two answers agree, the shadow has nothing left to do, and the switch is one more small change. From then on the user gets the new answer. The old readStatus read runs only when a user has no preloaded value, and a user with no row at all still gets the same answer as before. The comparison line goes too.
-const fresh = await readStatus(userId);
-if (preloaded !== undefined) {
- emitShadowSpan({ site, preloaded, fetched: fresh });
-}
-return fresh;
+return preloaded ?? (await readStatus(userId));Only for reads
The shadow could stay a few lines from start to finish because readStatus only reads, and a read can run twice without doing any harm. A write can't. One function on my server takes a life from a player who has just lost a game, and if both versions of it ran, the player would pay two lives for one loss. I'd never put a shadow around that one.
GitHub has had a library for this for ten years, called Scientist, and its rule is the same as mine, that it only goes around code that leaves the data alone. A write gets a dual write instead. For a while every write goes to both the old place and the new place, and the two get compared when something reads them.
Some reads stay out of a shadow as well, because they have to come from the database on every call. A read held under a lock is one, and a read whose value is about to be written back is another. The same goes for a read that comes right after a write, or one that already runs side by side with another read.
When the shadow itself breaks
Even a read that can run twice has a cost, because both versions run and each request does more work while the shadow is on. I expected that part. I didn't expect a bug in the shadow code itself, but one of mine had one.
That shadow's new code asked the cache for two keys in one call, but my cache is clustered and the two keys lived in different places, and one call can't reach two places. Every call failed, and the failures tripped the breaker that protects the cache, which left the real cache path unused for 37 minutes until the fix went out. I took the new code out and the shadow went with it.
The library has guards for exactly this. They go in before the shadow does, and when the new code throws, the library catches the error and counts it, and the user gets the old answer as if nothing had happened. At first it also runs the new code on only a share of the requests. My shadow had none of that, and I'll be honest, it should have.
GitHub had those guards on when it moved the merge button of its own site onto new code with the library. They started at 1 percent of requests and went up to 100, and after 24 hours they stopped with zero mismatches over tens of millions of merges. The whole thing took five days of part-time work, and then the old code got deleted.
What to do
If you want to try this on your own server, pick a function that only reads, the way readStatus does, let the AI rewrite it, and don't read the rewrite. Keep the old version serving your users and run the new one beside it, with the same userId going into both and one line written per comparison. After a window of real traffic, count the lines for each place in the code, then read the mismatches and nothing else, because the count already covers the matches. Once you can explain every mismatch, switch and delete the shadow the same day, because a shadow left in place doubles the reads on every request.
Put the guard in first, though. I skipped it on the cache shadow, and that's how the breaker tripped.