Run the shadow

An AI rewrote part of my server last week. I did not read the rewrite, and I am not going to. Theo said why in one line. "If you're still reading all of the code, you're not generating enough of it." So something other than my eyes has to check the code, and on my server that something is real users.

This page puts you in my seat. The check runs in your browser, on a copy of the function, the way it ran on my server. You will not see the rewrite either, and by the end you will know why that was fine.

1. The old read

One function on my server reads a user's status. Each request asks for it once, and the helpers inside the request ask again, five or six layers deep. That came to about 230,000 extra reads every 10 to 15 minutes. The AI rewrote the request to read once at the top and pass the answer down.

Start with the old read alone. Press the button, and each press is one request from a user. The old read answers it, and the answer goes back to the user.

2. Run both, serve the old one

Now the rewrite runs too. On each request the server runs the old read and the new read. The old answer goes to the user, exactly as before. The new answer goes to one trace line and nowhere else. This is a shadow read. The new code runs in the shadow of the old code, and nobody sees its answer but me.

Press the button again and watch where each answer goes. Then press the last button, which lands a save on the user's row in the gap between the two reads.

A mismatch is not always a bug. The new read runs at the top of the request and the old read runs later, inside a helper, a few milliseconds apart. A save that lands in that gap makes the old read see the newer row. On a mismatch the trace line carries both answers, so I can open it and see exactly that.

3. Try to break it

The shadow must never hurt the user. So the new read runs inside a guard. If it throws, the error is caught, the trace gets an error line, and the old answer still serves. Use the button that breaks the new read, then send a few requests.

One of my shadows did break, in a way this guard did not cover. It read two cache keys in one call, the cache cluster rejected every such call, and that tripped a breaker which turned the real cache off for 37 minutes. The shadow hurt the thing it was testing. Both changes went back that evening. A shadow must be able to fail without anything else failing with it, and it gets a short window, so a bad one costs minutes and not a day.

4. Let real users run it

On my server the shadow ran on real traffic, the requests of real users while they played. Press play to replay a round at high speed. Every request in the count is one a real user made, and the shadow compared both answers on each one.

Round one, 180,730 compared at five places in 45 minutes, and none different. I read none of the code behind that number, only the count. Round two, at two other places, 19,026 compared in 40 minutes and nine different, all at one place, and nine is small enough to read.

5. Read the nine

Here are the nine pairs, as I saw them. Open each one in turn. The left answer came from the new read and the right one from the old read, and the field that differs is marked.

Every pair is a save that landed in the gap between the two reads. A fresh read taken at the new read's instant would have returned the same values, so the new code is right. That was the whole review, nine pairs and no diff.

6. Switch, then delete

Thirty minutes after the shadow went in, I switched. The new answer serves, and the old read runs only when the new answer is missing for a user, so a missing row behaves exactly as it did before. The compare line goes with it. Send a request, press the switch, and send one more.

At the place in the code, the switch is six lines. The old read moves into the fallback, and the compare goes.

-const fresh = await readStatus(userId);
-if (preloaded !== undefined) {
-  emitShadowSpan({ site, preloaded, fetched: fresh });
-}
-return fresh;
+return preloaded ?? (await readStatus(userId));

That is the whole life of a shadow. A few lines in, a count, nine pairs read, a few lines out.

7. Only for reads

A read can run twice without harm, and a write cannot. A second function on my server takes one life from a player after a lost game. Lose a game with the shadow off, then turn the shadow on and lose another.

One loss, two lives gone. So the shadow goes on reads only. A write needs a different tool, a dual write, where both versions write and a version counter on the row tells when they disagree. I did not need it here. Some reads stay out too. A read under a lock, a read that is about to be written back, a read right after a write, and two reads side by side each keep their own fresh call.

What to do

1. Pick one read that runs far too often.
2. Let the AI rewrite it, and do not read the rewrite.
3. Run both on real requests for half an hour or more. Serve the old answer. Catch every error from the new one.
4. Count the pairs per place. Open every pair that differs and explain it.
5. Switch, then delete the shadow the same day.

I still have not read the rewrite. I read nine pairs of answers, and now so have you.

Sources

• Move Fast and Fix Things
• Scientist: Measure Twice, Cut Over Once
• github/scientist project page
• Theo, Stop Reading Every Line of Code