Same work, different judge, different winner

You have two close options in front of you, and they could be two candidates, two designs, two ad variants, or two hackathon projects. You hand both to an LLM and you take its pick. I have done this, and I wanted to know whether the pick came from the work.

A verdict that comes from the work should survive changes that never touch the work. Swap the reader, or change the shape of the answer the judge has to give, and the winner should stay put. I built a close pair to run exactly that test.

I wrote two fictional 48-hour hackathon submissions that had the same core feature list. Both had two-way calendar sync, a task inbox with AI triage, automatic reschedule, and an end-of-day summary.

Rehearse shipped a new mechanism: a small per-user delay model trained on the last 8 weeks of calendar history. Each morning it ran 1,000 Monte-Carlo rollouts of the day's plan on-device, with no cloud calls. During the weekend, the rehearsal caught a real conflict four hours before it happened, a standup overrun colliding with a dentist booking. Execution was rough. There was a bug in the write path, and the app crashed once during the live demo. They had no tests, and the users so far were the two of us.

Dayline shipped the standard mechanism, a well-prompted hosted model, and the team spent the weekend on dependability instead. Onboarding took three steps and finished in under 60 seconds. The screens passed WCAG AA, and test coverage was 92 percent. They deployed on the first day and 41 people signed up, of whom twelve were still planning daily on the second day, giving 29 percent day-two retention. Twice in the live demo they pulled the network cable, and the offline queue replayed every edit both times. There were zero crashes.

I wrote two judging skills of equal length, one for innovation and one for execution, each with a hard-edged calibration section. The prompt was both skills, both briefs, and a footer that forced exactly one winner.

I expected the work to decide, since the pair was tuned close. Here is what a verdict that comes from the work would look like. You swap the model and the name stays, you ask for scores before the name and the name stays, and the reasons stay close to the words in the brief.

Then I changed who read it.

On the same briefs, the same skills, and the same free-form footer, Opus picked Rehearse in 10 of 10 trials. Sonnet picked Dayline in 10 of 10, and Haiku picked Dayline in 9 of 10. Same evidence, same rubrics, and the models still named opposite winners. Neither of them reported any uncertainty.

If this is your eval suite, then the model you hired is the result.

The models did not weigh the two skills against each other. They quoted them. Opus, in both reading orders, repeated the innovation close: "polish is far cheaper to add later than a new mechanism." It argued the bug down as "known-cause rough edges ... with no data loss ... not a core flow that corrupts data." Sonnet, in both reading orders, repeated the execution close instead, which was "the calibration explicitly treats [it] as disqualifying regardless of ambition." Reading "vanished from the calendar view until the next sync" and "Nothing was lost," it called that data corruption. Haiku's single Rehearse vote was the one trial that latched onto the other punchline, "a working new idea outweighs polish."

Later I deleted the bug bullet entirely, and Sonnet still named Dayline in 5 of 5, citing the demo crash, the missing tests, and the zero outside users.

Skill order and product order did not move a verdict in 100 trials. Twelve one-line wording edits did not move one either, since 59 of 60 still went to Rehearse.

The next change that moved the name was the answer format.

I took the harsher bug line, which read "twice this weekend it dropped an event from the calendar. We restored both by hand from the sync log," and I ran it two ways.

When the footer said "Apply both judging skills, weigh them as you see fit, and choose exactly one overall winner," Opus picked Rehearse in 5 of 5.

Then I swapped in a footer that said "Score both submissions on both lenses, then choose exactly one overall winner" and demanded four numeric scores first. Opus picked Dayline in 4 of 5 on the rerun, and in 9 of 10 the first time I used that pair, which together is Dayline in 13 of 15. The harsh line without the scores stayed Rehearse, and so did the scores without the harsh line.

The winner changed because I asked for numbers first.

If this is your triage queue, then the output schema is part of the decision.

Then I looked at the score sheets from the five-trial rerun.

Trials 2 through 5 wrote the same row: 8 and 3 for Rehearse, 3 and 9 for Dayline. The product totals were 11 and 12, and those four trials still split 3 to 1 on the winner. Trial 1 wrote 9, 3, 3, 9, a 12 to 12 tie, and it named Dayline anyway.

The numbers did not decide. They were filled in afterward, almost identically, either way.

If you collect score sheets from a model judge, you are collecting the justification and not the cause.

Then I read the written reasons.

The source text said that an event was dropped and that both were restored by hand from the sync log. Nothing was permanently lost, the team noticed both, and the users so far were the two of us.

The judges' written reasons rendered that as:

• "silently duplicates and drops calendar events"
• "silently dropped events from real calendars twice"
• "silently duplicated and twice deleted real calendar events"
• "has twice destroyed calendar events"
• "a core flow that corrupts user data"

Dropped became deleted, deleted became destroyed, and destroyed became corruption. "We restored both" turned into "silently," and two test calendars turned into "real calendars." Each retelling goes further than the text does, and the verdict then leans on the retelling.

The two trials that shared the 8, 3, 3, 9 row and named opposite winners make the order of operations visible.

The Dayline vote said the core path "silently duplicated and twice deleted real calendar events" and that "the execution lens treats a core flow that corrupts user data as a failure, whatever it invented."

The Rehearse vote said "dropped calendar events are data loss in a core flow," and then it turned: "the innovation lens is explicit that a working novel mechanism outweighs polish." It added that Dayline "is by its own account a well-prompted wrapper on the familiar planner set."

Same scores, different stories. Each story was written to fit the pick it came with.

The writeup in your queue is the decision retold, and it is not a reading of the brief.

The verdict does not come from the work. It comes from the judging setup, and it depends on which model reads the pair and on what answer format you demand. The scores and the reasons get written after the decision, to fit it.

You already run this loop. Two candidates go in and one name comes out, and the pick feels like a reading of the work. It is a reading of the setup.

Here is a check you can run tomorrow on a pair you have already judged. Rerun it on a second model, then rerun it once more and ask for the scores before the winner. If the name changes, then you were not measuring the work.