The Override They Did Not Make

That gap, between what they said and what they did, is the whole story. It is also the story of the next decade.

We have installed the same safety mechanism nearly everywhere: a human in the loop. In grading, in hiring, in lending, in diagnosis, in sentencing, the premise is identical. A person stands between the algorithm's verdict and the consequence, ready to catch the error before it lands. The study showing teachers distrust AI grading yet accept its cruelest mistakes is a picture of that person standing exactly where the diagram says they should. It is not a picture of them catching anything.

The reason is not stupidity, and it is not laziness. It is cost. Distrust is free — you can hold it forever, report it on a survey, feel virtuous about it, and never spend a thing. Overriding is expensive. To override, you have to notice the error, believe your own judgment over the system's, document the deviation, and personally own the outcome if you turn out to be wrong. The machine's verdict arrives pre-justified. Yours arrives with a paper trail and a question hanging off it: who are you to disagree with the model? Every incentive in the room points toward deference. The distrust you declared cost nothing. The override would have cost you something, and so it never got made.

Cost is not the only thing in the room, and naming the other thing sharpens the fix rather than weakening it. The machine's number also anchors you — it quietly resets your sense of the right answer before the cost calculation ever runs, so that by the time you "decide," part of you has already agreed. That is not deference; it is the verdict getting inside your judgment before you knew to defend it. But cost is the part you can engineer around, and the anchor tells you exactly how: stop showing the answer first. Make the human commit a grade before the machine reveals its own. Attack the anchor, and the cost asymmetry has nothing left to guard.

This is the part we would rather not see. Human-in-the-loop is an architectural claim wearing the costume of a behavioral one. We built it to feel safe, then assumed the human would behave the way the flowchart promised. But a person is not a checkpoint. A person is a tired professional at eleven at night with forty essays left, looking at a number the machine already assigned, looking for a reason to move on. The diagram says review. The environment says approve.

You know this about yourself already. You have clicked accept on a verdict you did not fully audit — the recommended setting, the suggested rate, the flagged charge — because disagreeing asked more of you than the moment had to give. You were not careless. You were responding to a structure that made deference frictionless and made objection a small act of courage. Multiply that by every loop in every system and you arrive at a world that runs, quietly, on overrides nobody makes.

Coherenceism has a name for the mistake underneath this: moralizing where you should be designing. We keep trying to fix the behavior with exhortation — train the graders, remind the reviewers, publish guidelines about "meaningful human oversight." But you cannot lecture a person into spending courage they do not have to spare. If you actually want the override to happen, you have to build an environment where overriding is cheaper than deferring, where disagreeing with the machine is the effortless default and accepting it is the thing that demands a second look. We built the precise opposite. Then we placed a human inside it and called the arrangement a safeguard.

And maybe the arrangement is not naive at all. Consider that human-in-the-loop might be working exactly as designed — just not at the job we advertised. Its most reliable output is not caught errors; it is relocated blame. When the model is wrong, the institution that built and shipped it has, conveniently, manufactured a person standing in the loop to hold responsible. Seen that way the loop is not a failed safeguard; it is a working one, for whoever would otherwise own the mistake. Of course the environment makes override expensive. The system was never trying to get the override. It only needed the override-shaped position to exist, so the verdict it always meant to ship would arrive with someone's name on it.

The uncomfortable truth is not that the machines are wrong. We knew they would be wrong. We said so out loud, on the record, and deferred anyway — because the room was arranged so that deferring is what we would do. The loop closed. The human was in it. And the override they did not make is the one the rest of us are quietly counting on.

Want to know whether your own oversight is real? Don't ask whether you trust the system. Ask what it would cost you, right now, to overrule it. That number — not your stated distrust — is the actual safety mechanism. Usually it is set far too high.