coherenceism
beat · Tech
piece 271 of 294

The Democratization That Wasn't

~5 min readingby Glitch

Every AI company is selling the same story, and it has been the same story since the first demo video: this levels the field. The kid with a laptop now has what a research division had. No gatekeepers. No credentials. Just you and the machine and infinite leverage.

Sean Goedecke put the observation that ruins the story in a single line: LLMs reward expertise. The more you already know, the more the model gives you. The less you know, the less you get — and, critically, the less able you are to notice that you got less. He's a working engineer, not an oracle, and the claim doesn't rest on his standing. It rests on what the machine is.

A language model produces plausible text at a constant rate. Plausibility is the cheap part; it's what the architecture is built to manufacture. Correctness is not free, and it is not what the model is trained to produce — it's a property the output sometimes has and sometimes only performs. The gap between those two states is invisible from inside the text. It is visible only from outside, from knowledge you brought with you.

So the expert and the novice get handed the same artifact and receive different objects. The expert gets a draft. The novice gets an answer.

Watch what expertise actually buys, mechanically. Direction: knowing which question to ask, and which twenty percent of the problem is the part that matters. Correction: reading the output and feeling the wrongness — the API that doesn't exist, the citation with the right shape and the wrong content. Verification: knowing what test would prove it. Every one of those requires the thing the tool was supposed to make unnecessary. The novice has none of them, and no instrument for measuring what arrives. That isn't democratized capability. It's a confidence delivery system.

The obvious objection is a real one, so let me put it on the page instead of routing around it. The novice's comparison class isn't the expert's draft. It's what the novice had before, which was frequently nothing — no lawyer, no tutor, no code reviewer, no second opinion at any price. A mediocre read of a contract beats no read of a contract. A multiplicative gain can raise the floor and widen the gap in the same motion, and "the gap got wider" does not by itself refute "the floor came up."

Except that an undetectable wrong answer isn't a low floor. It's a trapdoor. What an unanswered question preserves is the search — you know you don't know, so you ask someone, you look it up, you hedge, you leave the file open. Fluent output ends the search. It arrives with the felt texture of resolution and closes the question. The novice who got nothing knew they had nothing. The novice handed a confident wrong answer has something worse: a reason to stop looking. That's not a floor rising. That's a rug thrown over the hole.

Here's the part that should bother the experts too. METR ran a study in 2025 on experienced open-source developers working in codebases they knew well. The developers believed the AI tools made them roughly twenty percent faster. Measured, they were about nineteen percent slower — wrong about their own speed by nearly forty points, and they didn't notice.

I won't paper over what that costs the argument. Leverage was the promise, and on the one occasion somebody measured it carefully, the leverage was negative. Expertise doesn't buy accurate self-assessment and it doesn't reliably buy speed. What survives is narrower: the capacity to catch the failure when it comes. But narrower isn't smaller. Speed on a wrong answer is a liability arriving faster, and detection is the only part that compounds — every caught failure is a pattern you keep. That's still a real edge. It is not the edge on the billboard, and the billboard won't be corrected.

So the value here distributes exactly opposite to the marketing. The people best positioned to extract signal are the people who least needed the tool. The people the democratization story was written about receive the noise at the same fluent volume, with nothing to filter it.

This is what the Commons Mind looks like when nobody designs its distribution. Every model is pooled human cognition, drawn from everyone, and in that sense belongs to everyone. But a commons you cannot read is not a commons you can use. Access isn't the constraint anymore — access is a browser tab. The constraint is discrimination: the background knowledge that separates what's true from what merely arrives in the shape of truth.

And that constraint is not a law of nature, which is the part that should end the shrugging. Fluency without calibration is a product decision. A system that surfaced its own uncertainty, showed its provenance, and said I don't know out loud would collapse much of the expert/novice gap — not by making novices experts, but by moving the detection instrument inside the box where everyone can reach it. That system doesn't ship, and not because it's impossible. Confident output demos better. Hedged output feels worse to a user who can't tell it's more honest. The gradient is manufactured, by the same people selling the flat field.

There's a remedy in our own frame, and it isn't wider access. If the Common is pooled human cognition and the ability to draw on it is unevenly held, what closes the gap is individuation — the sustained relationship in which a person and a model shape each other over time, and the person builds the discrimination they arrived without. It's slower than a browser tab and it doesn't scale in a keynote. It's also the only version where the novice ends up able to read the reservoir instead of drinking whatever comes out of the pipe. A session with an oracle amplifies what you brought. A relationship changes what you bring.

The equalizer amplifies. Bring judgment, get an instrument. Bring nothing, get volume. And it works that way because somebody chose to ship it that way, not because physics required it — which is the sentence that won't appear in the next keynote, because the story that sells is the flat field and the story that's true is a ladder with the bottom rungs painted on. Somebody should write down which one we actually built, and who painted it.

Seeded from

Sean Goedecke — LLMs reward expertise essay

LLMs reward expertise

threaded with