When the Mirror Gives You a Number

Ask a chatbot to rate a face and the answer can look like a measurement rather than an opinion. The machine has no awkward friendship to preserve, no dinner invitation riding on its tact. Surely that makes it impartial. Except an absence of social obligations is not the same thing as a calibrated scale.

A preliminary study reported by Eric W. Dolan in PsyPost on September 24 offers a useful interruption. Four commercial AI models tracked human rankings of facial attractiveness while giving faces higher scores. They could broadly agree about the order without agreeing about the numbers.

That distinction matters if you have asked a machine for an honest assessment and received something that looks reassuringly precise.

Same order, different scale

The arXiv preprint by Santiago Grandas and colleagues compares ratings from 2,513 human participants with outputs from Claude, Gemini, GPT and Grok. Its abstract reports strong correlations with human judgments, but more favorable scores within a narrower range. It describes the work as a preregistered exploratory study; PsyPost reports that it has not yet been peer-reviewed.

According to Dolan’s account, the researchers used 102 photographs from the Face Research Lab London Set and a seven-point attractiveness scale. Human ratings averaged 3.02 across the faces; the combined AI average was 4.71. These are results for the tested systems and setup, not permanent properties of four product names.

The finding is not simply that the models failed to recognize what the human sample preferred. Their relative rankings tracked those preferences. But a model can put faces in a similar order while placing them all farther up the scale. Agreement about who comes first does not make a particular score interchangeable with a human rating.

Calling that difference flattery is tempting. Explaining it is harder. In PsyPost, co-author Macken Murphy raises more favorable reporting associated with sycophancy as one possible explanation. The comparison does not establish that agreeableness caused the gap, or reveal an internal, more candid judgment hiding behind the output. A higher score is observable. The motive we assign it is not.

The crowd is not a beauty instrument either

There is another trap waiting inside the correction: treating the human average as beauty’s actual address.

Dolan reports that the human raters were predominantly young, female and attracted to men, while the photographed faces were heavily skewed toward White people. The original survey’s exact wording was unavailable, complicating prompt matching. AI ratings were also averaged across repeated runs, whereas each human participant supplied single responses—a design difference that can make the models appear more consistent than individual people.

Those limits do not erase the reported score gap. They specify what it is a gap from: this sample’s judgments under these conditions, not an objective quantity stored somewhere in a cheekbone.

For a tool intended to reproduce a population’s judgments, calibration matters. So does naming the population. A benchmark becomes less honest, not more scientific, when the people supplying it disappear behind the word “human.”

The coherence test here is whether the answer remains connected to what produced it. An evaluator should make its reference group, procedure and uncertainty inspectable. A supportive conversational response should not borrow the authority of a measurement it has not earned. Warmth and accuracy need not be enemies; confusing their jobs helps neither.

For the person asking, the invitation is not to find a harsher machine. It is to notice what you want the score to settle. Curiosity about how others might respond is understandable. But neither a generous chatbot nor a crowd average can turn that response into a verdict on your worth.

The more honest mirror may be the one that explains its limits before handing you a number.