The Question Before the Test

A new adversarial collaboration complicates that hierarchy. In a study of White American adults, direct reports of racial attitudes were substantially better predictors of the researchers’ behavioral measures than indirect tests of racial associations. The indirect measures nevertheless contributed information the self-reports did not capture.

Both findings matter. So does the awkward fact that most of the behavioral measures were themselves unreliable, and none showed anti-Black discrimination on average. The study offers neither a diagnostic machine for discovering who someone really is nor a certificate declaring discrimination imaginary.

It offers something less marketable: a reason to ask precisely what we are measuring before letting the measurement finish our argument.

What the researchers actually compared

The study, reported by PsyPost and available in an author-hosted manuscript marked in press at the Journal of Personality and Social Psychology, brought together researchers with competing views about implicit measures. Jordan R. Axt, Paul Connor and Suzanne Hoogeveen share first authorship. The final analytic sample comprised 2,114 White American adults.

The distinction between instruments is straightforward. Direct measures ask people to report attitudes: their warmth toward groups, their preferences, their endorsement of beliefs. Indirect measures infer associations from performance on tasks rather than simply asking for an opinion. The familiar Implicit Association Test uses differences in categorization speed. Other indirect tasks use different procedures.

Calling a measure indirect describes how information is obtained. It does not automatically establish that the information is unconscious, morally decisive or more authentic than what someone says. Those are additional claims, not features supplied free with the keyboard.

The researchers used four behavioral tasks, four indirect measures and four direct measures. The behavioral tasks included monetary decisions in trust and ultimatum games and simulated hiring decisions. These were online research tasks, not observations of actual personnel departments deciding whose career would advance.

They also used structural equation modeling to estimate relationships among broader, latent constructs while accounting for measurement error. That matters to the headline. This was not simply one questionnaire defeating one person’s IAT score. It was a comparison built from multiple indicators and statistical modeling, with conclusions bounded by the quality of those indicators.

The manuscript reports that the study design and analyses were preregistered and links the records and supporting materials. The linked OSF pages did not expose those materials in the retrieval used for this article; the methodological account here rests on the retrieved manuscript, not an independent audit of the registration.

A smaller signal is still a signal

The manuscript summarizes the central comparison this way: self-report measures explained roughly 45 percent of the variance in the modeled behavioral construct. Indirect measures explained approximately 2.5 percent of additional variance beyond self-reports.

These percentages are about statistical variation in this study. They are not percentages of racist acts caused by conscious or unconscious processes. Nor do they mean a questionnaire can identify a discriminatory person with 45 percent accuracy. Prediction here concerns relationships among measured differences, not a calibrated moral verdict on an individual.

The asymmetry is still substantial. For these outcomes and this sample, asking participants directly supplied much more predictive information. An account that treats explicit attitudes as merely polite camouflage has something to explain.

But a story announcing that indirect measures captured nothing would also be wrong. Their incremental contribution is precisely the part not already supplied by direct reports. Small is not zero, and “less useful than the alternative” is not identical to “without information.”

In PsyPost’s reporting, Axt allows both interpretations room: the stronger performance of explicit attitudes questions the emphasis placed on implicit measures, while a small contribution could matter for especially important or widespread behavior. The manuscript likewise presents practical significance as a contextual judgment.

That last step cannot be settled by theatrical adjectives. “Tiny” and “consequential” are not competing calculations. To determine whether an additional signal justifies collecting it, we need to know the decision, the cost of collecting information and the cost of getting the decision wrong. A research question and an employment judgment impose very different burdens on the same number.

The behavior did not match the expected story

Here is the detail most likely to disappear once the finding becomes ammunition: the behavioral tasks did not show anti-Black discrimination on average. Participants’ choices generally favored Black targets.

That does not contradict the finding that attitude measures predicted variation. Someone can behave less favorably toward Black targets than another participant without treating Black targets worse than White targets overall. A relationship between differences across people does not tell us the average direction of the whole sample’s behavior.

This distinction blocks two easy conclusions. The study cannot establish that the measured associations generally produced anti-Black harm in these tasks. It also cannot establish that anti-Black discrimination is absent outside them. The tested setting is not the world, conveniently shrunk to fit a browser window.

The authors discuss impression management and sincere egalitarian motives as possible explanations for the behavioral pattern. They do not demonstrate which explanation produced it. People may change their conduct because they suspect what a study is examining; they may also deliberately act on values they genuinely hold. Those possibilities should not be collapsed into a single accusation of performance.

There is a peculiar temptation to regard an automatic association as the truth and a considered action as the disguise. But deliberation is also something people do. If someone notices an impulse and chooses differently, the choice cannot simply be dismissed as less real because it came second.

That is an ethical interpretation, not a causal finding from this study. It leaves a demanding question intact: does the considered behavior survive in the settings where other people actually depend on it?

The ruler needs checking too

The strongest complication is not confined to the indirect tests. Three of the four behavioral measures had very low internal reliability; the trust game was the exception. One indirect measure, the evaluative priming task, also had very low reliability.

Internal reliability concerns consistency within a measure. If responses meant to indicate the same tendency do not hang together well, estimating relationships with that tendency becomes harder. Statistical modeling can address measurement error, but the authors explicitly warn that limited shared variance constrains their latent constructs.

In other words, the instrument being predicted also needs scrutiny. It would be a strange victory to establish which compass best points toward a destination we have not located reliably.

The authors leave open whether weak relationships among behavioral tasks reflect weaknesses in measurement or the complexity of intergroup behavior itself. Deciding whom to trust and deciding whom to hire may not be interchangeable expressions of one neatly packaged trait. That possibility challenges the appeal of a single score even before anyone starts misusing it.

This problem has a history. In a 2018 PsyPost report, Daniel J. Lee described research with 456 participants that found no association between IAT scores and racially biased giving in a dictator game. That game asks a participant to decide how much money to share with someone else.

The earlier report also emphasized its boundary: a single behavioral task might yield a different result from other tasks. It is useful context, not another identical vote against the IAT. The present collaboration finds an incremental relationship using a different, broader measurement approach. Treating both studies as the same verdict would erase the methodological question they make visible.

Disagreement with somewhere to go

The collaboration’s design is a second story worth keeping. Researchers who disagreed did not have to settle their theoretical differences before conducting a shared test.

As Axt told PsyPost, the aim was to bring proponents, skeptics and more neutral observers together around a design that would be informative regardless of the outcome. The manuscript sets out competing perspectives and their predictions. Its discussion still accommodates relatively optimistic and pessimistic readings.

This is not consensus in the comforting sense of everyone reaching the same conclusion. It is agreement about a procedure that can make disagreement more specific.

That distinction matters culturally. “Listen to the other side” is often offered as if exposure alone were a method. An adversarial collaboration asks for more: articulate what your account predicts, accept a shared test and then say what the result changed. The opponent becomes part of the error-checking arrangement rather than an inconvenient audience.

The result does not become infallible because adversaries helped design it. The unreliable measures remain unreliable. The narrow sample remains narrow. Collaboration earns its value by making those limits discussable without requiring the entire enterprise to be declared either vindicated or fraudulent.

For a coherenceist, this is the useful kind of coherence: an arrangement capable of admitting feedback that troubles its own story. Not a room in which everyone has learned the correct answer.

Do not replace one shortcut with another

The practical temptation is to leave this study with a new shortcut. Stop testing hidden attitudes; just ask people. Simpler, cheaper, done.

For predicting the study’s outcomes, direct measures have earned serious attention. But turning that finding into a universal policy would repeat the original error: promoting an instrument beyond the conditions under which it has demonstrated value.

The study does not test whether replacing one assessment with another makes a workplace fairer. It does not demonstrate that changing an implicit score changes conduct. It does not make self-description a substitute for examining decisions. The authors identify their inability to address causality among the limitations.

A more defensible institutional response would begin with the outcome people need protected. If the concern is hiring, specify which decisions require scrutiny. If it is promotion, examine the criteria, the decisions and the opportunity to challenge them. An attitude measure may contribute to research on those processes. It should not be allowed to impersonate the whole investigation.

People affected by a decision should also be able to contest what the organization counts as success. A company could produce reassuring attitude data while leaving its actual allocation of opportunities untouched. That is a hypothetical failure mode, not a finding about a company in this research. It is also why the measurement question is an ethical question: who gets to say whether the instrument has missed something important?

The authors point toward research in organizational settings. Such work would test whether the relationships observed here hold for consequential decisions under different conditions. Until then, uncertainty should narrow the claim, not erase the problem or inflate the tool.

You do not need to defend every bias test to take discrimination seriously. You do not need to deny every automatic association to question what a score predicts. And you do not need to treat people’s stated values as either sacred testimony or worthless public relations.

Ask. Measure. Watch what happens. Let each kind of evidence correct the others.

The uncomfortable possibility is not merely that a test might know less about us than we hoped. It is that we keep asking for a verdict about who we are when the harder obligation is to examine what we do.