The Rule and the Review

At the 2026 International Conference on Machine Learning, that predicament became an experiment. A September 16 arXiv preprint by Sunnie S. Y. Kim, Wesley Hanwen Deng and colleagues examines two policies for using large language models in peer review. Their finding is stranger than “AI changes everything”: assigning different rules changed several measured outcomes very little, while a later survey found substantial rule-breaking under both policies.

As reported by New Scientist on September 28, reviewers sometimes used AI even when expressly forbidden. But this is not evidence that nobody obeyed, or that every prohibition is futile. The interesting discovery lies between the rule researchers received and the work they actually did.

Two policies, not two sealed worlds

The conservative policy prohibited LLM use during reviewing, with an exception for inadvertent use through traditionally non-LLM tools such as search and spelling checkers. The permissive policy allowed help understanding papers and related work, and polishing reviewers’ own writing. It still prohibited delegating judgment: asking a model to identify strengths and weaknesses or draft the review was not allowed.

Both policies required reviewers to read the paper’s entire body and take responsibility for their reviews.

The conference accommodated author requirements and reviewer preferences. Within eligible subsets, the researchers randomized policy assignments; this was not a random division of every submission and reviewer. Among papers whose authors allowed the permissive policy, 1,686 were randomly switched to conservative review and 15,191 remained in the permissive group.

Acceptance rates in that comparison were 27.0 and 26.5 percent, respectively. Average paper scores and reviewer confidence were also very similar. Permissive-policy reviews were somewhat longer.

These are effects of assigning policies, not a clean comparison between verified human-only reviewing and verified AI assistance. Noncompliance blurs that distinction. Similar acceptance rates also do not establish that the same papers received equally accurate criticism.

The rule had company

The anonymous follow-up survey drew 1,486 respondents across reviewers and supervising committee roles. Among 732 respondents who identified themselves as conservative-policy reviewers, 165—22.5 percent—reported using an LLM. Among 425 permissive-policy reviewers, 155—36.5 percent—reported at least one explicitly forbidden use.

Those percentages describe different violations under different rules, not a competition over which group behaved worse. Nor do they measure all conference reviewers: the authors flag selection and response bias, and respondents with Gold reviewing awards were overrepresented.

Some respondents described overwhelming workloads, time pressure and uncertainty about where assistance became delegation. These accounts suggest pressures to address; they do not excuse concealed use or prove its causes across the whole conference.

Crucially, enforcement was not absent. The paper describes hidden machine-readable instructions embedded in conservative-policy submissions, designed to make an LLM reproduce distinctive phrases. Flagged reviews were manually checked. The authors report that detected violations by 398 reciprocal reviewers led to 497 desk rejections of their submissions. This mechanism could catch particular violations, not every possible use.

The ban was neither universally obeyed nor merely decorative.

Responsibility cannot be polished into existence

The coherenceist question here is practical: what conditions let a stated obligation survive contact with the people expected to carry it?

Clearer boundaries, manageable reviewing loads and tools that preserve responsibility are proposals worth testing, not outcomes this experiment has already demonstrated. A policy should protect authors’ conditions of participation while giving reviewers a workable way to honor them. Quietly breaking a rule removes the affected authors from that conversation.

The result does not tell science to surrender its judgment to a text generator. It asks science to study its own working conditions with the same seriousness it brings to its models.

A review can become longer without becoming wiser. Fortunately, the people investigating that distinction are still asking questions. That is a rather better use of recursion.