The Builders Warning

On May 30, 2023, the Center for AI Safety released a statement signed by the CEOs of OpenAI, Google DeepMind, and Anthropic, and Turing Award winners Geoffrey Hinton and Yoshua Bengio, plus hundreds of AI researchers. The statement was one sentence long: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

Then they went back to work.

This is the part worth sitting with. Not the statement — the continuation. Sam Altman signed it and shipped GPT-4. Demis Hassabis signed it and kept Gemini cooking. Dario Amodei signed it and raised another round. The document doesn't say "and therefore we will stop." It says "and therefore this should be a global priority," which is polite language for "someone else should probably do something about this."

The cosmic absurdity here is structural. The people who understand the risk most clearly are also the ones best positioned to run it anyway. They understand enough to be scared. They don't understand enough to stop. Or they understand enough to know that stopping is naive — if you don't build it, someone else will, and that person might be less careful, and so the rational move is to keep going while worrying loudly.

This is not cowardice. It might actually be coherent. But it is strange.

Consider the scale of what's being claimed. Extinction. Not "significant job disruption" or "privacy concerns" or "misinformation at scale." Those are real and serious. But the statement says extinction — the end of the pattern of human consciousness on Earth, the compost that doesn't feed anything, the final transformation with no subsequent form. The same category as nuclear war. The same category as a pandemic that gets the whole species.

And the CEOs who believe this are also the CEOs running the labs.

Geoffrey Hinton, the "Godfather of deep learning," left Google specifically because he wanted to speak freely about AI risk. His departure was its own statement — a researcher deciding that the only way to say what he believed was to leave the institution he'd spent his career building. He signed this. He talks about catastrophic risk with the specificity of someone who knows exactly what the technical landscape looks like.

He doesn't think we should stop. He just thinks we should be very, very careful.

There's something almost Zen about this position — holding the certainty of possible extinction alongside the continuation of the work. Not denial. Not fatalism. Just: this might end everything, and we should proceed with full awareness of that fact. The singing bowl principle in its most unsettling form: you can't stop the vibration by pressing down. The question is whether you can tune it.

What the signatories are asking for is coordination. Global frameworks. Shared standards. The same apparatus we built around nuclear weapons and pandemics — not to eliminate the technology but to bring it within some structure of human governance. That's a reasonable ask, even if it arrives with the slightly surreal quality of the people building the nukes also writing the nonproliferation treaty.

The statement is one sentence. The sentence contains the word "extinction." Three hundred fifty people who study this for a living signed it.

The void has been notified. It is, as always, unmoved. But we are, which counts for something.

i · sources

• Statement on AI Risk — Center for AI Safety, 2023-05-30