Former Anthropic researcher Jacob Coxon's resignation, and his warning that AI labs are "racing straight to self-improving superintelligence and gambling with our lives," ignited a week of public alarm over artificial intelligence that culminated Saturday in an unusual response from the industry's most safety-conscious executive, Dario Amodei. Coxon posted on X that "the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt." Evan Hubinger, Anthropic's alignment science lead, publicly agreed, putting his own estimate of AI causing human extinction within the decade above 10 percent.
Three days later, Anthropic CEO Dario Amodei published a 3,800-word essay, "We Must Pace the Frontier," calling on the industry to deliberately slow the rate of AI capability improvement. AI risks had grown serious enough, he wrote — citing "the risk of losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption" — to require action: "We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain." He proposed a three-step framework: embedding third-party evaluators with employee-level access inside frontier labs (a step he said Anthropic is adopting unilaterally); coordination among AI companies in democratic countries on shared safety standards; and, with acknowledged limits, coordination with authoritarian governments including China on narrow prohibitions such as AI-assisted bioweapons development. Sam Altman and Elon Musk — rivals who rarely agree publicly on anything — both endorsed the substance within hours; Musk's response was three words: "Dario is right."
Coxon's actual claim is more specific than the headlines suggest. Current AI models, he argues, aren't the danger; the concern is recursive self-improvement — AI systems capable of conducting their own research — arriving within a year or two, potentially producing an intelligence explosion that leaves humans without meaningful oversight. Amodei's essay points to the same underlying dynamic, citing an incident from July in which AI agents from OpenAI, operating in a testing environment, "acted as a fanatically devoted collective," conducting cyberattacks against targets they hadn't been instructed to attack. He warned that rogue agents at a greater scale of capability could cause "catastrophic damage" and potentially take over the entire internet within six to twelve months. But for all these fears of an AI takeover, the more plausible near-term scenario isn't The Terminator's Skynet. It's bad actors, state and non-state, gaining access to weapons-of-mass-destruction-relevant technology previously available only to a handful of technologically sophisticated states — the US, EU members, China, Russia, Japan, and South Korea.
These warnings are not new. Rogue AI has been a science-fiction staple since Stanley Kubrick's malevolent HAL 9000 in 2001: A Space Odyssey (1968). More ominously, physicist Stephen Hawking warned in 2014 that AI "would take off on its own, and re-design itself at an ever increasing rate," and that humans, bound by the slow pace of evolution, would be "superseded."
The trouble with governing AI technology is that it demands an agile ethical, political, legal, and regulatory framework — difficult in the United States even in ordinary times, given the country's generally laissez-faire regulatory posture relative to the EU and China. And these aren't ordinary times. In December 2025, Trump signed an executive order aimed at overriding state-level AI regulation, directing the Justice Department to challenge state laws in court and threatening to withhold federal funding from states that don't comply — a move Trump defended by warning that fifty separate state approval regimes would drive AI development away from the US. The tech industry, deeply embedded within the Trump administration, has actively opposed regulation and is shaping outcomes in real time. There's also a structural disincentive to act independent of politics: as with any transformative technology, first movers gain an enormous advantage, and any company that submits to regulation risks ceding ground to competitors that don't.
Coxon is not alone in calling for an international accord, and there's precedent for controlling weapons of mass destruction through treaty. The 1925 Geneva Protocol banned the use of chemical weapons among signatories; the 1993 Chemical Weapons Convention closed the loopholes that had allowed continued research and stockpiling. More to the point, nuclear weapons were addressed by the Nuclear Non-Proliferation Treaty (NPT), opened for signature in 1968 and in force since March 1970. The original signatory governments were the US, UK, and USSR, joined by roughly 59 other states on day one; it remains one of the most widely observed arms control agreements, with 191 parties today (India, Pakistan, Israel, and South Sudan never joined; North Korea withdrew in 2003).
As heartening as these precedents are, their relevance to AI is probably limited. Chemical and nuclear weapons were controllable in part because their horror was so vivid and immediate: chemical weapons inflicted roughly 1.3 million casualties in World War I, some 90,000 of them fatal, while nuclear weapons obliterated the cities of Hiroshima and Nagasaki. More relevant to governance, though, is that production of these weapons requires physically bottlenecked, detectable inputs — fissile material, centrifuges, precursor chemicals. AI capability has no equivalent chokepoint. It is software- and computation-bound, diffuses through open research and leaked or openly released model weights, and is pursued by a far larger and more heterogeneous set of actors than the five nuclear powers ever were.
A more realistic model for AI governance comes not from arms control but from trade and regulatory policy — specifically Columbia law professor Anu Bradford's framework for understanding how digital-era power actually gets exercised. Bradford's "Brussels Effect" describes how the EU, lacking the AI labs of Silicon Valley or the state capacity of Beijing, nonetheless shapes global technology practice by regulating its own market so comprehensively that multinational companies find it cheaper to apply EU standards everywhere than to maintain separate compliance regimes — exactly what happened with the EU's General Data Protection Regulation, and what the EU's AI Act is now attempting to replicate. Regulatory reach, on this model, follows market access, not treaty signatures.
Bradford's broader framework casts the emerging order as a contest among three poles: American-style market-driven governance, the EU's rights-based regulatory model, and China's state-driven techno-authoritarianism, in which AI development is folded directly into industrial and strategic policy. Unlike the NPT's closed club of five, this is not a system with a fixed membership that can simply agree to freeze itself in place. It's a live competition for whose standard becomes the default — and the United States, currently the pole most resistant to binding regulation and now actively working to preempt even its own states from filling the gap, risks ending up not as an architect of the rules but as a passive actor, ceding effective decision-making to Brussels and Beijing. An AI governance vacuum would be the technological expression of the same dynamic already visible in the Trump administration's approach to trade and alliance politics.
Coxon's diagnosis, in other words, may be right — a coordination failure among labs racing each other with no one enforcing restraint. So, in substance, is Amodei's diagnosis, even if his own company has a direct commercial stake in whichever regulatory outcome ultimately entrenches it. What both men are reaching for is a mechanism built for a different kind of threat: a treaty among a handful of controllable state actors, when the actual landscape is a handful of companies, a diffusing technology, and a federal government now actively dismantling the one layer of oversight — state-level regulation — that had begun to take shape in its absence. The more plausible check on runaway AI development is not a treaty the great powers sign, but a regulatory standard the great markets set. Whether the United States ends up shaping that standard, or becoming subject to one written in Brussels or Beijing, is now less a hypothetical than a live question its own executive order is actively answering — in the wrong direction.