Last week, Anthropic engineer Jacob Coxon sent the internet into a firestorm after publicly resigning on X due to his belief that AI companies are "racing straight to self-improving superintelligence and gambling with our lives." This was followed by a post from Evan Hubinger, Alignment Science Lead at Anthropic, agreeing with his former coworker, finding that there's somewhere around a 10% chance of AI killing us all in the next decade. Then, over the weekend, Anthropic CEO Dario Amodei released an essay calling for new safeguards on AI development, including "embedded third-party evaluators" and pacing the rate of development in both the US and China (while acknowledging the difficulty in accomplishing the latter). Fellow AI CEOs Sam Altman (OpenAI) and Elon Musk (SpaceXAI) quickly voiced support for the proposal. But President Trump immediately shot down the idea due to the risk that it might forfeit the American AI lead to China.
So, what does this all mean? Are the doomsayers right that AI could kill us all? What's driving AI executives to make statements that their technology might be as dangerous as nuclear weapons? And is there any hope for regulations to make AI development safer? (The answers: Not really, but it's sure to do some real damage; some legitimate concern but also financial self-interest; and not much, which is a problem.)
On apocalypse-watch, you should continue saving for retirement—despite what some AI doomsayers might tell you. First, the argument for how AI is going to destroy the world is. . .vague. Despite OpenAI attempting to claim that it has reached artificial general intelligence (AGI), the reality is that we do not yet have AI systems that are better than most humans at most tasks. And no one entirely agrees on the criteria for achieving AGI, including Sam Altman who recently derided it as an "irrelevant marketing term." So, on the notion that a godlike computer stands poised to take us out…color me skeptical.
That's part of why when Coxon was challenged to get specific about how AI will end humanity, the answer wasn't very clear: "the classic example is, like, synthesizing a new virus or taking down critical infrastructure by doing some sort of hacking spree. . ." The p(doom) crowd needs a new parenthetical. We should be worried about the dangers of AI without guardrails. Just in recent weeks, Anthropic has disclosed that Iran used its AI to attack American ships, China used it to surveil its persecuted Uyghur minority, and military researchers have tried to use it to create bioweapons. And that's just the tip of the iceberg for our vulnerabilities: we're at the precipice of a potential "cyberpocalypse," as AI makes it easier than ever for hackers to exploit weaknesses in aging code for websites across the web.
But these are crisis risks—events that could cause panic, even loss of life, but not the end of civilization. What we're talking about is bad actors using AI for bad purposes or a smaller version of the "paperclip problem," in which an LLM goes rogue and causes unexpected mischief. Not Skynet launching the nukes. This should prompt anxiety and regulation—not bunkers.
Second, it is in the interest of AI companies to hype their new models as all-powerful. Anthropic and OpenAI are moving towards initial public offerings (IPOs) while burning through massive amounts of cash (much of it borrowed) to fund the data center buildout that next generation LLMs require. Meanwhile, backlash against data centers is growing across the country. It's good business for AI companies to: 1) build excitement for their products ahead of a potential IPO frenzy and 2) encourage slowing the pace of development as they face significant bottlenecks when it comes to the financing and construction of data centers.
The historical track record of tech panics is not encouraging. Angry mobs of scribes destroyed early printing presses, incensed by fears of job loss. Newspapers once reported on the dangers of radio broadcasts on children because it was perilous for them to "be kept up at the jazz rate forever." Television was supposed to rot our brains. (And that's still a legitimate risk for every TV show with the unique exception of "GZERO World," available on your local PBS affiliate in the US and on YouTube.) With nearly every great innovation comes a great panic. . . And much like its technological ancestors, AI does offer reason for worry. This will likely be the most transformational technology of our era. It will cause unforeseeable problems. It should not cause panic.
As a consequence, the rhetoric of AI leaders is exaggerated—in some cases subconsciously and in others strategically—even if the underlying risks are real. That's why it's essential that we develop a cohesive international system to govern AI, something akin to a global central bank: an independent organization designed to combat the systemic risks of AI, as central banks do for the financial system. But there are two obstacles: Trump and China.
In a recent Truth Social post on the subject, Trump claimed that all "AI needs is a STRONG AND SMART (High IQ!) President," and decried the "SICK conspiracy going on against AI and Data Centers." He went on to claim that proposals by "Conspiracy Theorists, Treasonists, Traitors, and Leakers" to regulate AI favor China in a dynamic where, "WHOEVER WINS AI, WINS!" Excessive use of capital letters aside, this is an accurate encapsulation of the Trump administration's approach to AI development: relentlessly move forward, without any serious (and potentially slowing) regulatory burden, to beat China. In addition, the administration's Pax Silica framework seeks to ensure that the US has strategic autonomy over the AI supply chain—even if that means fracturing any potential international collaboration on AI.
Meanwhile, China has made US-critical statements ("Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance. . .") while simultaneously pressing forward on its open-source models, which might pose even greater risks given the lack of a "kill-switch" in their operation. To the extent that China is concerned about AI, their fears are not framed around doomsday scenarios but rather around the ways in which the technology might undermine the Communist Party's grip on the country and be used by foreign foes for espionage, as head of the Ministry of State Security Chen Yixin recently wrote in an essay. In a world where the American and Chinese governments are focused on winning AI supremacy, rather than protecting against disaster outcomes, there is not much room for a bilateral deal on a research slowdown, even with Xi and Trump set to discuss AI in their upcoming summit.
So, we are staring down the barrel of a dangerous future due to unregulated AI development: one that is sure to include crises created by rogue AIs and rogue actors using AI. When one of those crises strikes, hopefully it will be competently managed and serve as a catalyst for urgently needed regulations and negotiations between the US and China. In the meantime, the p(doom) crowd would be better served warning of the many real threats AI poses, from bioweapons to cybersecurity, rather than claiming that the apocalypse is around the corner.
