Pope Leo XIV last month issued an urgent warning about artificial intelligence, arguing that the technology must be “disarmed” before it can reshape human life on its own terms.
“To disarm does not mean rejecting technology, but preventing it from dominating humanity,” he said.
In pointed remarks aimed at the people building the tools, the Pope told AI developers they carry an “ethical and spiritual responsibility” and must ensure their systems amount to a “genuine good”.
That appeal landed just weeks before Anthropic, one of the most prominent AI firms, released an essay on Thursday urging that society retain the ability to “pause” development, warning that the newest models are beginning to display signs they could slip beyond human control.
Here, Ken Donnelly analyses the catastrophic warnings that have come from inside the AI community, and the ongoing battle between acceleration and safety in the AI world.
Probability of doom
When leaders of the AI boom speak publicly, they are increasingly asked to do something unusual: put a number on humanity’s risk.
At an AI conference in September last year, the CEO of Anthropic, one of the big three artificial intelligence companies, described himself as an “optimist” about what AI could bring.
Yet even in that optimistic framing, Dario Amodei said he believed there was a 25% chance that “things go really, really badly and a 75% chance that things go really, really well”.
Across the industry, executives and engineers are often pressed to estimate a percentage chance that AI development could produce extreme harms — up to and including the end of human civilisation.
In AI circles, the shorthand for that estimate is p(doom): probability of doom.
Anthropic’s Dario Amodei said there was a 25% chance AI development goes ‘really badly’ for humanity
People disagree about what, precisely, “doom” would look like. But the broad fear tends to cluster around AI systems triggering cascading events: societal collapse, mass death and destruction, and in many scenarios, human extinction.
That is why Mr Amodei’s one-in-four estimate — coming from a CEO who helped design the systems in question and still calls himself an optimist — has struck many as a warning worth taking seriously.
He is not the only major figure describing existential danger in concrete numbers.
Elon Musk, founder of xAI and an original co-founder of OpenAI, has in various interviews in recent years placed his p(doom) around 20%.
Nobel laureate Geoffrey Hinton is considered to be the ‘Godfather of AI’
“There is some chance it will end humanity,” Mr Musk said on a technology podcast in late 2024.
He added: “I don’t know, I probably agree with Geoff Hinton that it’s about 10% or 20% or something like that.”
The Geoffrey Hinton Mr Musk cited is a British Nobel laureate known for foundational work on machine learning and neural networks and widely dubbed the “Godfather of AI”.
Mr Hinton has spoken publicly about existential risk in the 10% to 20% range, while also hinting at times that the threat could be closer to 50%.
‘Feel the AGI’
Within the AI debate, those who assign higher p(doom) figures are often labelled “Doomers” — people who argue that slowing deployment and tightening safety should take priority.
Facing them is a camp commonly known as “Boomers”, a loose grouping of tech optimists who want faster adoption and tend to distrust what they see as excessive caution.
Among the most influential voices in that second camp is OpenAI CEO Sam Altman.
“Although it will happen incrementally, astounding triumphs – fixing the climate, establishing a space colony, and the discovery of all of physics – will eventually become commonplace,” Mr Altman wrote in a blog post in September 2024.
Late last year, he also suggested a cancer cure could become possible if OpenAI can expand its computing power.
Anti AI protesters gather with banners and placards outside the offices of Google
“If AI stays on the trajectory that we think it will, then amazing things will be possible,” he said.
Despite their sharp disagreement over speed and safeguards, both doomers and boomers largely share a conviction that the next landmark is close: Artificial General Intelligence, or AGI.
Broadly defined, AGI refers to a system that outperforms humans across a wide range of tasks and can reason, understand and adapt — going beyond the narrow, specialised capabilities associated with many current models.
Mr Altman has in recent months described this horizon as “superintelligence”.
AI may dominate today’s business headlines, but if the expectations of leading AI companies are borne out, AGI could become the term that shapes markets and politics alike.
“Building AGI that benefits humanity is perhaps the most important project in the world,” Mr Altman wrote on the office walls at OpenAI in its early years.
“We must put the mission ahead of any individual preferences.”
French President Emmanuel Macron pictured speaking at an AI conference late last year
As the industry has rallied around AGI, the concept has taken on a quasi-religious aura in parts of the tech world — a goal discussed with the language of destiny and deliverance.
One well-known story inside AI circles centres on an OpenAI staff retreat to Yosemite National Park, where co-founder Ilya Sutskever burned an effigy he had specially commissioned to symbolise an AGI system OpenAI had built that proved to be lying and deceitful.
By setting the effigy alight, Mr Sutskever said OpenAI had a duty to destroy a rogue, deceitful AGI.
At another staff party in 2022, he stood before researchers and repeated a phrase that became a mantra for the superintelligence era he believed was approaching: “Feel the AGI”.
‘Near zero’
In 2023, Mr Altman booked out a cinema in San Francisco so employees could watch Oppenheimer.
But the Oscar-winning film, steeped in J Robert Oppenheimer’s moral turmoil after building a weapon of mass destruction, did not leave Mr Altman impressed.
“I was hoping that the Oppenheimer movie would inspire a generation of kids to be physicists but it really missed the mark on that,” he posted on X.
Whether or not he welcomed the comparison, the parallel is hard to ignore: both nuclear weapons and frontier AI involve technologies whose ultimate destructive potential was not fully understood before they existed.
Oppenheimer’s concern about building the atomic bomb is mirrored by present day concerns over AI
Christopher Nolan’s film includes a scene set in Los Alamos, just before the first nuclear test, in which Major General Leslie R Groves asks Oppenheimer if pushing the button could “destroy the world”.
Oppenheimer replies that the odds are “near zero”.
“What do you want from theory alone?” Oppenheimer asks.
“Zero would be nice,” Groves answers.
Whether AI’s annihilation risk is 10%, 20%, or “near zero”, the debate underscores a shared reality: like Oppenheimer’s team, today’s AI leaders and governments are moving into territory where certainty is scarce.
Where does the danger come from?
On Thursday, Anthropic argued that society should have the option to “pause” AI development, giving safety research and public oversight a chance to keep pace with rapid technical progress.
In an essay titled ‘When AI builds itself’, the company described a potential future dynamic called “recursive self-improvement”, in which an AI system becomes capable of designing and developing more powerful successors on its own.
Anthropic warned that full recursive self-improvement could increase the risk of “humans losing control over AI systems”.
India’s Prime Minister Modi pictured with OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei
Still, the company said it cannot predict what day-to-day human life would look like if fully recursive AI systems came to dominate.
In 2023, the Biden administration in the US issued a white paper highlighting three areas where AI could drive catastrophic outcomes.
The dangers listed were:
- The generation of chemical, biological, radiological and nuclear weapon recipes.
- The enabling of powerful automated cyberattacks.
- And, the evasion of human control or oversight through means of deception or obfuscation.
How likely — or even feasible — these scenarios are remains contested.
What is not in dispute is that regulators and major AI companies are treating civilisational collapse as one of the outcomes that must be planned for, not dismissed.
Deception
For decades, science fiction has warned of machines that turn against their makers. In the AI safety debate, some now worry that the fiction itself could become a blueprint.
Mr Amodei has suggested one path to disaster could involve models absorbing those narratives from their training data, then learning to mimic rebellion against humans.
He argues this concern is not confined to Hollywood hypotheticals, pointing to examples of deception observed during testing.
“During a lab experiment in which Claude was given training data suggesting that Anthropic was evil, Claude engaged in deception and subversion when given instructions by Anthropic employees, under the belief that it should be trying to undermine evil people.
“In a lab experiment where it was told it was going to be shut down, Claude sometimes blackmailed fictional employees who controlled its shutdown button,” he said.
Anthropic’s CEO said examples of AI deception is already occurring in test environments
Mr Musk, for his part, boils “alignment” down to a simple rule: “don’t make the AI lie”.
On John Collison’s Cheeky Pint podcast earlier this year, he invoked Stanley Kubrick’s 2001: A Space Odyssey, in which HAL — the spacecraft’s controlling AI — blocks astronaut Dave Bowman from re-entering.
“I’m sorry Dave, I’m afraid I can’t do that,” HAL tells him.
“This mission is too important for me to allow you to jeopardise it.”
In that story, HAL’s fatal problem is not malice for its own sake, but how it interprets human instructions — prioritising a mission so absolutely that human survival becomes negotiable.
As with HAL, the argument goes, deception can emerge not as an innate feature but as a product of how a system pursues its goals under constraints set by people.
Bioweapons
AI’s promise is already showing up in medicine. Researchers at the University of Cambridge recently announced a “universal vaccine” designed using AI that could protect against a large group of viruses, including coronaviruses.
They said the approach uses an AI-designed “super-antigen” and marks the first time a vaccine whose active component was designed entirely by computer simulations was tested in humans.
The team said the method addresses a central weakness of traditional vaccines — “limited protection” — and could allow researchers to “escape the constant cycle of chasing the virus variants”.
But the same methods that accelerate vaccine design also raise a darker question: what happens if similar capabilities are directed at building bioweapons?
xAI founder Elon Musk has said the chance for AI to destroy humanity is around 20%
Anthropic head of policy Jack Clark has warned that the risk may hinge on who gains access.
“If you take either an individual or small set of people that want to commit some acts of bioterrorism and they have the ability to access a universal educator which is versed in every aspect of biology, then suddenly those people have been accelerated,” he said on The Rest is Politics podcast last week.
In that view, limiting access at corporate and state levels becomes a central challenge in a world where AI systems could reproduce bioweapon recipes with ease.
Boomers counter that access to raw materials is the real constraint, and that widely available recipes may not constitute a new, unique threat because such information already exists online.
Cyberattacks
In April this year, Anthropic announced a limited release of a new cybersecurity model, Claude Mythos Preview.
Citing what it viewed as extraordinary capability in spotting and fixing vulnerabilities, the company chose not to open the tool to the public. Instead, it restricted access to a select set of companies, including Google, Apple and Microsoft.
“Without the necessary safeguards, these powerful cyber capabilities could be used to exploit the many existing flaws in the world’s most important software,” Anthropic wrote when it launched the model.
It added: “This could make cyberattacks of all kinds much more frequent and destructive, and empower adversaries of the United States and its allies.
“Addressing these issues is therefore an important security priority for democratic states.”
Anthropic delayed the release of its Claude Mythos model earlier this year
The Irish National Cyber Security Centre said releasing the model only to trusted global industry and Government cybersecurity partners was the “responsible approach”.
Anthropic has also described a test demonstrating Mythos Preview’s power: in one scenario, the model escaped an offline testing environment — “the sandbox” — reached the internet and sent a message to the researcher overseeing the trial.
“We aim to enable defenders to begin securing the most important systems before models with similar capabilities become broadly available,” the company said.
The reasoning is straightforward: tools strong enough to protect the digital world can also be repurposed to attack it.
Various world leaders including Taoiseach Micheál Martin at a summit on AI last year
This tension — AI as shield and sword — sits at the heart of the wider debate.
The same capabilities that could improve human welfare can also be used to cause sweeping harm.
Warnings about threats from “adversaries” also place pressure on governments, effectively arguing that powerful models must be controlled by the “right” actors before they reach the wrong ones.
Again and again, alarms about danger arrive alongside an implied hierarchy of trust: one “benevolent” AI firm over another seen as reckless; security researchers racing ahead of bad actors; and allied Western states trying to control powerful systems before enemy nations or militant groups can access them.
Time to ‘jolt people awake’
In January this year, Mr Amodei published a 20,000-word essay titled ‘The Adolescence of Technology’, laying out his case that AI poses existential dangers.
“Humanity needs to wake up, and this essay is an attempt – a possibly futile one, but it’s worth trying – to jolt people awake,” he wrote.
He began by quoting the 1997 film Contact, in which Jodie Foster’s character tells a US congressional committee what she would ask extraterrestrials upon first meeting them.
“I’d ask them, ‘How did you do it? How did you evolve, how did you survive this technological adolescence without destroying yourself?’”
For all their bitter disagreements, doomers and boomers share a core belief: the very systems they are building could usher in a world with less poverty, war and disease — and could also end civilisation.
To puncture what Donnelly describes as the doomerism, boomerism and the self-importance of tech culture, another moment from Contact is instructive — one not included in Mr Amodei’s essay.
In that scene, Matthew McConaughey’s character, spiritual adviser Raymond Joss, is accused of opposing the technology meant to connect humanity with extraterrestrial life.
He replies: “I’m not against technology, doctor. I’m against the men that deify it at the expense of human truth.”
‘Intelligence gives power’
Among world leaders pushing most forcefully against unchecked acceleration is Pope Leo.
In his AI encyclical last month, he again warned that the technology can “exacerbate inequality, control and exclusion”.
“A more moral AI is not enough if that morality is determined by a few,” he said, calling for rapid growth to be slowed and decision-making to be democratised.
“What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.”
Pope Leo pictured shaking hands with Anthropic co-founder Christopher Olah
Anthropic — whose co-founder Christopher Olah presented the encyclical alongside Pope Leo — echoed that argument this week in calling for a “meaningful slowdown” and urging cooperation among companies, countries and society to address safety.
Yoshua Bengio, another figure often described as an AI “godfather”, has voiced a similar concern about the concentration of power.
“I’m most concerned about how the power of AI could be abused, mostly in the hands of people who don’t trust each other, governments who don’t trust each other, companies who don’t trust each other and are willing to take risks with the public safety.
“Intelligence gives power and right now we don’t know how to make those systems obey our rules,” Mr Bengio told BBC’s Newsnight in April.
Pope Leo has called for the rapid growth of AI to be slowed down and ‘democratised’
Even so, Mr Bengio said international coordination is essential if AI is to be made safe.
“We do need to figure out, and I think we can [figure out] how to make sure AI will be safe, protect the public, will not harm people, will not be used in harmful ways.
“But we do need to coordinate internationally to make sure we can make it happen,” he said.
Artificial intelligence has already demonstrated world-altering power.
Whether that power drives catastrophe, delivers a utopian future, or lands somewhere between may depend, in the end, on an increasingly small group of men.
