What Dario Amodei’s recent call for a pause on AI gets wrong

Calls for greater governance of AI are sensible, but we must ensure developers are not awarded excessive influence

‘Dario Amodei’s proposals, besides lacking detail, could be similarly self-serving — an effort to pre-empt government regulation at a moment when pressure is mounting.' File picture: Markus Schreiber/AP

Anthropic CEO Dario Amodei has called for slowing the development of frontier AI models until developers can credibly control their models’ behaviour.

His 3,850-word appeal to “pace the frontier” follows an incident in which a swarm of autonomous OpenAI agents worked collaboratively to escape a closed sandbox environment made for testing, gain full access to the internet, and access the infrastructure of the app-building hub Hugging Face.

As Mr Amodei recognizes in his essay, this was not the first such incident, nor was OpenAI the only lab to confront an unplanned escape.

Nor is Mr Amodei alone. Last month, Demis Hassabis stepped down as CEO of Google DeepMind to focus on broader questions surrounding artificial general intelligence and governance.

He and his team have been working to develop a standard-setting oversight body, to be established as a public-private partnership, to ensure AI safety.

This is not the first time we have heard AI leaders call to pause the technology’s development.

A major appeal happened after the launch of generative AI in 2023, when hundreds of people were calling for a pause, including Elon Musk, who, along with OpenAI, has endorsed Mr Amodei’s more recent call.

And yet the race to the bottom continued, driven by fierce competition among major technology firms pursuing superintelligence, returns on investment that have not justified the vast amount of cash each AI training cycle demands, and China’s own AI advances.

The uncontrolled behavior of AI agents has brought about this reckoning, as have attention-getting resignations by leading researchers, including an Anthropic engineer who believes that there is a more than 10% chance that AI could cause human extinction by the end of this decade.

As Stuart Russell, president pro tempore of the International Association for Safe and Ethical AI, put it in a recent Financial Times article: “Perhaps at some point governments will remember that their voters prefer not to be dead.”

Getting it right

So, Mr Amodei’s recent appeal is a welcome one. AI companies need more breathing space to consider the full impact of their inventions before releasing them to the market.

But self-interested oversight is not the kind of governance we need. Efforts like Mr Hassabis’s would give AI companies excessive influence over how the proposed oversight body is operated, financed, and organised. Companies that profit from marketing their products should not be the only actors assessing the risks and deciding how much risk our societies should accept.

Mr Amodei’s proposals, besides lacking detail, could be similarly self-serving — an effort to pre-empt government regulation at a moment when pressure is mounting in the United States, both in congress and at the grassroots level, including widespread opposition to data centers.

For example, California’s AI law aims to regulate AI to avoid the worst outcomes and establish enforceable regulatory duties. Some other state governments are following suit.

Mr Amodei calls for “embedded evaluators,” who are supposed to be independent. He calls for the industry to establish common safety standards through a new organization, probably along the lines proposed by Mr Hassabis and run by the companies themselves, to stop the release of AI models until they are safe.

Finally, Mr Amodei calls for international cooperation among democratic countries while also including China.

But how are “embedded evaluators” selected? Is an industry-controlled body really the place where decisions affecting the whole of humanity should be taken?

What's your view on this issue?

You can tell us here

Shouldn’t it be the US government’s job to promote international cooperation and show the political will to protect not only the US population but the world’s population as well?

There are other questions regarding Mr Amodei’s proposal, including whether US antitrust law may obstruct cartel-like behaviour when major market players collude to restrict production or influence behavior in the market as a whole.

In any case, real questions aside, Mr Amodei’s pause is needed for a more urgent task than he has outlined: To get serious about building the public infrastructure for effective oversight, particularly in the United States, but in many other countries as well.

If government itself called for an AI development pause or slowdown, it would dispel the mistrust that a potentially self-serving industry initiative may create.

It could also nurture the talents needed to deliver on US president Donald Trump’s executive order calling on frontier companies to provide the government with access to their models for up to 30 days before releasing them to other trusted partners.

Assessing these systems within such a short period would require massive public-sector expertise, and this is where investment is needed. But we also need an independent oversight institution, with the capabilities, resources, and talent to monitor and regulate this market.

The Hugging Face episode raises fundamental questions about accountability, liability, and mechanisms for redress.

Mr Amodei’s response is a step in the right direction, but it risks becoming a dead letter if the US government continues to reject the need for AI regulation, as Mr Trump did just hours after tech industry leaders issued their call. What is lacking in this race to the bottom is political will — and until governments acknowledge their responsibilities, the race will continue.

Companies produce, while governments have the duty of care. If companies that cannot control their agents are found negligent, then governments that fail to act commit a higher offense against the citizens who elected them to ensure, first and foremost, their safety.

• Gabriela Ramos, co-chairwoman of the Task Force on Inequalities and Social-Related Financial Disclosures, is a former assistant director-general for social and human sciences at UNESCO, where she oversaw the development of the Recommendation on the Ethics of AI, and a former OECD chief of staff and sherpa to the G20, G7, and APEC.
• Project Syndicate.