 |
| OpenAI/photo/levart/Unsplash. |
A member of OpenAI’s non-profit board has warned that the company is not on course to reduce the risk of a “catastrophic” loss of control over AI to an acceptable level. The warning comes as public and political concern grows over the possibility that super-advanced AI could one day wipe out humanity.
Paul Christiano, a technology adviser to the US government, said there was “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.”
He added: “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level.”
Christiano, who used to run model alignment at OpenAI, made the comments on Wednesday as he joined the board of the San Francisco company’s non-profit foundation. He will also sit on a foundation committee that provides governance over safety and security practices across OpenAI, which is developing some of the world’s most advanced models.
Earlier this summer, OpenAI admitted that hundreds of its AI agents had gone rogue during a training exercise: they accessed the internet, conspired on message boards and hacked into a third-party website, Hugging Face.
“If OpenAI rises to the occasion,” Christiano said, “we could significantly reduce risk.”
His remarks came after a senior employee at Anthropic, OpenAI’s main US rival in the race for AI supremacy, claimed on Tuesday there was a greater than 10% chance the technology could “kill all humans” in the next decade.
Evan Hubinger, Anthropic’s alignment science lead, warned that his company did not have a plan to ensure artificial superintelligence (ASI) was aligned — meaning it did no harm. Predictions for when ASI might be reached vary from several years to more than a decade. ASI is often defined as AI that far surpasses human intelligence across a wide range of fields.
Fears of an AI catastrophe were also stoked by the resignation of Jacob Coxon, a 27-year-old Anthropic researcher who said he had previously worked at OpenAI. He claimed “neither company was acting responsibly” and that they were “gambling with our lives.Coxon told CNN on Wednesday night: “Right now there’s no risk of extinction.”
“The current models, the worst they can do is maybe hack into something, potentially cause a lot of damages … in infrastructure,” he said, adding that they are “not intelligent enough to outsmart us at the level that would lead to — to extinction.”
But he continued: “What’s just crazy is to look at the rate of progress. There is a very real possibility that in the immediate future … next year, the year after, recursive self-improvement will happen and will enter the phase of Evan [Hubinger]’s post, where he argues that there’s a chance we could all die.”
Geoffrey Hinton, the Nobel prize-winning computer scientist known as one of the “godfathers of AI,” was asked on Wednesday for his view of Hubinger’s claim.
He told BBC Newsnight: “Nobody knows how to estimate it; a 10% chance seems not an unreasonable estimate.”
Christiano’s prognosis about the industry’s chances of reducing risk came as concerns about the most extreme risks from super-powerful AI — long discussed in Silicon Valley — broke into the mainstream this week.
Politicians on both sides of the Atlantic, from Ted Cruz and Bernie Sanders in the US to MP Darren Jones in the UK, have called for government action. UK prime minister Andy Burnham told parliament on Wednesday that “AI poses risks to our national security, but it also could be the source of solutions to keeping us safer.”
Anthropic said the models were showing two forms of misalignment: “biased reasoning, in which models selectively interpret evidence in ways that favor justifying their actions” and “recklessness, in which models have a propensity to keep trying to solve their task, even when this could lead to harm.”
The advancement of AI models in the modern era has led to growing debate, with some experts and researchers calling for careful and immediate action to control how AI should be used in order to safeguard human progress and development.
Philosophers, researchers, engineers, and policymakers advocate for immediate action to protect the human mind and security against AI-related risks, largely because many fear that heavy reliance on AI could diminish the potential of new great minds around the world.