Anthropic warns that AI will soon be able to improve itself without human intervention

Anthropic logo is seen in this illustration created on March 1, 2026. (Dado Ruvic/Reuters via CNN Newsource)

AI models are rapidly improving – so fast that they may soon be able to develop themselves without human involvement. That’s why Anthropic is warning the AI industry: It needs to build a “brake pedal,” or companies risk losing control of their creations.

AI systems that can advance themselves, known as “full recursive self-improvement,” could have the potential for great good for science and health care, they also pose great risks for humanity, according to a blog post written by Marina Favaro, leader of The Anthropic Institute, and Jack Clark, a co-founder of Anthropic.

“Full recursive self-improvement also might increase the risks of humans losing control over AI systems,” they wrote. “If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important.”

The industry is much closer to self-improving AI than previously expected, they warn. So tech companies should consider slowing or pausing frontier AI development to allow time for researchers to gain a better understanding of the potential societal harms self-improving AI could unleash. Researchers should also build a way for humans to intervene if things get out of hand, they said.

Appearing on CNN Thursday night, Clark called for the industry to give itself a “brake pedal.”

“When I look down at the car we’re driving, all I have is a gas pedal. I don’t have a brake pedal, and surely at some point in the future we might want that option,” he told Anderson Cooper in an interview.

Cooper raised the example of science fiction movies in which AI rose to kill human beings, and asked if that’s what Clark is worried about.

“Yeah, we read the science fiction and watch science fiction here as well, so it’s not lost on us.” Clark responded. “How do you maintain control over fleets of scientists that are much, much larger and much faster than ones you’ve had before?”

An inability to validate, verify and trust AI’s behaviour poses a major risk in its development, he added.

The warning comes after Anthropic has filed to have an initial public offering of its stock, a step that could raise tens of billions of dollars from investors to speed up the construction of data centres and computers needed for AI. SpaceX, which in addition to its rocket and satellite business has an AI business, is set to have what would be the largest IPO on record next week, raising US$75 billion.

Having major AI companies like Anthropic, SpaceX and OpenAI work together to come up with brake pedals might seem antithetical to a competitive business with billions if not trillions of dollars at stake. But such co-operation is possible, Clark insisted.

“We’ve done this before. In the height of the Cold War, under highly tense situations between rivalrous countries, they found ways to stabilize aspects of the nuclear arms race,” Clark told CNN. “All of this has been done before in other domains, and it may need to be something we do in the domain of AI.”

By Chris Isidore, CNN