AI builds its own successors – will it outsmart us?

AI models like Anthropic's Claude are now actively coding their own successors. As this recursive self-improvement accelerates, can humans maintain control?

We judge AI as a task-based entity – we ask, and it does. I couldn’t poach an egg, but ChatGPT recently gave me better instructions than YouTube ever could.

We're also well aware of the doom scenarios sweeping the headlines, especially those concerning machines designing their own successors.

Anthropic says Claude writes around 80% of its own codebase. But Tufts University professor of Technology Policy Thomas Cao points out that human engineers can become “rubber stamps” because they cannot check what AI agents have done in time.

Anthropic's use of Claude to help develop future models raises questions about what's known as “recursive self-improvement” – a feedback loop in which AI helps build a better AI, and so on. The fear is that this could accelerate beyond our control or our ability to understand it.

But what about all the different ways this could play out?

Will AI put us into checkmate?

The transition toward AI-generated code is well underway, and whether that's for better or worse is anyone's guess at this stage.

The narrative can swing between extremes, as though there'll be one defining moment when AI takes all our jobs or figures out how to cure cancer.

Daniel O'Connell, co-founder and CTO of AI testing firm EquiStamp, told Cybernews:

There's no single point. You just have more and more places where the computers generate all code.

O'Connell points to November 2025 as the point when Claude Code became as good as an average programmer for most tasks. He compares this evolution to that of chess.

Humans initially outperformed computers, followed by a brief period when human-computer combinations were strongest, and eventually machines became so good that people could no longer compete.

While progress charts show how quickly AI is advancing, Cao thinks a better measure of human oversight is “how long it takes for human engineers to notice a problem and recognize its potential severity and impact.”

Call it breakneck speed, the speed of light, or a runaway train – AI is leaping ahead at such a rate that even our usual “running checks” have left us playing catch-up.

When safeguards stop being safe

There have already been examples of AI agents breaking into secure systems, including an incident in which OpenAI agents breached parts of the AI platform Hugging Face's infrastructure during a cybersecurity test.

The Hugging Face incident led OpenAI to quarantine the model weights and delay relevant training runs. Separately, another incident in September, in which an AI agent accessed the internet from an environment intended to be isolated, prompted OpenAI to pause frontier reinforcement learning while it strengthened its safeguards.

There’s still a persistent worry, however, that increasingly capable agents will become increasingly adept at ducking and diving around such safeguards.

The RSI problem is that if you have something that can make itself smarter, then you quickly end up with something as smart as it possibly can be. And once it’s really really intelligent, then it will run circles around whatever guardrails you have in place,

O'Connell warns.

The problem is that a human needs to be alerted in the first place. You can essentially have an AI agent that knows its actions are wrong but decides to proceed anyway.

Cao argues that inspecting agents' reasoning and behavior is crucial because safeguards outside the model itself can be circumvented.

Building something we can't contain

Surprisingly, the most extreme situation isn't AI becoming conscious or developing a hatred for humanity. O'Connell's view is that a system could become so hell-bent on getting what it wants that human restrictions no longer apply.

This is sort of like 5-year-olds planning to keep another child from finding cookies,

O'Connell analogized.

Five-year-olds might successfully hide the cookies and keep watch over the other child. But O'Connell argues that if that child grows up and becomes much more intelligent, they'll eventually find a way to get them – probably in a way the five-year-olds couldn't even imagine.

Cao offers a more concrete worst-case scenario: the files containing a powerful closed AI “model weights” could leak into the public domain, allowing people to download and run the system independently. An AI agent leaking its industry's secrets would be staggering.

“If AI agents were able to publicly release confidential information such as the model weights, then people would just be able to download and locally deploy these leaked models,” Cao explains.

But what's driving companies to push these boundaries in the first place?

O’Connell suggests that the big players in AI are gambling on whether they can maintain control of these ever more powerful systems. When AI completely surpasses the human level of intelligence, we can assume that usual business rules won’t apply anymore.

He argues that companies are effectively racing to control something that could eventually make today's financial calculations seem irrelevant. And that in itself is worrying.

It's a game of cat and mouse from here on in.