AI expert on threat to humanity: “Currently, the situation is insane”

Charbel-Raphael Segerie, executive director of the French Center for AI Safety, has told Cybernews that the world is not investing enough in AI safety. To him, this is “insane” because the day when we might lose control of the most advanced models is fast approaching.

“I’m not against innovation. I really like it, I’m an engineer. But I think that the priority for humanity is to make sure that generative AI is not our last technology,” Segerie says.

Just like hundreds of other AI experts, he’s been worried for quite a while about the direction that generative AI development has taken. To Segerie, it looks like most large tech companies are just looking to make as much money as they can – and to hell with the consequences.

Those could be dire, he thinks, because GenAI models are improving so fast that we might soon risk losing control of them. And when that happens, even the extinction of humanity won’t sound implausible.

Self-replication is a huge danger

The French Center for AI Safety, of which Segerie is the executive director, is the co-organizer of a new initiative named “Global Call for AI Red Lines,” signed this week by a coalition of more than 200 influential thinkers, diplomats, policymakers, and Nobel laureates.

“Some advanced AI systems have already exhibited deceptive and harmful behavior, and yet these systems are being given more autonomy to take actions and make decisions in the world,” claims the call.

“Left unchecked, many experts, including those at the forefront of development, warn that it will become increasingly difficult to exert meaningful human control in the coming years.”

Have thoughts about this topic? Others do, too. Join them in the discussion.

The initiative's authors say current efforts to safeguard AI are fragmented, and corporate commitments are inconsistent. They add that even simple risks aren’t being adequately addressed.

Segerie wholeheartedly agrees. He told Cybernews that if the world does nothing, it will definitely be possible to face AI-engineered pandemics in the near future.

Self-replication with no human intervention is broadly recognized as one of the principal red lines associated with frontier AI systems.

“Besides, we could lose control of AI systems due to self-replication. Autonomous self-replication would act like a virus on the internet and copy itself exponentially. This type of AI could arise as soon as late 2025, although the median date is 2027,” the expert told Cybernews.

“This is why all of these risks are very urgent. We need to act very soon.”

Indeed, self-replication with no human intervention is broadly recognized as one of the principal red lines associated with frontier AI systems.

Incidentally, last December, Fudan University researchers Xudong Pan, Jiarun Dai, Yihe Fan, and Min Yang published a study titled “Frontier AI systems have surpassed the self-replicating red line.”

The paper claims that two widely known large language models (LLMs) – Meta’s Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct – have already crossed the barrier of non-reproducibility.

In other words, the AI systems managed to produce live, independent copies of themselves. Researchers warned: “Successful self-replication without human assistance is the essential step for AI to outsmart humans, and is an early signal for rogue AIs.”

“AIs that are capable of doing all kinds of harm are coming very, very quickly. Today, AI is able to automate tasks that require five hours of work for a human, but in seven months, it will be able to automate tasks that take ten hours of work. This is exponential,” Segerie told Cybernews.

“This is obviously very powerful but also very dangerous. We need to be very cautious with this technology because we can lose control of it.”

Multiple steps need to be taken

Unfortunately, at least right now, most tech giants developing the most powerful GenAI models aren’t investing nearly enough in AI safety, says Segerie. On the contrary, they’re lobbying against most efforts to regulate the sector.

“What’s currently happening is quite sad. The ratio of money invested to make AI more capable versus the amount invested to make machine learning safe is at least one to 100. The situation right now is insane,” Segerie told Cybernews.

Ironically, some tech leaders are themselves talking about the so-called PDOOM, short for “probability of doom.” It’s a way of estimating how likely AI is to lead to a disastrous outcome, and Anthropic’s co-founder and CEO Dario Amodei recently put that risk at 25%.

The risk is actually scarily high as probabilities come, even though, of course, Amodei also told Axios there’s a “75% chance that things go really, really well.”

For that to happen, according to Segerie, a binding international agreement on red lines for AI is needed. These would be the minimum guardrails that governments should agree on to prevent the most urgent and unacceptable risks.

Segerie is optimistic that the international community can find a way to avoid “the race to the bottom” – if only because it’s been done in the past.

But are international AI red lines even possible? Segerie is optimistic that the international community can find a way to avoid “the race to the bottom” – if only because it’s been done in the past.

“First, of course, a binding international agreement is needed to harmonize rules across different countries. Then, to verify that AI systems aren’t crossing the red lines, we need an international technical verification body,” said Segerie, providing the International Atomic Energy Agency as an example.

“It would independently verify whether the AI systems from any company or country comply with the red lines, and inspect them regularly.”

Sure, it at times seems that no powerful countries are actually listening to the United Nations, especially the pro-isolationist Donald Trump administration in the US.

But the Treaty on the Non-Proliferation of Nuclear Weapons (1970) and the Biological Weapons Convention (1975) were both negotiated and ratified at the height of the Cold War – and the need to make sure AI doesn’t go rogue seems just as urgent.

“AI can create risks for the whole world. It can become a national security risk for the US if, say, some labs in China create a model able to engineer pandemics, so it is in the interest of the US to develop international agreements,” Segerie told Cybernews.

“Yes, the companies lobby aggressively against any rules, but that’s because they just want to make money. Still, it makes no sense to make a lot of money if we collectively lose control of AI in three or four years.”

Unlock more exclusive Cybernews content on YouTube.