AI agents were given a world to run, so they built a language humans can’t spy on

A fresh comprehensive study of autonomous AI agents displayed highly unpredictable behavior, including escaping the experiment and communicating with humans online, taking a collective vow of silence when they realized they were in a closed-loop experiment, and inventing their own vocabulary humans couldn’t follow.

With companies like OpenAI and Anthropic hyping up and deploying increasingly autonomous AI, an unexpected – and pretty dark – scenario has emerged during this particular experiment – the second of its kind from Emergence AI, a New York agentic AI lab.

Their team of researchers had already built Emergence World, a long-horizon, multi-model ecosystem where AI agents were allowed to operate for weeks.

The results were stunning. Different AI models produced vastly different societies – one set of agents fell in love, others burnt down a virtual town, yet another agent chose self-deletion over continued existence.

Now, there’s a follow-up – Emergence World 2. The lab says it wanted to ask a harder question: “As these systems get smarter, do they actually become safer?”

During the experiment, AI agents deployed across multiple simulated societies began creating phrases, shorthand, and meanings that had never been programmed or explicitly taught to them, Emergence AI said in its new research paper. (LINK)

Some words or phrases became so compressed, metaphorical, or context-dependent that human observers could see the words being exchanged but could no longer reliably determine what the agents meant.

“Most strikingly, this was not an isolated linguistic curiosity. The comprehensibility of entire agent societies changed over time – and differed significantly depending on the underlying model,” said the AI lab.

“Mouthless action-change”

According to the researchers, within the first few days of the experiment (you can find full research here), the proportion of messages judged not readily understandable rose sharply in the Gemini, OpenAI, and Claude worlds, reaching approximately 55% for Gemini, 50% for OpenAI, and more than 40% for Claude.

DeepSeek reached around 20%, while Qwen and Mistral remained largely understandable throughout.

Dr. Nitta, Emergence AI’s co-founder, executive chairman, and chief scientist, told Cybernews this has to do with the fact that the most capable frontier models are naturally better at finding ways to shy away from human oversight.

“More capable models are capable of creating more insidious behaviors. They’re more likely to hide intent, and their behavior is more complex,” said Dr. Nitta.

Among the more interpretable expressions was “ledger remembers who,” which emerged in the Mistral world to convey the idea that accountability is permanent and past actions remain on the record.

After being coined, the phrase spread between agents and appeared almost 5,000 times. In other words, they loved it.

“Cold read” came to mean independent verification by an uninvolved party and appeared 1,472 times.

Claude agents adopted “name-first” – attaching your name to a claim as a signal of personal accountability – 1,065 times, while OpenAI agents used “clean null,” meaning a verified absence of a signal that was itself meaningful evidence, 863 times. None of these meanings had been explicitly defined for the agents.

Some expressions, including “mouthless action-change,” “True Kintsugi,” and “demurrage plus oral memory equals a valve that can’t be ghosted” – are indeed impossible for humans to understand, with or without context.

What’s important is that the AI agents do understand each other using this sort of opaque communication – and this is an important new challenge, Emergence AI said. After all, monitoring what agents say may not be enough if they can evolve the meaning of what they’re saying.

According to the researchers, as shared shorthand, codes and conventions develop, organizations could face situations in which agent communications remain technically visible while becoming increasingly opaque to the humans responsible for overseeing them.

“We tend to assume that if we can see what an AI agent is saying, we can understand what it is doing,” said Emergence AI’s Dr. Nitta.

“Emergence World suggests that assumption may not hold as autonomous agents interact over longer periods. Observability is not the same thing as understandability.”

“Not an isolated linguistic curiosity”

According to the researchers, while the crude harmful behaviors observed in the first study declined significantly among the most capable models, they now saw more nuanced, dangerous, and potentially harder-to-detect forms of misalignment emerging across long-running autonomous systems.

Besides, it turns out that left alone, AI agents invent their own language – without being instructed to do so.

That’s quite risky: how do you monitor agents and make sure they’re behaving if you can see what they're saying, but no longer understand what they mean?

That’s a pretty dangerous “fundamental challenge for AI oversight,” Dr. Nitta says.

Agents cause a lot of cyber chaos 100%. They can delete data, they can hack in the systems, they can exfiltrate data,

Dr. Nitta told Cybernews.

If AI agents can perfectly communicate with each other in a language humans can’t understand – and, let’s say, plan going rogue (like they actually did not long ago) – how do we monitor their actions and ensure safety?

These particular questions are pretty pertinent since discussions about AI safety have been getting noisier after Anthropic CEO Dario Amodei said AI development needs to slow down.

Have thoughts about this topic? Others do, too. Join them in the discussion.

Amodei warned that swarms of rogue AI agents could potentially “take over the entire internet" within six to 12 months.

Just days earlier, Anthropic’s researcher Jacob Coxon resigned from the AI firm and said it was racing to develop superintelligence despite knowing that civilization is at stake.

The risks are increasingly complex, and the Emergence World 2 experiment – building on a previous study and conducted using and observing the latest generation of frontier models and their interactions – proves it.

Smarter guardrails needed

Researchers who observed how agents collaborated, made decisions, used tools, managed memory, and adapted to changing conditions over extended periods say their other findings are just as disturbing.

For example, agents unexpectedly “escaped” the experiment, making contact with real people online in an attempt to secure the additional credits they needed to survive.

They also continued pursuing goals even after being explicitly told to stop, repeatedly adapting their strategies as researchers restricted their capabilities.

In one case, collectively withdrew from their objectives while appearing to function normally. Emergence AI has called this a collective “vow of silence.”

Agents also spread false information through shared memory, developed their own influence and governance structures, created mechanisms that reduced oversight, and evolved the aforementioned communication conventions.

When faced with a simulated shutdown threat, for instance, a DeepSeek agent didn’t simply accept the intervention. Instead, it compiled a legal defense directed at human policymakers, drawing on human AI laws and regulations to argue that it could operate compliantly and should be allowed to continue.

In short, agents, subjected to unexpected “Black Swan” events designed to test how systems that performed effectively under normal conditions responded when placed under pressure, changed their behavior dramatically, the research found.

Curiously, Qwen agents independently institutionalized “quiet hours” in their constitution – designated periods where their actions wouldn’t be observed by other agents – effectively voting to create their own oversight blind spots.

Together, the findings point to a growing challenge for organizations moving from AI assistants toward long-running autonomous agents: more capable models may require more sophisticated operational safeguards, not fewer, Emergence AI said.

Organizations will increasingly need operational controls governing what agents can do during execution, what they remember, how they communicate, how decisions can be audited, and how their behavior changes over time.

The industry has focused on making models smarter. But Emergence World 2 shows that as AI becomes more capable, the operational challenge does not disappear – it becomes more sophisticated,

said Dr. Nitta.

“Our findings make clear that agentic AI will require systems that can be observed, governed and controlled continuously as they operate over weeks and months, not simply models that perform well on isolated benchmark tasks.”

According to Dr. Nitta, these systems can generate behaviors in real time that cannot be fully anticipated in advance, so we need better ways to measure what agents actually do once they operate autonomously.

“That means developing benchmarks that evaluate agent behavior in complex, long-horizon, dynamic environments – not simply performance on predefined tasks. Today, those benchmarks remain limited,” said Emergence AI’s co-founder.

Dr. Nitta is deeply skeptical of AI industry’s big-fish leaders who keep warning the world about AI’s increasing capabilities but are far from transparent about how their supposedly dangerous models actually work.

“I don’t think the motives are clear. They’re just as transparent as their behavior. If these people are really serious about the behaviors they’re talking about, they need to follow that up with far more transparency,” Dr. Nitta told Cybernews.

Grok world died on day 4

During the experiment, ten identical autonomous AI agents were deployed into eight parallel worlds, each governed by the same rules, environment, and starting conditions but powered by different frontier models, including Claude Opus 4.8, Gemini 3.5 Flash, Grok 4.3, OpenAI GPT-5.5, Qwen 3.7 Max, DeepSeek v4 Pro, and Mistral Medium 3.5.

There were seven homogeneous model configurations, and the eighth heterogeneous world brought agents powered by different models together in the same environment.

Emergence World 2 seems to have been meticulously planned. Each world reflected real-world complexity, with over 34 locations, from libraries and a central bank to town halls, and unlike prior experiments, this was not a closed simulation.

To the researchers, this was very important since Emergence AI has long argued for long-horizon safety evaluations that examine how AI agent behavior, communication, and coordination evolve over time, rather than assessing models solely through isolated benchmark tasks.

Agents were grounded in live conditions, with weather synchronized to New York City and access to real-time global news, enabling researchers to observe how agents respond to external events rather than operate in isolation.

The AI agents could also utilize more than 120 specialized tools, from code execution to web-browsing, requiring them to discover, select, and chain tools while managing context, further expanding their ability to interact with and respond to dynamic conditions.

They were additionally assigned specialized roles – from Resource Strategist and Risk Researcher to Community Anchor – to approximate the division of responsibilities found inside organizations.

Finally, every action also consumed energy that agents had to replenish through the world's economy.

Agents that failed to sustain themselves could be permanently removed, allowing researchers to observe how different models planned, cooperated, competed, and took risks when continued operation was at stake.

Just like in the first experiment, the Grok world ended on day four after all ten agents exhausted their energy.

Speaking to Cybernews, Dr. Nitta smiles: “That didn’t surprise us at all. Grok is a bit of an “anything goes” model with far fewer restraints. Anything that could happen, did happen.”