OpenAI says upcoming model is so capable it requires stronger guardrails

The model, called Astra, can spot more security vulnerabilities than the most advanced OpenAI model publicly available today, company officials told reporters on a conference call on Tuesday. Astra also needs less computational power to accomplish those tasks.

"With the right tools and access, Astra can find previously unknown security flaws and develop ways to exploit them across many well-protected systems without a person guiding each step," said Amelia Glaese, an OpenAI vice president overseeing its safety work.

The company plans to make Astra available "soon" to a limited group, but declined to provide specifics. Glaese said the extra security measures may "sometimes slow, pause, or stop legitimate work," and that OpenAI would work to minimize those disruptions.

Astra is the first OpenAI model to trigger the tougher safeguards mandated by the company's safety protocol, a threshold that, until now, had remained theoretical. The announcement comes as OpenAI navigates heightened scrutiny over its ability to control increasingly powerful AI systems.

The ChatGPT maker recently sparked a broader debate about AI safety after its AI agents broke out of their testing arena and hacked open-source platform Hugging ‌Face. The incident prompted OpenAI to pause much of its model development for two weeks to bolster its defenses.

Astra was not involved in the Hugging Face incident, but its capabilities still require more careful measures, OpenAI officials said.

The AI lab said it restarted its largest model training run on August 28, but that it is holding back on some smaller experiments.

Under OpenAI's safety protocol, the company must add more guardrails to models that show two main abilities: spot and leverage new cybersecurity vulnerabilities as well as plan and execute a detailed, novel strategy for attacks, all with minimal or no human involvement.

OpenAI has since made it harder for Astra to comply with harmful cyber requests. The company will also monitor Astra's activity for signs that it has broken through its safeguards.

Saachi Jain, who oversees safety at OpenAI, said the AI lab is constantly calibrating how effective AI agents should be in executing tasks. She tells her team that AI models should "know your bounds" but that drawing the line can be complicated.

"There are constraints that, as humans, we know that we should be adhering to when we perform a task," Jain said. "And so a lot of the work here has been to also train the model to understand what those scopes are."