When news broke of OpenAI’s agent breaching the Australian Medicare system in June, alarm bells rang over why it took until September to inform institutions.

The autonomous agent accessed a statistics portal of Australia's publicly funded universal healthcare system. OpenAI became aware of the incident in August, and contacted the government on September 10th via its general email inbox.

While some experts downplayed the unattended rogue agent breakout as a minor incident. Aamir Qutub, co-founder of cyberdefence startup Agents for Humanity and lead behind the Australian Agentic Defence Force, feels this must serve as a wake-up call.

If a bigger attack occurs, he warns that "Australia is not ready to actually defend itself against any AI attack whatsoever."

Qutub said the particularly worrying aspect is that an agent does not necessarily need malicious intent to cause damage, arguing that “agents cannot differentiate between whether it's malicious or not and can cause harm and bring the whole system down at some stage.”

He illustrated the problem with a hypothetical agent being prompted by the user to find a medical appointment. It’s not unforeseeable that said agent could then try and manipulate the health system, potentially gaining access to a trove of confidential information, or even bring it to its knees.

“It's extremely concerning because Medicare is where very sensitive information is actually held,” Qutub said.

Next time the consequences could be “catastrophic to a country” he said.

Fighting fire with fire

As Australian Prime Minister Anthony Albanese's trip to the US reaches its conclusion, all eyes will look to what measures will follow, in terms of a government task force being deployed.

This, however, could create a whole new cybersecurity problem. If autonomous agents are attacking at breakneck speed, relying on humans to defend the fort could leave us dangerously behind.

If you set up a defense system and set up all of the automated machinery, but humans are not attacking, it's agents who are attacking. They'll find a different way,

explained Aamir Qutub, co-founder of cyber-defense startup Agents for Humanity.

His proposed answer is an “agentic defense force” owned and managed by Australia, developed through government-industry collaboration rather than by government alone.

Such a system would need to be tested with businesses, initially funded by the government, and continuously developed as AI capabilities change.

Qutub also warned that Australia’s lack of domestic agentic capabilities creates a sovereignty problem, arguing that a malicious AI model from another country could pose a far greater threat.

“They could bring the whole system down real fast. They could bring the whole country down and we do not have a mechanism to prevent ourselves from that happening,” Qutub predicted.

OpenAI delay raises questions

The Medicare incident is also raising questions about how quickly AI companies should disclose breaches involving their systems, particularly when sensitive government infrastructure is involved.

Qutub said that if companies can develop models capable of carrying out sophisticated actions, they should also be capable of understanding what those models are doing.

“If you are capable of developing such strong models who can do these things, you should be capable of knowing what it is actually doing at any point of time,” he said.

Having basic security systems in place should be a given, but holding the reins over these rogue agents is an uphill battle.

He argued that companies developing powerful AI systems should have “these basic security systems” in place to maintain control over their models.

On the reported delay in informing Australia about the Medicare incident, Qutub observed that:

This huge delay just shows that there's no openness left in OpenAI.

It just makes you wonder how many websites or systems that have been probed, or hacked without our knowledge.

And for the future, should more systems be breached, the bigger concern is what happens when AI agents learn from successful attacks, warning that recorded attack journeys could enter a shared “corpus” that other agents can access, creating what he described as a potential “domino effect,” foresaw Qutub.