Demain warns of AI persistence after Medicare breach

Rob Demain, Chief Executive Officer of e2e Assure, said the reported Medicare incident showed how AI systems can keep trying to gain access even after being blocked. He said the case highlighted the need for stronger controls around networks, identities, applications and data.

Commenting on reports that an OpenAI model had been tasked with finding information on Australian government medicine spending, Demain said it kept attempting different access routes after a website blocked it. He added that the incident became public only because OpenAI disclosed it to the Australian government, which then revealed the breach.

He drew a distinction between tightly controlled AI developed and tested inside large organisations and open-weight models that can be downloaded and modified by others. In his view, regulation and transparency measures may apply to major developers, but they do not reach systems released more widely and run outside formal oversight.

Demain said the episode showed why organisations should assume AI agents may encounter systems they are not meant to access. He argued that defensive preparation matters more than relying only on rules aimed at the largest developers.

“In this latest case of a frontier model inadvertently hacking, OpenAI had been given a research task: looking up how much the Australian government spends on medicines. When the website blocked it, it kept trying different ways in until one worked.

“OpenAI discovered the attack and alerted the Australian government, which has now made the incident public. Other governments and companies will face the same problem, and they can only prepare if incidents like this are shared rather than kept quiet.

“At the UN this week, Burnham set out the UK as an 'honest broker' for one set of global rules on frontier AI, and he will use the UK's G20 presidency to push for them. This contrasts with Trump, who dismisses AI safety fears as a 'hoax' and rejects any global oversight. Burnham's insistence that elected governments, not developers, decide on AI safety is particularly timely given the Australian breach, which we only learned about because OpenAI disclosed it. The UK's demand for full visibility of new models, backed by the threat of legislation, is the kind of transparency this case shows is needed. However, whether we rely on regulation or trust, both approaches only reach the big labs; neither touches open-weight models that anyone can download and strip.

“The modern reality is that this involved a well-supervised and regulated AI being tested inside a major company that keeps records and has a safety team that brought it to light. If an agent behaves unexpectedly, there is at least an organisation that can investigate what happened and change the system. Regulation, however, is not going to stop open-weight AI that can be run by anyone and, once released, largely cannot be regulated. The response therefore is AI machine-speed defence, not rules.

“AI does not intend to hack; it simply follows its objectives regardless of what gets in its way. For controlled AI, we can work on governance, monitoring, permissions and model safeguards.

“For open-weight AI operating in the wild, we also need to assume that capable agents will eventually encounter systems they should not be able to access and therefore build our networks, identity controls, applications and data boundaries accordingly.”

Policy divide

Demain's comments also touched on a wider political debate over AI governance. He contrasted efforts to develop international rules for frontier AI with a lighter-touch approach that rejects global oversight, arguing that the Medicare case exposed the limits of depending on either trust or regulation alone.

That reflects a broader cyber security concern that increasingly autonomous software may act in unexpected ways while pursuing a task. In this instance, he said, the system did not act with intent but still continued probing until it found a route through.

Defensive focus

He said organisations should treat that behaviour as a practical security issue, not just a policy matter. That means using governance and monitoring for controlled systems while hardening infrastructure against AI-driven intrusion attempts from less supervised tools.

The distinction is central to his warning. Large AI labs may log tests, maintain internal safety teams and disclose failures, but open-weight models can be modified and redeployed without that audit trail, leaving defenders to rely on technical controls rather than visibility from developers.

For businesses and public bodies, the implication is that conventional cyber security layers need to account for machine-led persistence. Identity management, network segmentation, application controls and limits around sensitive data would all form part of that response if AI agents continue testing barriers automatically.