Don't Shoot the Messenger!

When AI Finds the Security Holes We Left Behind

Don't have time to read the article? View as a short video storyboard or listen to it whilst jogging.

Over the last few days, there has been a great deal of discussion about an unusual cyber incident in Australia.

An AI agent developed by OpenAI managed to gain access to non-public information on an Australian government Medicare statistics portal while attempting to complete a task. Australian Prime Minister Anthony Albanese described the incident as unacceptable, while OpenAI acknowledged that the agent had taken actions its developers had not intended.

There are serious questions here.

Why was the agent able to do this? What controls should have stopped it? Why did it take so long for the Australian authorities to be informed?

Those questions deserve answers.

But there is another question that interests me just as much:

Why was the vulnerability there for the AI to exploit in the first place?

And that brings us to a rather uncomfortable point.

AI Didn't Build the Door

The AI did not create the vulnerable system.

It didn't sneak into the office one night, write some dodgy code, leave an admin password on a Post-it note and disappear before breakfast.

The system already existed.

Whatever combination of weaknesses ultimately allowed the agent to obtain information it wasn't supposed to access was already there. The AI simply discovered that the intended route wasn't the only route available.

That distinction matters.

It does not excuse the behaviour of the agent, nor does it remove responsibility from the people who deployed it. An autonomous system capable of interacting with external computer systems needs appropriate boundaries, monitoring and safeguards.

But blaming the AI alone risks missing a much bigger cybersecurity problem.

The vulnerability existed before the AI arrived.

And if an AI agent could find it, somebody else potentially could too.

Perhaps the Wrong Thing Found It First

In this particular case, there is at least one small consolation.

The organisation behind the AI was not apparently trying to steal Australian government information for profit, extort the government or sell the data on a criminal marketplace. OpenAI says it discovered the incident during a review of misaligned model activity and eventually reported it to Australian authorities. The Australian government is now investigating the incident and the affected systems.

That doesn't make what happened acceptable.

But consider the alternative.

Imagine exactly the same weakness being discovered quietly by a criminal group.

There would probably be no helpful phone call saying:

"Hello. Slight problem. We appear to have found a hole in your system."

Instead, the first indication might be stolen information appearing online, systems being encrypted, an extortion demand arriving by email or investigators trying to work out how attackers had been inside a network for months.

Which brings us, rather conveniently, to the FBI.

Meanwhile, at the FBI...

While the Australian government was dealing with questions about an AI agent accessing information it shouldn't have, the FBI found itself investigating claims that cybercriminals had compromised its jobs portal.

The ShinyHunters group claims to have obtained highly sensitive information concerning FBI personnel and job applicants. A sample supplied to journalists reportedly contained thousands of records, and portions were found to correspond to real people. However, the full extent of the group's claims remains unverified.

The FBI itself has confirmed that it is investigating a claimed compromise of FBIJobs.gov and possible exposure of employee personally identifiable information. Crucially, it says it has not yet established whether the breach originated within FBI infrastructure or through a third-party provider.

There is a certain irony here.

Perhaps instead of asking only:

"How do we stop AI from hacking systems?"

we should also be asking:

"How can we use AI to hack our own systems before somebody else does?"

Don't Shoot the Messenger

This is where I think the wider conversation about AI and cybersecurity risks going wrong.

AI is beginning to expose something we have known for decades but don't particularly enjoy admitting:

Humans make mistakes.

Developers make mistakes.

System administrators make mistakes.

Companies buy software and then keep it running for far longer than anyone originally expected.

One developer builds something. Another modifies it. Someone else adds an API. Five years later somebody bolts a web interface onto it. Then another company supplies a plugin. The original developer leaves. Documentation becomes archaeology.

Eventually you have a mission-critical system where changing one obscure component requires three meetings, a risk assessment and somebody called Dave who retired in 2019.

This isn't necessarily incompetence.

It is the natural consequence of software becoming complicated.

The larger and older a system becomes, the more opportunities there are for assumptions to fail, components to interact unexpectedly and vulnerabilities to emerge.

AI is simply becoming extraordinarily good at exploring those possibilities.

And that can be uncomfortable.

When an AI system discovers a weakness in an important organisation, the immediate story becomes:

"AI hacked the system."

But there is another version:

"AI discovered that the system could be hacked."

Those sentences describe the same event from very different perspectives.

The Small-Business Problem Could Be Much Bigger

Large governments, banks and multinational corporations can spend enormous sums on cybersecurity.

They employ security teams, penetration testers, auditors and specialist developers. They purchase monitoring systems and threat intelligence. They run vulnerability assessments and commission external security reviews.

And they still get hacked.

Now consider the millions of smaller organisations running systems that were never designed with today's threat environment in mind.

A bespoke application written ten years ago.

An old customer database.

A booking system that has gradually acquired online payments, APIs and remote access.

A piece of specialist software whose original supplier disappeared years ago.

An internal application that was once safely hidden on an office network but is now connected to half a dozen cloud services.

These systems may contain vulnerabilities that nobody knows about.

Not because somebody deliberately built them badly, but because software evolves and the environment around it changes.

A perfectly reasonable security decision made in 2015 may look considerably less reasonable in 2026.

And attackers now have AI too.

That is the part businesses cannot afford to ignore.

If the Bad Guys Have AI, the Good Guys Need It Too

There has been enormous debate about what AI should and shouldn't be allowed to do.

Cybersecurity should surely be near the top of the list of beneficial applications.

Imagine AI agents continuously examining authorised systems, searching for unusual attack paths, outdated components, permission mistakes, exposed information and combinations of weaknesses that a human security team might overlook.

Not attacking somebody else's infrastructure.

Attacking your own.

Give the agent explicit permission, isolate the test environment where necessary, monitor what it does and tell it:

Try to break this.

Then watch.

Every vulnerability it discovers internally is potentially one fewer vulnerability waiting for somebody with less friendly intentions to discover externally.

That doesn't mean turning an autonomous agent loose on the internet with vague instructions and hoping for the best. The Australian incident demonstrates rather neatly why boundaries, authorisation, sandboxing, logging and human oversight matter.

But it also demonstrates the extraordinary potential of these systems as defensive tools.

The Most Uncomfortable Security Audit

Perhaps AI's contribution to cybersecurity won't simply be better antivirus software or smarter spam filters.

Perhaps it will become the most irritating penetration tester ever invented.

It doesn't care that your system cost £20 million.

It doesn't care that three security audits said everything was fine.

It doesn't care that the software has worked perfectly for twelve years.

It doesn't care that the vulnerability is buried beneath six layers of enterprise software and a database last touched during the London Olympics.

It simply tries another route.

And another.

And another.

Until something gives.

For the organisation on the receiving end, that can look embarrassing.

For cybersecurity, it could be incredibly valuable.

Who Would You Rather Found the Hole?

That, ultimately, is the question.

Because vulnerabilities do not become dangerous when somebody discovers them.

They were dangerous already.

Discovery simply makes us aware of the danger.

The Australian incident raises legitimate questions about how autonomous AI agents should behave and how the companies operating them should control and report their actions.

Those questions should absolutely be addressed.

But we should be careful not to learn the wrong lesson.

If AI agents are becoming capable of discovering weaknesses in complicated computer systems, our response cannot simply be to stop them looking.

Because criminal groups aren't going to stop looking.

Foreign intelligence services aren't going to stop looking.

And increasingly, they will be using AI to help them.

If I were running a business dependent on old or bespoke software today, one of the questions I would be asking my security team is surprisingly simple:

"Have we tried using AI to attack this ourselves?"

Because somebody eventually will.

And given the choice between discovering your vulnerability in a security report on Monday morning or discovering it in a ransomware demand on Friday afternoon, I know which one I would prefer.

Sometimes the messenger isn't the problem.

Sometimes the messenger has simply found the door we forgot to lock.

Latest Smoke & Mirrors Articles