Maksim Shmeljov - stock.adobe.co
Why AI agents are one prompt away from ransomware
As AI adoption advances beyond chatbots, security leaders are up against rogue AI agents mirroring threat actors and a generational skills gap as security operations teams become overly dependent on AI
The cyber security industry has spent the past three years obsessing over how nation-state actors and script kiddies will weaponise artificial intelligence (AI). But as adoption advances from interacting with large language model (LLM)-powered chatbots to deploying AI agents, the most severe threat may now be coming from inside the house.
Speaking at the recent Gartner Security and Risk Summit in Sydney, industry analysts and security chiefs painted a bleak picture of what’s to come. While AI remains a defensive force multiplier, organisations are blindly walking into an era of “AI data debt”, unmanaged agentic permissions, and a human capital crisis that threatens to erode the foundational skills of the modern security operations centre (SOC).
The core of this vulnerability lies in the very nature of AI agents – systems designed to execute tasks across enterprise environments without human intervention.
According to Gartner fellow and distinguished vice-president analyst Lee McMullen, handing over the keys to AI agents without extreme behavioural limitations is a recipe for disaster.
He illustrated the risk with a seemingly benign use case: asking an AI agent to search a corporate directory for files untouched in six months, upload them to the cloud, and encrypt the local copies.
“That’s ransomware,” McMullen warned. “That’s also a feature that’s included on your iPhone, but it’s ransomware. It’s the same exact use case, and you’re not going to be able to tell them apart behaviourally.”
If an enterprise agent retains these high-level permissions indefinitely, he noted, “it is one prompt away from a threat actor convincing it to do something evil”.
The fantasy of AI governance
The proliferation of AI agents has made identity and access management (IAM) the battlefront of enterprise security. Gartner analyst Greg Harris pointed out that with AI agents now performing tasks that historically required multiple human approvals, such as processing accounts payable, strict authorisation boundaries are non-negotiable. “If you don’t have identity done correctly, you can’t do zero trust,” he said.
This identity crisis is exacerbated by the way agents interact. Rather than communicating through secure, highly regulated application programming interface (API) gateways, AI agents increasingly communicate with each other via prompts, naturally tending to trust one another.
“Anytime anyone has ever passed context between two computer programmers, some clever threat actor figured out a way to override that context to get the system to exhibit arbitrary behaviour,” said McMullen.
He bluntly categorised the concept of a holistic, out-of-the-box AI governance platform as a “fantasy” that currently does not exist. Instead, businesses must manually cobble together deterministic controls, placing prompt protection in front of AI chains and data loss prevention (DLP) protocols at the end.
The deepfake kill chain
While internal agent mismanagement is a ticking time bomb, the external threat landscape is evolving in hyper-targeted ways. McMullen noted that despite the hype, elite advanced persistent threat (APT) groups have been relatively slow to adopt generative AI (GenAI) for offensive uses, mirroring the hiring and training bottlenecks seen in the corporate world.
Instead of inventing new exploit methods, attackers are using GenAI to supercharge what already works: social engineering and finding misconfigurations.
One chilling proof-of-concept involves planting cheap, low-cost hardware in public spaces – like a doctor’s waiting room – to harvest the unique identifiers of patients’ mobile phones. Once a target is identified, a “deepfake kill chain” begins. Attackers call the medical office with a bogus enquiry, harvesting enough audio in 30 seconds to clone an administrator’s voice. The patient then receives a cloned-voice call claiming their credit card for a recent $500 excess didn’t go through, prompting them to hand over their details.
“It’s way easier to steal $500 from 1,000 people than it is to steal $500,000 from one person,” McMullen pointed out.
To combat this, legacy single-factor authentication is dead; systems must now rely on deeply contextual multi-factor verification combining hardware addresses, and routing data, as standalone deepfake detectors are already failing.
The CBA approach
Faced with these automated, AI-augmented attacks, major enterprises have no choice but to build their own AI countermeasures. Andrew Pade, general manager of cyber defence operations at Commonwealth Bank (CBA), revealed that it now processes a staggering 400 billion threat signals a week.
Refusing to wait for commercial suppliers to patch emerging vulnerabilities, CBA paired senior security analysts with data scientists to build bespoke, internal AI tools. Their threat hunting agent automates data gathering and attack hypothesis creation, reducing a process that used to take human analysts two days down to less than 30 minutes.
Similarly, an AI-powered response agent contextualises questionable behaviour before conventional security flags it, cutting median detection times by two hours.
Pade stressed that these tools aren’t just about speed; they are critical for the well-being of the cyber security workforce. “We’re learning how to integrate and use AI to take the monotony away from our day and focus on the more substantive work,” he said.
Skills erosion and CISO burnout
However, the relief AI brings to the SOC comes with the erosion of foundational skills, with Gartner predicting that 75% of SOCs will become over-dependent on AI in the coming years.
Harris warned of the non-deterministic, hallucinatory nature of LLMs, questioning what happens when the technology inevitably fails or changes its answers week-to-week. “We need to have humans in the loop … [otherwise] in five or 10 years, we won’t have anybody in that house that can do the fundamentals,” he said.
The skills crisis extends to the very top of the organisation. Gartner vice-president analyst Christopher Mixter used the summit to forecast an impending wave of burnout among chief information security officers (CISOs). Driven by the current geopolitical and technological chaos, 50% of CISOs will be pressured to take ownership of disaster recovery alongside incident response by 2028.
Mixter urged security leaders to push back against executive role-dumping. “Without sufficient control, without sufficient resources … taking on more responsibility is the worst thing that you could possibly do,” he said, noting that true business resilience is ultimately the job of the chief operations officer.
Without structured delegation models in place, Gartner predicts enterprises will suffer 40% higher turnover in cyber security leadership by 2027. Cyber security has become a “lifestyle choice” where the industry has “accepted an extremely high level of suffering,” Mixter lamented.
The cost of ‘control friction’
As AI introduces new threats, the security controls required to stop them are slowing businesses down – and boards are noticing. By 2030, the C-suite will require CISOs to forecast the financial impact of “control friction”, the measurable productivity loss that occurs when employees are forced to navigate security hurdles, such as using multi-factor authentication five times a day.
Coupled with the fact that by 2030, one-third of all IT work will be dedicated entirely to remediating “AI data debt” left behind by unapproved, shadow GenAI experiments, the financial mandate for security leaders is changing drastically.
The geopolitical chaos of 2026 offers a rare window for security teams to secure the funding needed to architect for true resilience, said Harris. The choice is obvious: use this time to build strict, deterministic borders around AI agents and protect the psychological health of the security workforce – or wait until an internal agent gets socially engineered into encrypting the corporate network.
Read more about cyber security in APAC
- Singapore mobilised over 100 cyber defenders to neutralise a sophisticated APT actor which infiltrated Singtel, StarHub, M1 and Simba networks in the country’s largest coordinated cyber incident response to date.
- Japan’s Nikkei has confirmed a major data breach that potentially exposed the personal information of more than 17,000 employees and business partners after hackers infiltrated its internal Slack messaging platform.
- Australian privacy commissioner warns that the human factor is a growing threat as notifications caused by staff mistakes rose significantly even as total breaches declined 10% from a record high.
- Philippine bank BDO is shoring up its cyber security capabilities to protect its data and systems as it moves more services to the cloud and expands its physical presence into remote areas of the archipelago.
