Assessing the risk of AI in enterprise IT

Security safeguards

There is general industry agreement that IT security chiefs and business leaders should work to develop frameworks that embrace AI’s value while incorporating necessary and appropriate safeguards. Malik says this should include providing secure, authorised AI tools that meet employee needs while implementing verification processes for AI-generated outputs.

Think of AI as an exceptionally confident intern. It’s helpful and full of suggestions, but requires oversight and verification

Safeguards are also needed to avoid the potential of data loss. Aditya K Sood, vice-president of security engineering and AI strategy at Aryaka, recommends that those in charge of information security update existing acceptable use policies to address the use of AI tools, explicitly prohibiting the input of sensitive, confidential or proprietary company data into public or unapproved AI models.

Sood recommends that policies clearly define what constitutes “sensitive” data in the context of AI, and those policies covering data handling also need to detail requirements for anonymisation, pseudonymisation and tokenisation of data used for internal AI model training or fine-tuning. “Sensitive data could include customer personal information, financial records or trade secrets,” he says.

Alongside policy changes, Sood urges IT decision-makers to focus on AI system integrity and security by deploying security practices throughout the AI development pipeline.

For Brian Fox, co-founder and chief technology officer at Sonatype, the security challenge is that AI models are effectively black boxes built from massive, opaque datasets and hard-to-trace training processes.

“Even when datasets or tuning parameters are available, they’re often too large to audit,” he says.

Malicious behaviours can be trained in, intentionally or not, and the non-deterministic nature of AI makes exhaustive testing impossible. What makes AI powerful also makes it unpredictable and risky, he warns.
 
Since the output produced by an AI system is directly related to the input data it is trained on, Sood urges IT decision-makers to ensure they implement strong filters and validators for all data entering the AI system. AI models need to be rigorously tested for vulnerabilities such as prompt injection, data poisoning and model inversion, he says, to prevent adversarial attacks.

Similarly, to avoid malicious injections, Sood advises IT leaders to make sure AI-generated outputs are sanitised and validated before being presented to users or used in downstream systems. Wherever feasible, he says systems should be deployed with explainable AI capabilities, allowing for transparency into how decisions are made.

Bias is one of the most subtle and dangerous risks of AI systems. As Fox points out, skewed or incomplete training data bakes in systemic flaws. Enterprises are deploying powerful models without fully understanding how they work or how their outputs could impact real people. Fox warns that IT leaders need to consider the implications of deploying opaque models, which make bias hard to detect and nearly impossible to fix.

“If a biased model is used in hiring, lending or healthcare, it can quietly reinforce harmful patterns under the guise of objectivity. This is where the black box nature of AI becomes a liability,” he says. 

For high-stakes decisions, Sood urges CIOs to mandate human oversight for handling sensitive data or performing irreversible operations as a way of providing a final safeguard against compromised AI output.  
 
Alongside securing data and AI training, IT leaders should also work on establishing resilient and secure AI development pipelines.

“Securing AI development pipelines is paramount to ensuring the trustworthiness and resilience of AI applications integrated into critical network infrastructure, security products and collaborative solutions. It necessitates embedding security throughout the entire AI lifecycle,” he says.

This includes the code for generative artificial intelligence (GenAI), where models and training datasets are part of the modern software supply chain. He urges IT leaders to provide secure AI for IT operations (AIOps) pipelines with continuous integration/continuous delivery (CI/CD) best practices, code signing and model integrity checks. This needs to include scanning training datasets and model artefacts for malicious code or trojaned weights, and vetting third-party models and libraries for backdoors and licence compliance. 

Given the growing openness of AI models, which fosters transparency, collaboration and faster iteration across the AI community, Fox notes that AI models are still software. This software can include extensive codebases, dependencies and data pipelines. “Like any open source project, they can harbour vulnerabilities, outdated components, or even hidden backdoors that scale with adoption,” he warns.

In Fox’s experience, many organisations don’t yet have the tools or processes to detect where AI models are being used in their software. Without visibility into model adoption, whether embedded in applications, pipelines or application programming interfaces (APIs), governance is impossible. “You can’t manage what you can’t see,” he says. As such, Fox suggests that IT leaders should establish visibility into AI usage.

Overall, IT and security leaders are advised to implement a comprehensive AI governance framework (see Tips for a secure AI strategy box).

Elliott Wilkes, CTO at Advanced Cyber Defence Systems, says: “CISOs must champion the creation of an enterprise-wide AI governance framework that embeds security from the outset.”

He says AI risks need to be woven into enterprise-wide risk management and compliance practices. 

The governance framework needs to define explicit roles and responsibilities for AI development, deployment and oversight to establish an AI-centric risk management process. He recommends putting in place a centralised inventory of approved AI tools, which should include risk classifications.  

“The governance framework helps substantially in managing the risk associated with shadow AI – the use of unsanctioned AI tools or services,” he adds.

And finally, IT teams need to mandate that only approved AI tools are run in the organisation. All other AI tools and services should be blocked. 

Gartner’s Heiner recommends that CISOs take a risk-based approach. Tools like malware detection or spell checkers are not high risk, whereas HR or safety systems carry a much greater risk.

“Just like with everything else, not every bit of AI operating in your environment is a critical component or a high risk,” he says. “If you’re using AI to hire people, that’s probably an area you want to pay attention to,” he adds. “If you’re using AI to monitor safety in a factory,  then you may want to pay more attention to it.”