DWP accused of shielding AI deployments from public scrutiny

Refusal to publish information openly

Big Brother Watch added that the operation of the DWP’s “suspicion machine” has been shielded from public scrutiny because of its refusal to publish information openly.

Similar points were made by Amnesty, which added that the DWP is justifying its opacity on the basis that disclosing information could enable individuals to exploit the benefit system. “This fundamentally misunderstands how these systems are frequently discriminatory and demonstrates the adoption of a punitive approach,” it said.

“These systems often rely on identity characteristics to establish profiles or risk scores, which for ‘race’ or ‘disability’ are fixed and therefore cannot be changed by an individual to cheat the system. This results in a system that is optimised for fraud detection, rather than being optimised to serve the majority of applications which are legitimate while still being able to detect outlying cases of fraud.”

Both organisations are therefore calling for greater transparency from the UK government about how it is using these high-risk data tools, so that DWP can be held to account for the negative impacts of its systems, and people can challenge wrong decisions made about them.

“The DWP’s ongoing roll-out of high-tech algorithmic tools, which its own assessments have found to be riddled with bias, is alarming,” said Jake Hurfurt, head of research and investigations at Big Brother Watch. “This becomes even more concerning when the DWP is hiding behind a wall of secrecy and refuses to disclose key information that would allow affected individuals and the public to understand how automation is used to affect their lives, and the risks of bias and to privacy involved.

“Instead of pressing forward, the DWP should take a step back and pause the use of any model containing unexplained disparities, and it must become more transparent about how it uses high-tech tools. It is wrong to subject millions of innocent people to shadowy automated or algorithmic decisions, and refuse to explain how these work.”

Amnesty said in its report that while the “real-world impacts” of the DWP’s AI systems are apparent from people’s experiences of interacting with them and what’s been uncovered by civil society groups, the full impacts cannot be truly known without greater transparency. “Without transparency over the use of technology, there can be no meaningful evaluation of whether these systems are operating efficiently or lawfully, and whether or not discrimination is occurring,” it said.

Big Brother Watch similarly highlighted how documents such as data protection impact assessments containing vital details about the systems – including the types of personal data they process and how these are ultimately used by the models, as well as how they meet legal tests around proportionality and necessity – are either not disclosed, or are otherwise redacted heavily if they are published.

“The DWP uses the threat of welfare fraud as a get-out-of-jail-free card to operate in secret, but with machine learning tools allowing for human-light profiling on a large scale, more scrutiny and transparency is clearly needed – not less,” it said.

Imogen-Richmond Bishop, a researcher on technology, economic, social and cultural rights at Amnesty International, added that the DWP’s mission to reduce cost is at the heart of its over-reliance on these problematic technologies. “People are struggling to make ends meet and put food on the table due to cuts in social security, and yet the DWP is more concerned about experimental technologies to surveil claimants,” she said.

“The tech-enabled system to claim and manage welfare benefits is resulting in relentless dehumanisation and strain for people who are already wrestling to access their basic needs in a broken system.” 

Amnesty also highlighted in its report that despite the DWP’s stated goal of “cost-saving”, it doubts whether these professed savings will ever actually materialise.

“The UK’s National Audit Office has expressed doubts that these savings will ever materialise,” it said. “As of 2024, the DWP estimates that, since 2010, it has cost £2.9bn to implement UC. These growing costs and continually delayed completion dates bring into question whether it is even possible for the digitalisation process to introduce the efficiency and cost savings which were used to justify its introduction.”

DWP comment

Computer Weekly contacted the DWP about both reports and the claims made about its use of AI throughout the social security system.

“We want to improve the experience for everyone who needs to access and use our services, and technology plays an important part in that,” said a spokesperson.

“We ensure the appropriate safeguards are in place to guarantee the lawful, proportionate and ethical use of data and technology. All decisions regarding benefit entitlement or payment are made by DWP staff, who look at all available evidence.”