AS Photo Project - stock.adobe.c

News

Amnesty calls for ban on AI risk-profiling systems

Amnesty International says AI-driven risk profiling systems are discriminatory and may lead to misleading results that violate international human rights law

Larissa Steel
By
Published: 16 Jun 2026 9:30

The use of artificial intelligence (AI) for predictive profiling or risk assessment should be banned in a range of “high-stakes contexts”, such as policing, migration and welfare, says Amnesty International, on the basis that it can only entrench existing patterns of discrimination.

According to a report published by the group on 11 June 2026, automated AI-powered risk profiling systems are leading to false criminal accusations against marginalised groups, and are incompatible with international human rights law.

The report highlights how AI risk profiling – used by public authorities in law enforcement, social security and migration to identify potential offenders and assess whether a person or group is likely to break a law, before an offence is committed – is resulting in discrimination based on race, ethnicity, socioeconomic status and disability, violating the right to equality and non-discrimination.

“The way these tools are deployed can be the result of pre-existing stereotypes and prejudices which regard marginalised groups as inherently criminal or dangerous,” said Alexander Laufer, Amnesty International Netherlands researcher on technology and human rights. “Individuals or groups are transformed from statistical, hypothetical suspects into actual suspects, solidifying pre-existing prejudices or generating new ones. This is the result of existing systemic discrimination.”

AI-based risk profiling systems trained on discriminatory data

So far, the technology has been deployed across Sweden, Denmark, France, the Netherlands and Australia, in social fraud investigations and debt recovery schemes.

Use of one AI system was suspended in Sweden in November 2025, after investigations found that the technology disproportionally and incorrectly flagged marginalised groups for investigation over social benefits fraud.

Prior to the suspension of this system, Amnesty described the use of this profiling system as “akin to a witch hunt”.

Data based on existing systemic discrimination means marginalised groups can be targeted with an increased risk of being labelled as “suspicious”. Amnesty flagged that this includes “racialised people, Muslims, people on the move, people with disabilities or chronic illnesses, and people on low incomes”.

Laufer said AI risk profiling can lead to false criminal accusations, imprisonment, homelessness, deportations, and denial of social benefits, while the lack of transparency in the technology can leave individuals “unable to challenge these systems and decisions that affect their rights”.

He added: “It also poses risks to other human rights, such as the right to fair trial, the presumption of innocence, the right to privacy and data protection, social security and an adequate standard of living, and full realisation of human dignity.”

Can AI algorithms predict potential criminal behaviour?

Despite concerns raised by Amnesty and others that the technology is consistently inaccurate and scientifically dubious, governments continue ramping up their use of AI-powered risk profiling as a “cost-effective way to fight crime, social security fraud and irregular migration”.

However, in Laufer’s view, it is “impossible to design an objective or neutral risk profiling algorithm – data about people is never objective”.

The report added that the data required to predict whether an individual will commit a crime does not and cannot exist, meaning unreliable and biased proxy data is being used as indicators to predict potential criminal behaviour.

It also noted that, because human behaviour is “inherently indeterminable” and “adaptive”, it is impossible to predict future behaviour accurately.

“In some cases, there is no plausible connection between observable data and the proposed behaviour being predicted, such as between race or ethnicity and criminality – namely, racial profiling,” it said.

“In other cases, regardless of the amount of data, there is no data, or proxy data, that is good enough or objective enough to adequately model the underlying phenomenon. Such systems include risk profiling that attempts to predict criminality, life course or the propensity to commit social security fraud at the individual level or a specific location. These predictive systems have been debunked and decried as scientific malpractice.”

Targeted surveillance feeds existing biases

Laufer added that using social data to predict if someone will commit a crime inevitably targets individuals who belong to historically oppressed or marginalised groups that are overrepresented in the underlying datasets, therefore exacerbating past injustices.

“Attempts to predict fraud or criminality often amount to automatically turning marginalised communities and individuals into suspects rather than evidence-based decision-making,” he said.

AI-powered tools have previously been flagged by Amnesty as discriminatory, where surveillance tools were being used to track and deport migrants in the US.

Petra Molnar, migration and human rights lawyer and director of the Refugee Law Lab at York University, previously told Computer Weekly that AI-powered tools can replicate existing biases: “Algorithms are socially constructed, and our world is built on systemic racism and historical discrimination.”

In the UK, Statewatch previously revealed that the UK Ministry of Justice was developing data-based profiling tools to predict crimes, including a tool aimed at predicting potential murderers. Critics argued these systems worsen feedback loops by increasing surveillance of poor and racialised communities, with Amnesty International warning separately that predictive policing risks “supercharging racism”.

In January 2026, the Home Office announced a £140m investment in PoliceAI – including funding for 40 new live facial recognition units – signalling a substantial scale-up of AI in policing set to roll out over the next three years.

Read more about artificial intelligence

  • Publishers can now opt out of Google AI summaries and training: The UK’s competition watchdog has ruled that Google must provide online publishers and news organisations with the ability to opt out of their work being summarised by artificial intelligence, or otherwise used to train the company’s models.
  • Challenging AI hype narratives with director Valerie Veatch: Computer Weekly speaks with Valerie Veatch, the director of a documentary charting the historical development of artificial intelligence, about the difficulties of challenging hype narratives and the pressing need to build a culture of technological refusal.
  • Google AI engineer claims dismissal for opposing tech sales to Israel: ‘Our work on AI was sold to facilitate genocide’: Artificial intelligence engineer claims Google unfairly sacked them for internally criticising the company’s decision to continue supplying technology to the Israeli military, despite credible claims of war crimes committed in Gaza.

Read more on AI and storage