Bristol authorities have discontinued AI models for assessing child crime risks due to accuracy concerns and verification challenges.
The Avon and Somerset police, alongside Bristol City Council, have halted the use of two AI models designed to assess the risk of crimes against children due to significant accuracy issues. Authorities faced challenges in verifying the systems, as independent auditors were unable to access the source code or the variables used in the models, as reported by WIRED.
This decision follows an investigation by WIRED, civil rights group Liberty Investigates, local outlet Bristol Cable, and nonprofit Lighthouse Reports, which analyzed extensive documentation obtained through freedom of information requests. This scrutiny coincides with the launch of PoliceAI, a national initiative aimed at the testing and scaling of AI tools within police forces throughout England and Wales.
Data Collection Methods
The controversy centers on the Think Family Database, established by the Bristol City Council in 2016 to support local families and children. While the database is said to cover approximately 55,000 families, internal documents suggest it may include records for nearly 500,000 residents, highlighting discrepancies in its reported scope.
This database amalgamated various data points, including police and social information such as housing status, mental health, school attendance, and eligibility for free school meals, often without direct consent from the individuals concerned. A police data specialist described the approach as simply pooling disparate data sets.
“I put all of this into a big bucket,” he remarked.
Leveraging the Think Family Database, authorities developed machine learning models that assigned risk scores to both children and adults. Reports indicated that at least 23 models were created, which aimed to predict various outcomes including burglaries and domestic violence risks.
In conjunction, the Offender Management App targeted around 300,000 individuals, functioning as a “leaderboard” for identifying dangerous offenders.
Discontinuation Reasons
Among the models discontinued was one assessing the risk of crimes against children, which utilized data from multiple governmental bodies and anonymized information from Barnardo's concerning 1,000 previous child victims. The model's risk scoring was influenced by factors such as the child’s need for assistance, school absences, and mental health challenges.
Warnings regarding potential algorithmic bias were raised as early as 2016 by a police ethics committee, which advised caution in applying such analytics. A subsequent evaluation by the nonprofit Social Finance found that the risk scoring was particularly weak, with low accuracy undermining the models' effectiveness. By the time of this review, two child crime risk models had already been discontinued.
Social Finance attributed the decline in model quality to inadequate data sharing agreements among local councils, resulting in models primarily based on police data without essential social indicators. Staff members expressed concerns that the models often overlooked vulnerable children, leading to situations where recent victims received lower risk scores than individuals involved in burglaries.
Further complicating matters, Social Finance was unable to verify the models due to missing documentation regarding their development and discontinuation.
Audit Results
In a separate audit, WIRED reviewed over 36,000 assessments from the police concerning 13 models tested from 2017 to 2024. The findings revealed that most models exhibited poor accuracy in identifying true risks, with the burglar identification model achieving a positive identification rate of less than 10% over three years. Such performance metrics are notably below the standards expected from operational models.
The police clarified that some models, including the burglar identification tool, were never implemented, attributing the long-term performance assessments to an oversight in managing static files. They also stated that AI tools are used only as advisory resources for officers and do not determine actions independently.
Currently, the Bristol City Council utilizes only the NEET risk model, which assesses the likelihood of children disengaging from education post-school, emphasizing that it does not replace professional judgment.
Broader Context with PoliceAI
This development occurs alongside the UK Home Office's establishment of PoliceAI, a center dedicated to testing and scaling AI tools across police forces in England and Wales, with a budget of £75 million over three years. In its initial year, the focus will be on tools that analyze and summarize digital evidence, with trials planned for 2026-2027.
Notably, the head of PoliceAI previously served as Chief Constable of Avon and Somerset Police, where the controversial AI models were developed. This highlights ongoing concerns about the risks associated with AI in policing, including data quality and transparency issues.
The broader implications of AI in law enforcement have been underscored by past incidents, such as the wrongful incarceration of Maryland resident Alonzo Sawyer due to a flawed facial recognition match. In another instance, London Mayor Sadiq Khan blocked a £50 million contract for an AI-based analytical system with the Metropolitan Police, citing procurement breaches. Additionally, revelations surfaced in 2023 about the FBI's covert development of a facial recognition system in collaboration with the Pentagon.
