Do people like having AI make welfare decisions?

Citizens are skeptical about letting AI make welfare decisions, especially those who actually depend on social services, according to new research that reveals a troubling gap between policymakers' assumptions and recipients' real concerns about AI systems determining their benefits.

Since generative AI erupted onto the scene, governments around the world have been tempted to streamline operations and get technology to do what was previously done by humans. This so-called "AI First" approach is designed not only to be cheaper but also more efficient, but is it effective, and how do citizens feel about interacting with technology rather than humans?

That was the question posed by recent research from the Max Planck Institute for Human Development, which looked at the use of AI in various aspects of social services. These are among the most sensitive services provided by governments, and include the distribution of unemployment and housing benefits, as well as a wide range of social welfare.

A lukewarm reception

The study shows that the people depending on these services are generally positive about the role AI can play, and especially on the fairness and efficacy of automated decision tools. The researchers found that those designing and implementing these solutions seldom took into account the needs or perspectives of those affected.

The authors highlight the role of Smart Check, an AI-based system in Amsterdam that was used to try and spot potential welfare fraud. The system was designed to trawl through a comprehensive dataset to produce a "risk score."

Anyone whose score surpassed a certain level was assigned for further investigation. It won't perhaps come as a great surprise that the system didn't function quite as the developers expected, and gave disproportionate attention to vulnerable groups, such as immigrants and women. What's more, those under suspicion were given no route to challenge this accusation.

After prolonged criticism by lawyers and advocacy groups, the program was suspended earlier this year, with a subsequent evaluation highlighting the shortcomings that were evident to so many for so long.

It's a narrative that few observers of AI will be unfamiliar with. Vendors promise great leaps in terms of efficiency and effectiveness, but the reality sees pre-existing biases being reinforced and trust among users rapidly eroded.

The receiving end

This kind of scenario prompted the researchers to try to understand how people feel about being on the receiving end of AI-based decision-making. They quizzed around 3,200 people from the United States and the U.K.

The survey aimed to present participants with a realistic scenario before asking them whether they would be willing to trade faster decisions for a fairly high error rate.

The results suggest that there is a general willingness to tolerate a degree of unfairness in return for faster decisions, but this was not the case for those on the receiving end of these faster decisions. For them, it was far more important to get it right than risk getting it wrong, quickly.

"There is a dangerous assumption in policy-making that the average opinion represents the reality of all stakeholders," the researchers explain.

Trust gap

Interestingly, the results show a clear trust gap between recipients and non-recipients of social welfare. Not just in terms of how much each group trusts the technology, but how much they believe the other group trusts the technology. Non-recipients consistently thought that recipients would trust AI more than they actually do.

This trust gap endured, even when participants were given a financial incentive to try to assess things as realistically as possible. The most vulnerable groups were, in fact, much better at understanding the perspectives of the majority than the majority were at understanding the perspectives of the vulnerable minority.

The trust gap wasn't reduced when participants were told they might be able to appeal any decisions made by the AI system, with the appeal overseen by a human administrator. This perhaps reflects the lack of trust that's evident in governments more broadly, which can often seem Kafka-esque in their inpenetrability.

Government matters

This underlines the importance of building trust in government more broadly. The study shows how trust is a fluid process, with a lack of trust in government undermining trust in the AI systems produced by the government, and vice versa, the lack of trust in AI undermining trust in the government that deploys it.

This was evident when the researchers examined the potential use of AI to support the distribution of the Universal Credit welfare payments in the UK. Many of the potential recipients of this benefit complained that they would rather have human case workers ensuring the payments were made, even if AI was faster and as accurate as the humans. This didn't really change, even if they were told about the appeal process.

The study is a timely reminder that policymakers should be wary of deploying AI technology, especially in sensitive areas, such as social welfare. The findings also remind us that the apparent support of the majority is an insufficient reason for deploying AI, as the majority often lack the same perspective as those actually on the receiving end of these decisions.

"If the perspectives of vulnerable groups are not actively taken into account, there is a risk of wrong decisions with real consequences—such as unjustified benefit withdrawals or false accusations," the researchers explain.

The answer is to do a much better job of co-creating solutions so that recipients of decisions are involved in the systems that make those decisions. As soon as people feel powerless, the trust in both the system and the institution making the decisions is stripped away. With governments around the world increasingly adopting an "AI-first" mindset, it's perhaps something for them to keep in mind.

Unlock more exclusive Cybernews content on YouTube.