Lucy Suchman: human oversight of autonomous weapons doesn’t mean much when the aim is to maximize destruction

Lucy Suchman, an internationally renowned expert on autonomous weapon systems, says they cannot distinguish between civilians and combatants. Therefore, they can never be made lawful.

Suchman is a professor emerita at Lancaster University and a member of an international committee of experts in robotics technology, artificial intelligence (ICRAC).

Cybernews spoke to Suchman amid the ongoing standoff between Anthropic and the Pentagon, after the company refused to remove its guardrails prohibiting its tools from being used for domestic mass surveillance and power autonomous weapon systems.

Although Anthropic was labeled a supply chain risk and banned from being used by government agencies, its chatbot Claude was allegedly used in the US attack on Iran.

Systems to “accelerate and maximize destruction”

When we talk about autonomous or semi-autonomous weapons, we hear terms like “human in the loop” or “appropriate level of human judgment.” What does it mean in practice, and is there a consensus on it?

I'm concerned that, at the moment, it doesn't mean much in practice. The premise of the narrative is that these systems make recommendations based on intelligence, which is then reviewed.

The clearest evidence and most detailed reports that we have now come from these systems being used in Gaza. What we know is that, first of all, these systems are used to accelerate and amplify the number of targets. So it’s really about maximizing destruction.

The whole premise that humans will be doing the evaluation becomes increasingly less credible. We've heard reports that it is becoming a kind of pro forma approval, with the recommendations serving as the authoritative input to the machinery of warfighting.

As long as the basic aims of these systems are speed, acceleration, and maximizing destruction, then the whole reference to human judgment becomes increasingly meaningless.

Lucy Suchman

As long as the basic aims of these systems are speed, acceleration, and maximizing destruction, then the whole reference to human judgment becomes increasingly meaningless.

On the one hand, militaries assert that they are maintaining those levels of human control. On the other hand, we have increasing evidence that it is impossible to exercise judgment within the ways in which these systems are being used.

We need to spend more time on the question of the intelligence on which these targeting recommendations are based.

We know from the case of Gaza that there’s mass surveillance, producing data profiles of people, places, or objects. Targeting recommendations are based entirely on the validity of those profiles.

When we get into situations of being guilty by association, affiliation, or proximity, the claims of any validity or precision in these systems that designate what constitutes a legitimate target are becoming completely unsustainable.

Anthropic may not object to mass surveillance elsewhere

Anthropic was praised for drawing two red lines for how its AI tools can be used. Are they enough to ensure the ethical use of AI in the military? Does the focus on domestic mass surveillance mean that they are okay with surveillance in other countries?

The idea that a company would take its own guidelines seriously enough to take a stand like Dario Amodei has taken is really important, and I don't want to dismiss that.

On the other hand, we see that weapon systems that are actually in use – Anthropic’s chatbot Claude is claimed to be built into this – they depend on mass surveillance.

The red line of domestic mass surveillance is important, but it doesn’t in any way rule out its being done elsewhere. In occupied territories, mass surveillance becomes the basis for these targeting recommendations, with all the problems that I was talking about earlier.

On the one hand, we have the qualification on restrictions on mass surveillance. On the other hand, we have this restriction on fully automated weapon systems. The military can then say it still has human oversight and control. So there's a lot of wiggle room there.

What’s interesting about the US military’s response is the unfeasibility of removing Anthropic from the supply chain, and the rapidity with which OpenAI has stepped in to fill the gap.

It's enabling us to see some of the relations and the machinery that this larger system of warfighting is based on. ​

I think it was an important stand, but if we look closely, it doesn’t do much to mitigate the problems we’re up against here.

Anthropic reportedly competed in the Pentagon’s prize challenge for creating technology for what is described as voice-controlled autonomous drone swarms. Could you explain what drone swarms are and why they are dangerous?

Drones that were used during the so-called “War on Terror” were very large and expensive, operated mainly by the United States. What we’re seeing now is the advent of much smaller drones for surveillance, which are also increasingly weaponized.

I think this is gonna be a huge problem, because you can have multiple weaponized drones operating where it’s increasingly difficult to know who’s actually operating them to know whether they’re weaponized.

This kind of terror is going to really amplify the dangers and the uncertainties. The project of drone swarms is again the idea of amplifying the terror that you can deploy with much smaller drones by enabling them to be networked with each other.

Accelerating and amplifying destruction further closes down the possibilities for deceleration, de-escalation, and opening up spaces for diplomacy as well as accountability in terms of who is actually responsible.

Tech workers feel too precarious to protest

Back in 2018, Google employees publicly protested Project Maven, and this time, the backlash from employees of companies like OpenAI appears much weaker. What happened in the tech industry over the years?

I suspect there may be more unhappiness among employees than we’ve been hearing about. The fact that we haven’t been hearing about it, I think, reflects the degree to which tech employees feel more precarious than they used to.

We’ve seen the extent to which employee protests have been met with firings at Google and other places. So there’s been more of a crackdown on expressions of unhappiness on the part of employees.

I suspect the unhappiness has gone more underground and is happening offline. Therefore, it has a lesser effect on public consciousness or on any action.

It's safe to assume that civilians in the conflict zones are most vulnerable to these autonomous weapons. What would you say to Americans who are far away from these zones and think it doesn’t really affect them?

The crucial distinction between civilians and combatants is another area of frameworks that is being eroded as we increasingly don’t have traditional military operations in battle spaces with combatants.

There have always been attacks on civilian spaces like cities, but now it seems to be just the way of fighting. As soon as you start having attacks on densely populated areas, determining who is a combatant and who isn’t becomes pretty impossible.

There have always been attacks on civilian spaces like cities, but now it seems to be just the way of fighting. As soon as you start having attacks on densely populated areas, determining who is a combatant and who isn’t becomes pretty impossible.

I have very little confidence that in the current US operations on Tehran, which are undoubtedly being accelerated through these technologies, there are the traditional principles of international humanitarian law.

The principles of distinguishing between combatants and civilians and requiring accountability in terms of proportionality.

You cannot count on the fact that you have not enlisted in the military or that you're not uniformed, because these surveillance systems work by determining guilt through association. We all have to recognize the dangers it poses to us.

The United States has been incredibly protected from this kind of threat, with the exception of Pearl Harbor.

I think Americans really have no sense of what it's like to be in the kind of danger that people experience in cities that the Israel Defense Forces and the US military are now bombing.

Autonomous weapons are inherently unlawful

Is it possible to have autonomous weapons that adhere to international humanitarian law (IHL)?

The principle of distinction is rule number one of international humanitarian law. For an autonomous weapons system to adhere to IHL, it would have to be able to distinguish between civilians and combatants.

There is no way to encode the specifications for that distinction. Therefore, I think autonomous weapons systems are inherently unlawful and could never be made lawful.

A common argument for accelerating AI development, whether for superintelligence or weaponry, is that the US must do so before its adversaries do. What's your take on that?

Anything that accelerates the scale and speed of weapons development makes us less secure on a planetary basis.

These narratives of competition and the imperative to do it first serve the interests of those who, for political and economic reasons, are invested in the expansion of the arms trade, one of the biggest problems we face.

Lucy Suchman

These narratives of competition and the imperative to do it first serve the interests of those who, for political and economic reasons, are invested in the expansion of the arms trade, one of the biggest problems we face.

There’s never a discussion of alternative approaches to these competitions. If we could even begin to imagine that the kinds of resources that are invested in militarism would be invested in diplomacy.

What are the prospects of having an international framework for prohibiting or regulating autonomous weapons?

My colleagues and I have been fighting the good fight at the United Nations (UN) for the last decade, and it’s very difficult. There are always countries within the UN that can block progress on these issues.

The focus has recently been on the General Assembly, and a large number of countries are in favor of some form of binding prohibition on lethal autonomous weapons.

Even though it feels like the prospects aren’t great at the moment, given how legal accountability has been so undermined, it’s crucial that we double down on mobilizing resources for these legal frameworks rather than abandoning them.

The majority’s view around the globe is that, as with nuclear weapons, autonomous weapon systems need to be banned.

Unlock more exclusive Cybernews content on YouTube.