Killer Robots
Chris, Nick and Fraser pass the time in hiding from a T-1000 by discussing automation of weaponry.
For more information on Aleph Insights visit our website https://alephinsights.com or to get in touch about our podcast email podcast@alephinsights.com
Transcript
Hello and welcome to the Cognitive Engineering Podcast produced by Tell Me Studios for Aleph Insights. In this series of podcasts we take a look at interesting topics and discuss what we think they tell us about analysis and decision making. I'm Fraser McGruer and I'm here with Chris Wragg and Nick Hare of Aleph Insights and this week we're discussing killer robots. So Chris, what are killer robots and why are we talking about it?
Speaker B:Well, the reason we're talking about it at the moment is because Elon Musk and a group of other technologists have written an open letter to the UN calling for action to be taken with regards to the restriction of autonomous weapons and the use of autonomous weapons. So from the point of view, this has been dubbed by the media as killer robots, but in terms of the technology we're talking about, we're talking about weaponry which has a degree through technology, through artificial intelligence, has a degree of autonomy, can decide how it deploys itself without human interference and that is the area of concern at the moment.
Speaker A:What has been the tipping point?
Speaker B:Well, so I think the tipping point really is that at the moment we have got weapons which we're currently having humans operate, but which are pretty close to and in many cases can currently be entirely autonomous. So we are inserting a human into the kill chain of that weapon. For example, if you think about what are commonly referred to as drones or autonomous aerial vehicles, then these things are, you know, maybe piloted by a human or the weapons on them may be fired by a human, by a human pressing a button or making a decision about the deployment of those weapons and that is the case at the moment. But there is no reason why the technologies that pilot these things around while they're just doing something boring like watching what's going on, there's no reason why those technologies couldn't be applied to the selection of targets and the prosecution of targets and that's why we're talking about it because it's now possible with current technologies and it is certainly going to be possible with future technologies.
Speaker A:And just with this petition to the UN, I presume that ultimately the end user is governments and their militaries. That's who's being petitioned in effect to change the rules or to apply, to abide by rules to be set by the UN?
Speaker B:Yes, so the UN to arbitrate in the way that you might have an international treaty like the Geneva Convention or the Non-Proliferation Treaty for Nuclear Weapons or various others. The idea, my guess would be, is to have some sort of international set of rules to govern the development and the utilization of these technologies. Now the problem is, from a government point of view, once a government has made something, so a government might make technology and then say we're only going to use it in this way, but once that technology is available and copied and so on, then non-signatories and terrorist groups and malign actors can get hold of that technology and use it however they want. So it's as much about the development of the technology as it is about how we use it, I guess.
Speaker C: y. And that was introduced in: Speaker A:But sorry to interrupt Chris, but for example, with the system that you mentioned on the ship, one thing there is that's a defensive system and with mines, they're sort of fairly localized. Okay. So I'm guessing that although we use the example of drones, or you mentioned the example of drones, I'm assuming there are all sorts of systems that are now using artificial intelligence. And I'm guessing that the concern is more about the scale and the impact that these weapons can have now. Yeah, but that's true.
Speaker C:That's true of things like nuclear weapons. Now you could say, well, yes, fair enough. We've got controls on nuclear weapons. But I mean, that scale is irrelevant to the autonomy. That's the point. I mean, if you've got a really big bomb and you drop it on a city, there's no sort of autonomous decision making element there. But in terms of scale and in terms of its ability to discriminate, it's very, very low. But we don't have a problem with dropping bombs. That's part of conventional warfare.
Speaker A:Sorry, I know you want to cut us. But there is a difference because it's when you combine these factors of scale and autonomy, that seems to be what the issue might be. Yeah, but why?
Speaker C:I mean, that's better. See, what I'm saying is that I can't see why that's in any way worse or scarier. I mean, when someone drops a bomb on an area where there are potentially civilians, they are not choosing which people are going to die, right? They do not have that choice. They choose to drop the bomb. But the consequences are out of their control. I don't see how that's different to launching a drone from a ship and then letting it go and kill people. Why? I can't see that there's a relevant moral distinction. I think this is topical simply because of the technology, not because it's introduced some new moral problem. That's all. I think this is covered by existing moral paradigms.
Speaker B:And I think, you know, going even further than that, there are already hundreds of thousands of autonomous agents in control of weapons being utilized by Western governments with only fairly loose sets of controls over their sort of conduct and how they operate. And those are human soldiers who, you know, who are not immune from going on killing sprees, you know, through history, you know, the sack of Troy. But more laterally, if you look at, you know, the killings in Haditha in Iraq where U.S. Marines killed 24 unarmed civilians as documented, if you look at the Marine A case that we've had in our media of a Marine who shot a wounded Taliban soldier. And as understandable as those actions may be, when you look at the stresses those people are under, the emotions and, you know, factors like PTSD and fatigue and all of those kinds of things. So understandable as that is, it's still not the commander's intent when they deployed that weapon, i.e. they sent those people out to do their job to kill innocent civilians or wounded combatants. And so actually, in many ways, the question is, and we talked about, you know, nuclear weapons, do we feel safer with humans in control? If you look at, you know, the greatest existential threat arguably we face in terms of the U.S. nuclear arsenal and the potential for its deployment. At the moment, you have, you know, one individual potentially in control of that and subject to all the whims of human irrationality and miscalculation and cognitive frailty and all of these factors. You know, for me, the question is not how dangerous is it to deploy AI in control of weapons, but also how dangerous is it not to do so? Currently, we are exposed to a wide range of risks, which we know about, and some of which we know about, at least. But, you know, are we actually, by cutting off this technology, are we actually making ourselves more at risk?
Speaker A:But this reminds me of a previous podcast that we did with autonomous vehicles, where when they're self-driving, when we reach that stage of development, it looks like inevitably you will have less accidents, roads will be safer, road deaths will decline. And yet people still feel uncomfortable with a robot making a decision, which at some point would result in someone's death, even if it's still safer. And so just to finish off, and so I think behind this letter being published is just that people and institutions just feel uncomfortable with artificial intelligence making decisions, even if those decisions are usually better or always better than humans.
Speaker C:I think there is something here which might make an interesting distinction, whether or not it's morally relevant, I don't know, but the fact that you can clearly identify a chain of decisions when you have humans, where you have reasons, each person has a certain reason, whether it's good or bad. And if we think that the outcome has been bad, we can investigate and at least say that everyone intended to do the thing that they did. Yeah, if it turned out they didn't, then that's fine, maybe they're not culpable. But I mean, when the person who launches the drone intended to launch the drone, the people who analyze the imagery intended to produce a correct assessment of what they were looking at, the person who pulled the trigger intended to fire it and so on. Perhaps the problem is that when you have sufficiently advanced autonomous weapons, is that those reasons are opaque. We no longer have transparency of the rationale. And the rationale might be just too complex to explain or require backwards engineering to understand. In the same way that, you know, I think one of our very early podcasts where we talked about AlphaGo, the autonomous, sorry, the artificially intelligent Go playing software, which beat the best player in the world. The humans looking at what it was doing just couldn't understand why it was doing the things it was doing. It's just that they were effective, they worked. And, you know, we might find the same thing with autonomous weapons. They might be choosing to, you know, to kill someone for reasons that we don't actually fully understand, but which, you know, actually makes sense, if you like, in a kind of information terms.
Speaker B:And I think this gets to the crux of why we're concerned. If you look at weapons that we tend to prohibit the use of through international treaties, they are ones which are indiscriminate. You know, the thrust of the Geneva Convention is about protecting non-combatants or retired combatants through, you know, surrendering or injury. And so when we ban chemical weapons, it's because a chlorine gas cloud can't decide whether it's an enemy soldier or whether or not it's, you know, a child. It's the same with landmines and cluster munitions. You know, these things are not able to discern, whereas we decide that a human is able to discern and make some judgment and minimize the risk of killing, you know, non-lawful sort of targets. And I think the fear of autonomous weapons is that once deployed, we lose that human ability to rein them in at the point at which, oh no, that's, you know, that's not an enemy combatant, that's a child. But I mean, we face that problem with bombs. Once they fall out of the aeroplane, there's nothing you can do about it. And in fact, even more than that, we already have autonomy within our bombs. So where is a weapon deployed? Because a pilot presses the button and then the weapon goes off. But that is a guided weapon and it makes decisions as it goes down using, you know, artificial intelligence about whether to go in through this window or to adjust for this gust of wind or whatever. And it makes it more precise, not less, you know, less precise. It reduces collateral damage. And the same thing can be said of, say, you know, a drone pilot. A drone pilot might be sat there looking at a screen for hours, getting tired, blinking, you know, can they make that call? Is that a rifle that man is holding or is it a hoe that he's got over his shoulder? They have to make those calls. And I would argue it is our duty to push technology to make those decisions better than us. It won't get tired. It won't blink. And, you know, we can actually make our weapons more discriminating.
Speaker C:Yeah, I think so. I think here, like making the world as hard as possible for the sort of let's ban killer robots camp, right? Let's assume that our artificial intelligence is such that it's essentially perfect, that it poses no risks to civilians or at least that it is capable of targeting who we want it to target and no one else. Let's put it that way. So that takes out any concerns about whether the technology actually works. Let's assume it does. And then say, well, are there still concerns there? And I think there might be an additional issue, which is to do with the ease with which software can be replicated. And this is, you know, as it happens, right? We've been quite lucky in that really destructive weapons are really expensive to build traditionally, you know, that you have a nuclear nuclear weapons, thankfully, turn out to be incredibly complex. It's not very easy to have a nuclear weapons program. Now, it might have turned out otherwise. It might have turned out that actually, you know, you just you can you can knock one together in your shed. Well, if that had been the case, you know, we probably wouldn't be sitting here by now. And I think the problem we might have with autonomous weapons is that they don't, you know, they they it's quite easy to, you know, obtain a drone and stick software in it. That's not, you know, these things are kind of just putting together two things that are relatively easy to obtain. And I think that the concern is that that just makes terrorists, for example, much more effective. You know, if they have a bunch of killer micro drones, which they're able to get off eBay and tell them to go and kill, you know, a thousand innocent civilians in a in a in a in a, you know, football stadium, it's just very easy for them to do. So I think that might be the concern. It's just that it actually is just making it too easy because of the cost of replicating software, which is very low.
Speaker A:We need to wind up shortly. But do you think that's what the people in the letter are thinking about? Or do you think it's a more general?
Speaker B:I think there's a second fear, which is that through some wonky utility function in the in the AI controlling the weapons, you know, let's say we we program them to reduce human suffering as much as possible or end human suffering. And they they somehow determined that the best way to do that is to destroy all of humanity. You know, obviously, well, maybe that that isn't a far fetched scenario, but but it's that kind of fear that if you if you increasingly gift on large scales control over over weapons of mass destruction to autonomous agents at some point, they might decide to use them not in our interest. And we may lose lose control of that.
Speaker C:But that is I mean, that's this is no longer an issue of autonomous weapons per se. But the dangers of having a general artificial intelligence, which and and having complex or opaque values, right. So and this is so this this sort of AI, you know, global destruction problem where you give it an objective, which seems superficially benign. And and as a result, you know, in order to achieve that objective, it sort of effectively destroys the world. So, for example, you know, the examples like the, you know, the the paper clip machine where you say, well, I want I want you to, you know, to make me I love paper clips, make me lots of paper clips or work out to collect, you know, as many stamps as possible. And the machine sort of effectively says, well, the best way of doing this is to turn the entire world into a massive stamp producing factory. And because it's a general artificial intelligence, it knows how to solve that problem. And if it can't solve that problem, it knows that it can create new, a more advanced version of itself. And that will help solve the problem. And it's just doing what you told it to do. But on the way, because we haven't specified its objectives properly, it happens to destroy the world. And I think that's the that's now that's you don't need killer robots for that. You don't need things that are told to be weapons. That scenario is a kind of something that artificial intelligence theorists are worried about without without needing to. You don't need guns to kill people. Yeah, exactly. So so I think there's that, you know, that's if you like, maybe we could say, well, autonomous weapons gives us an excuse to raise this as an issue because it seems obviously applicable. But actually, the real threat is is more to do with the the wider problem of what do you do with the general artificial intelligence to stop it destroying the world?
Speaker A:OK, so that I mean, just to draw this to a conclusion, the sort of getting close to the Terminator sort of scenario. Right. That's one of the issues that kicks off that whole sort of series of films.
Speaker C:Yeah. Remind me what they why? Why did they? Well, I think it's so. Why does Skynet decide to destroy humans? Is it because humans are the source of all evil?
Speaker A:Yeah, which is being a more efficient world or something, or they saw humans as a threat. Yeah. So go and destroy what's the biggest threat.
Speaker B:And the robots suddenly think, well, I think humans, which is not which is not a million miles away from if if you program these things with protect protect civilian life and noncombatants life. And then it sees one of your soldiers wandering along and it thinks, well, there's a reasonable chance they might end up shooting those civilians. And, you know, it kills it kills your own soldiers. So so that that type of scenario is not plausible. It's plausible. Yes.
Speaker C:Yeah. I mean, if because so, you know, if you've got a suitably general artificial intelligence and, you know, you decide that you're going to install it in a sentry gun, which is ostensibly a defensive thing, you know, if it's suitably general, the sentry gun will go well to achieve this objective of defending this base. The best thing is to not be a sentry gun anymore. The best thing is to turn myself into a, you know, a massive stockpile of nuclear weapons and destroy humanity. That'll stop people coming near the base. And, you know, a general artificial intelligence is capable of solving those sorts of problems and working out how to get there. That's that's the concern. The fact that it happens to start as a sentry gun is irrelevant. If it's a general artificial intelligence, doesn't really matter what you call it to begin with. You know, it's capable of solving real world problems.
Speaker A:Well, to wind up on a happy note, this sort of makes me think of, as always, there tends to be an analogy in Star Trek. And in Star Trek The Next Generation, there's a really nice episode where Data, who's a robot, gets given control of a starship. And his crew are really uneasy about this because he's a robot. And they're not robots, right? Yeah. And they're not robots. And they and at one point he makes what's a difficult decision. And the crew sort of almost mutinies because they're just saying, oh, you're just some you're just a cold hearted robot. And we shouldn't trust you. And, you know, we don't want you to be our captain. But actually, somehow, I can't remember how Data pulls it all out the bag. Yeah, of course he does. He's just made a brilliant decision, which everyone, you know, applauds.
Speaker C:Yeah, that's the this is called Hollywood rationality, where you, you know, for some reason, if you're rational in a film, it means that you disregard the existence of emotions. But of course, you know, a truly artificial, a truly intelligent robot would acknowledge the existence of emotions and take them into account in its decision making. But, you know, it's a bit boring.
Speaker B:And who would you rather had their finger on the button? Donald Trump or Data? Well said.
Speaker A:And let's leave that question out there. Okay, gentlemen, thank you very much. I'm Fraser McGruer. We've been here with Nick Hare and Chris Wragg of Aleph Insights. This has been the Cognitive Engineering podcast. Thank you for listening. And until next time, goodbye.
