To Err is Human

Speaker A: 00:00:00

Hello and welcome to the Cognitive Engineering podcast produced by me, Fraser McGruer for Aleph Insights. In this series of podcasts we take a look at interesting topics and discuss what we think they tell us about analysis and decision making. I'm here with Chris Wragg, Peter Coghill and Nick Hare of Aleph Insights and this week we're discussing the YAM cryptocurrency bug. Peter, what is the YAM cryptocurrency bug?

Speaker B: 00:00:28

Yes, so in August 11th a new cryptocurrency was launched. It was quite an interesting experiment. It was bringing together lots of different emergent new technologies around cryptocurrencies that people have been trying more piecemeal but this has brought together loads of different things into one approach. It seemed like it was getting lots of traction, there was plenty of investors in the technology and setting up teams to support and build it and lots of trades going on. So there were lots of people trading other cryptocurrencies and real hard currencies into this thing and it was starting to actually trade like a currency. So it all looked good but within 48 hours, I've yet to pin down the exact timings but within 48 hours a bug was discovered in the source code and it was discovered as a result of it actually starting to go off the rails that basically undermined the entire thing in quite a dramatic way. So a lot of software bugs might cause an inefficiency or something that's recoverable but this bug went right to the heart of the entire governance model of the currency. So I don't quite understand in detail how it works but basically the currency itself scales the amount of units of that currency available in response to what's going on in the market and part of what it does in doing that is it accumulates a sort of cash reserve. So I think the analogy would be like a gold bullion reserve that can be used to sort of help stabilize the price and various other things. But there's a bug in the code that caused it to accumulate more and more and more currency in this reserve basically causing a massive liquidity problem meaning there wasn't any currency left to trade bringing the whole thing sort of essentially crashing down. But the numbers are quite staggering so at one point it was estimated to be worth sort of the market capitalization was nearly half a billion pounds or yeah nearly half a billion pounds which is quite staggering in two days worth of trading which was wiped out pretty much within hours of this sort of thing

Speaker A: 00:02:53

of this thing happening. Hold on I've just remembered that fortunately we're actually not really talking about cryptocurrencies in this podcast right? We are not talking about it.

Speaker B: 00:03:02

What we're talking about though is this so what sort of the way I kind of looked at this was this seeming mistake and there's a great article in the register which actually tells you what the software bug is and it's about four or five characters that were missed off on one particular line of code. So that's all it was. The actual mistake is minuscule in terms of actual information value but has had a huge effect. So what I was interested in is is there any natural correlation between the kind of size of mistake inverted commas and the size of the impact and is that a meaningful thing to sort of even talk about? Chris? Yeah well I think this is really

Speaker C: 00:03:44

interesting because as Peter said the error itself as an input error is really quite small. I think it's dot div brackets base was what they missed like 10 characters of text effectively and but what it missed was the fact that that was a multiplying factor within the equation that was being used. So they effectively forgot to multiply by something and so you've got this tiny input error and I think what it illustrates is the difference between input and output errors you know but you've got this very large and consequential output error which is you know messing up the entire system and happening really quickly and I think that you know is a great example of two things illustrating you know the difference between an input and an output error and also the consequence of the error that you make. And if you think about you know I mean to kind of illustrate this if you think about you know the difference between a million plus two and a million plus three right if you've got those two numbers wrong it would be relatively minor right but if you used to make the same kind of error when you're using two or three as an exponent of a million then you've got a much bigger output error and I think it highlights on an interesting thing particularly when it comes to computer code that what you actually do wrong can be quite small but the impact of it

Speaker D: 00:05:19

can be enormous yeah I think and I think what we want to get into is can we mean it can we say small or big who's to say that you know lines of code are a small error like I mean does the universe know that that's a small error it's like there's a big difference to us between the words yes and no but does the you know the universe doesn't see that as different particularly so you know is it can we meaningfully say this is a small error that's that's the question I think but worth saying that this is not unusual at all that actually a lot of disasters you can boil down to a very small decision you only have to think of recent days to think of the gender reveal party which set Eldorado on fire what the gender reveal party they had some fireworks which I think were a certain color because it was a boy or a girl and it then started this fire which has now destroyed 22,000 acres of uh of I think California oh is that what that was and uh yeah and killed a firefighter 12 people injured um and I I would venture to say that whoever the uh gentleman in Wuhan who thought he'd have a bat for supper was guilty of the same thing you know um it was probably some very small decision which leads to a giant outcome and and so intuitively we understand that um I I tried to and and so I sort of thought well how what kinds of errors are we talking about here um there are lots of lists online of different kinds of mistakes you know the biggest historical mistakes and and they are not um very scientific obviously uh they're generally not trying to quantify big they're just relying on an intuitive understanding of what that is um but I've sort of classified them the things that appear appear in this list and again I think I think that some more underlying model is probably better explaining it than this but where'd you get this list from oh it just I looked at various lists of the world's biggest mistakes right so is it clickbait yeah it's all clickbait articles that's all you're going to get you won't believe this yeah yeah number number 17 will astound you uh so they and they generally till the end I think they're almost all five they fall into one of five categories and they are so first of all failing to take an opportunity that was offered and that's the you know the people who turned down the Beatles in favor of the tremolos the person who turned down JK Rowling not the guy who didn't kill Hitler in the first world war and could have done yeah um you know these are all things which you know retrospect you can say well I wish that person had done one thing or done a different thing but but actually you know the Beatles still exist it's not like anything's been genuinely destroyed this is just you know individuals who have failed to take an opportunity that was offered yeah um then the loss or destruction of valuable things so this is the guy who lost that bitcoin you know which is now worth a billion pounds that he threw it into a into just dumped it with his old computer um the burning of the library of alexandria um nasa taping over the moon landing tapes there's sort of things that are that are lost or destroyed because of um you know something stupid that someone did um good old-fashioned fat fingers I can't think of any other way of doing this but this is people who just you know could like physically cock up something and um you know make a kind of physical error of some kind and it leads to disaster and that's there was a Japanese trader who meant to sell one share and accidentally sold 610,000 shares um there was the you know the people who accidentally hit reply all uh there's quite a few stories of that you know people people bringing down their company because they share something by mistake um the uh I would say also the Exxon Valdez captain you know for example he was he was drunk and obviously couldn't couldn't captain his ship properly um underestimation of consequences so that's category four this is the person who beheaded Genghis Khan's diplomat or you know the decision to invade Russia or whatever so it's kind of just failing to realize that something is going to be much harder or more disastrous than it was so you go ahead with the course of action that turns out to have terrible consequences and finally and this is the mysterious slightly mysterious category but what I've called distributed errors these are where actually it's very hard to find an individual who made a mistake but it but it's more that you know the the machine has failed to account for some kind of uncertainty which has disappeared somewhere in the system and I would say things like the um you know those kinds of where the French uh ordered 15 billion pounds worth of trains that it turned out didn't actually fit on the rails um the Chernobyl three mile island kind of things where that you know actually every every single person you kind of understand why what they did what they did but collectively it all added up to a disastrous set of circumstances so that's anyway I thought you know I'd share my taxonomy of errors yeah um I think of those you know the ones which you could say well these are definitely like the kinds of mistake that I think we're talking about you I mean fat fingers falls into that it's like okay this is a kind of physical mistake you've made you've physically done something different to what you wanted to do um and uh and the sort of distributed errors I think would would be the yeah I think the last one is is a very interesting whereas underestimation of consequences feels more straightforward to me it's more like well you've just got this you thought this would be easy and it was hard yeah it's not really quite the same thing as a kind of proper good old-fashioned mistake

Speaker B: 00:10:36

yeah quite Peter yeah so to try to sort of categorize this particular the the GAM cryptocurrency problem um probably yeah it's it feel it on on the face of it looks like a fat fingers thing it looks like somebody wrote some code and they didn't write it correctly um however I think it sits more squarely within the distributed uh error because we we as a race and as a sort of collaborative race have developed mechanisms for trying to prevent and correct these kinds of errors and in the distributed model this is uh a classic example of how you address this is you have code reviews and you have testing and you have money you spend money ahead of time before launch to to to try and pick holes and try to test it try to stress test it and try and break it and see where it fails and then correct it that's in this case some of the some of the punditries suggested that a lot of that didn't happen and this was uh this happens a lot in uh in in financial uh sort of speculative financial uh products that there's a big first mover advantage particularly in a currency because you capture a lot of the a lot of the investment will um uh so there's a so this thing was rushed to market with with all these caveats were public there was no nobody was making any pretense that it was like perfect and was going to work in fact the github page that where the source code is published publicly states there's going to be bugs here we've only done limited testing etc etc um doesn't that doesn't just doesn't it doesn't uh doesn't uh prevent lots of people willing to be willing to invest in it possibly naively but had there been more testing and verification you know maybe even just a few tens of thousand pounds worth of code review and more more unit testing it would have been picked up highly very high chance it would have been picked up and that seems like a really worthwhile

Speaker A: 00:12:31

investment compared to the overall loss that it suffered yeah um look i don't know because i want to bring you in chris but i don't know if this is what you want to talk about i don't know but i think it's quite interesting this last one where no there's nothing wrong it seems and everything seems to make sense and yet it results in a mistake yeah this what would you call it the distribution distributed areas yeah distributed areas i mean and i don't know if this is analogous but i remember a long time ago i used to play a lot of chess and but i was never any good at chess um and i was never very good at sort of seeing however many you know extrapolating i was never very good at that but also even worse than that was uh concentration and blunders so i used to be a bit of a blunder player um but anyway there's one guy he used to play with all the time he always used to beat me and um you know like any good chess players immediately afterwards we would always have a post analysis and say okay this is the moment where it went wrong right um but i remember it started getting to the stage where in a lot of the games where we couldn't really see the error there wasn't really an error and it was yeah there was no mistake in maybe not even a weak move and yet somehow uh maybe that's a maybe that's a false analysis no i think it reminds

Speaker D: 00:13:41

me of poker you know when when you look at the winnings of good poker players is they the difference between a good and a bad poker player will not be visible in any small run of hands you know it comes down to winning um you know if you think of an average pot being you know sort of maybe 10 20 pounds to say in a game a really good poker player is going to on average be winning perhaps one one pound you know uh or less per hand i want to bring chris in yeah well i think i think

Speaker C: 00:14:11

we're sort of hedging towards a kind of theory for what you know what creates the worst kind of errors and one of those things is about a difficulty in seeing the error in the first place because it is um in a complex system right so when you're talking about a you know a system that's based on loads of people it's very difficult to analyze all of those people and check they're all doing their job right but you can also get that in uh engineering systems like um you know the challenger disaster was effectively the fault of the the o-rings like but basically washers you know that weren't accounted for a particular temperature and had a catastrophic effect but and so you think well well that was a simple error why didn't they do that but of course you think how many components there are on a space shuttle and you know your your requirements to analyze those under all those different conditions it's very difficult to see the error if you think about a huge body of code yes there are review processes but it might be very difficult to wage yourself through that so i think i think one component of uh you know making errors more likely is a more complex system that you're engaging with like a chess game right where each move builds to each other you know the next move and it's not one place oh there it was it's it's a you know it's an interaction of things and i think the other thing that um that uh builds to significant errors are um you know the the the under estimation of the of the consequences so so normally like with a space shuttle you are you know we know it's high risk and so you check everything so you know most of them were were um had a good you know good safety record and so on but um but it's where you're not sure you're not taking that degree of of rigor about your your checking and you can't really see things um because it's a complex system i think that's where you're most at risk of making kind of catastrophic type type errors yeah yeah and they have that i mean the responses are

Speaker D: 00:16:17

always uh you know it's always shutting the door after the whole horse has bolted in a sense in that you know all of these kinds of engineering errors lead to a you know a kind of solution which involves having a new rule that we now implement which in you know they always use that phrase of that all of these health and safety rules are written in blood because you you know you've got you you've now got 200 things you have to check um which all seem totally irrelevant and pointless but of course they at some point once upon a time no one checked it

Speaker C: 00:16:49

and that's where it went wrong and i think i think that hits on a really important issue and there is the phrase to err on the side of caution right you know so you deliberately under you know you deliberately um estimate something incorrectly on the side of you know whatever the safety concern is and i think we're seeing that at the moment with you know uh the our current body of government scientists and the estimates they're making about you know coronavirus and what what could go wrong because of what we've seen previously you know they are erring on the side of caution within those forecasts um and you know we're we're potentially going to pay the consequence

Speaker A: 00:17:32

accordingly yeah um yeah i never thought about the phrase in that in that sense and you were absolutely right aren't you um so look um we've got a few minutes to go before we wind up um where do

Speaker D: 00:17:44

we want to go with this i do have i do have one thought and this is more of a kind of slightly philosophical speculation about the about the nature of you know errors in general and this issue of you know the size of the the size of the mistake versus the size of the consequence and it goes back to something i mean chris was saying you know we we built a lot of systems that are designed to stop this kind of thing happening but i think i would say that the reason we need to do that is that um actually we've designed as humans we've designed systems which have huge multipliers in them which don't exist in the physical world so you know as such like like if you want to blow up the moon you generally need another moon-sized object or you could have a lot of energy somewhere you know if you want to but if you want to blow up a rock it's you need much less energy you need much less input right to that system generally actions and consequences are more or less the same order of magnitude in the physical world but in the in the social world you know we've designed structures to magnify the power of individuals so that you know a a leader can sign a piece of paper which leads to you know 100 million people being killed and um you know so in the human world we created systems that give us the ability to turn a small action into a big consequence um so in a sense we've had to design equally you know rigorous ways of stopping that happening we we need we need those those systems because the the social world has these has these multipliers and i think you know if you think about the example one which always crops up in these lists of you know where um of archduke ferdinand's driver taking the wrong turn and you know and gavrilo princip happens to be walking down that road and shoots him at the first world war starts well you know one guy shooting another guy the only reason that turns into a world war is because we let it we've decided that that's how it works you know it's really just one guy shooting another guy that wouldn't be a story if it happened in chicago but because of the systems that we've designed that turn an archduke into a thing that can cause a world war you know we have to then we have to worry about that yeah interesting point yeah no i like it

Speaker A: 00:19:53

um peter chris anything you want to finish off on before i ask a question yeah i i was just going

Speaker C: 00:20:02

to say i think there's a there's an element of um error which i think nick has picked up on there which is you know um the because there are there are um things within the the natural world that cascade you know out of control the whole the whole um uh sort of metaphor of the butterfly's wings and you know the the cyclone you know across across the planet and so on so so it does exist there but i think where error comes in is it's very difficult to have error without humans right because the natural world doesn't make mistakes in the same in the same way right and it's not trying to achieve something it's not trying to achieve something it doesn't have an an objective as such and um so so yes and and you know error is you know even though you talk about system errors or equipment error ultimately they all boil down to a person somewhere using the wrong bit of equipment or you know following the wrong process or whatever it is so i think there's something fundamental about error which requires a human to be there and not necessarily to be to be culpable but to be involved in the in the process yeah yeah it's like that famous phrase

Speaker A: 00:21:12

to err is human but completely fuck things up you need a computer yeah right but um anyway um look i don't know if this is a good question or not um but um you know my rather pedestrian manner that i normally that i usually do can you think of an example in your own life where you're aware of a decision that you made that led to a huge um mistake or terrible consequences for you or

Speaker D: 00:21:37

someone else or humanity um i got a near miss actually yeah go on so when i was working in defense intelligence i i issued i issued a uh an intelligence report which was very unusually lowly classified i was following kind of the principles of good of kind of good intelligence production which is to try and classify things accurately you know not just stick top secret on everything yeah to try and maximize the number of people who can read it yeah and uh and it got sent um through uh it was it was classified it was unclassified enough that it could be shared to sort of nato various other people and about an hour after it went out a particularly diligent um analyst phoned me up and said i've just noticed on the back page it's got a load of uh it's got the distribution list and this distribution list was absolutely full of of of like the names of people and designators names of jobs that just were extremely top secret so in a total panic uh you know i rang up the the people who were doing all the kind of the distribution through the computer system and thank god you know they they were they were sort of had just finished their lunch break and had not pressed send and uh you know so we were able to we were able to intercept this while they were still in a top secret environment um but that was that could

Speaker A: 00:23:00

have been catastrophic so yeah i can just imagine also your physical reaction that makes that

Speaker D: 00:23:06

horrible feeling that accidental kim philby yeah the coiling of the guts you know no thank god for this guy he was the kind of person who read everything in ridiculous detail and um you know i i which normally would be really annoying but actually yeah you do you need pedants you need

Speaker B: 00:23:21

pedants yeah stop us blowing ourselves up i'll call them auditors yeah yeah nice peter i'm really struggling to think of anything peter doesn't make mistakes mistakes make him we could probably leave it at that actually anything i wasn't really struggling to think of any kind of like concrete single thing yeah one more more of an attitude problem in my case

Speaker C: 00:23:44

asa's mars climate orbiter in: 1998

Speaker D: 00:24:40

and they think it's an act of war yeah exactly yeah well fraser would it be easier to ask you when you haven't made a giant that's just dropping this day well look i was trying to think i was

Speaker A: 00:24:50

trying to narrow this down because i have made so many mistakes in my life it's not true um but and i i i honestly can't you know identify you know one above the others suffice to say this is that recently um we entertained ourselves down magro away um with my one of my kids i got him a metal detector um uh for his birthday because he he loves finding coins he just he's very good at it anyway without a metal detector and one of the ways that we tested the metal detector was on me um and we had it was family larks because you put it basically if you get a metal detector near me it will go off because i've got all sorts of metal mainly in my arms as a result of at least one accident that i had that was definitely because of poor judgment on my on my part and that's just i think that sort of just stands totemic amongst all my errors that i've made and big mistakes strong ale on a bicycle yeah you're not far off to be honest so um so yes there's just too numerous to mention all my mistakes that i've made um but anyway but it particularly pleased me that the metal detector went off because one of the most um disappointing things that i have um after that particular accident is that i never ever set off um airport um alarms because i always thought that would be the case and i well i thought i'd be special and have a little card that i made up for that so so there we go all right thank you as always for listening to the cognitive engineering podcast i'm fraser mcgrew being here with chris ragg peter cogill and nick hare of aleph insights