US Election Forecasts

Speaker A: 00:00:00

Hello and welcome to the Cognitive Engineering Podcast produced by me, Fraser McGruer for Aleph Insights. In this series of podcasts we take a look at interesting topics and discuss what we think they tell us about analysis and decision making. I'm here with Chris Wragg, Nick Hare and Peter Coghill of Aleph Insights and this week we're discussing our predictions for the American presidential election. Okay so just to frame this as we're recording right now this is in late September and what we're going to do is we're going to give our predictions now and attach a probability to them and then we're going to combine that we're going to do a session after the results have been announced and look back on our predictions and talk about why we were right or wrong. Right? Sound about right? Yeah. Okay I think to spice this up a bit I think we should have some sort of prize. Okay. Because I think I'm going to win this. Okay. Okay so just in the context of what winning or losing means here. Yeah. Whoever has assigned the highest probability to the outcome that actually occurs. Yeah. Right so for example if I were to say I think you know Trump's going to win with a or Biden will win with 90% probability I'm also assigning a 10% probability to Trump winning. Okay. Yeah. So if Trump actually won my score would be 10%. Get it? So the person who had the highest probability of the outcome that actually occurs needs to get some sort of prize from the person who put the lowest probability on the outcome that actually occurs. What is that prize? A bottle of bourbon. I don't like bourbon. Well it's a suitably American type prize isn't it? What else? Packet of lucky strikes. A maggot hat. A baseball. An apple pie. I think actually let's go back to the bourbon because there are different kinds of bourbon right and so. None of which will have any appeal to me

Speaker B: 00:01:49

whatsoever but. You're not going to win anyway. Give it to someone else. Give it to a friend. Yeah. Oh no that's not going to work. Now we've gamified it though I'm going to change my prediction accordingly. And what order do we do it in as well because clearly then you've got. Yeah

Speaker C: 00:02:04

it could be an anchoring. Yeah there is. Maybe we should write down a bit of paper and just the probability. I think that's what we did. Fraser will announce it. Yeah. Then we can go into.

Speaker A: 00:02:15

Good. Yeah I think do you know what that's what we did four years ago as well. Yeah let's do that. Okay. Okay so what I've got I've got a bit of paper in front of me and on it is written Trump 40 percent Biden 60 percent and it's got Nick scrawled next to it. That's my forecast. Yeah okay and I know who this funny little paper is. This is Chris's. Hold on. I'd have to put my glasses on. Hold on. 100 percent Biden. Okay. Good grief. See that's where the gamification came in I think. Yeah but he's definitely going to lose if Trump wins. So that's not. That's yes. So if he thought differently. Yeah. Then he shouldn't have put that. And then we've got this is from Mr. Peter Coghill. Biden 60 percent and I'm going to use my mathematical powers. Okay so I reckon you've got Trump at about 20 percent. The remainder is like a big legal process. But Fraser what are you going to say? So I have got Fraser 100 percent to zero Trump. You've got a Trump at 100 percent. So I've got the exact opposite. Right so either you are buying Chris. You'll be buying Chris Bourbon or the other way around. And by the way it is interesting that bet because the only reason I put 100 percent was precisely because of our bet.

Speaker B: 00:03:30

Yeah no I. Funnily enough guess what my actual prediction was before we before it turned into

Speaker A: 00:03:36

a game. Was it Trump before. It was 60 40 Biden. Right. Yeah. I mean I think Chris is just playing this game badly because you haven't worked out that you know if you don't think it's 100 percent then you are building in some probability. There's only a winner there's only a winner and a loser

Speaker B: 00:03:52

right. And and so I don't to me actually I shouldn't I shouldn't have done this at all because I don't want the bourbon. But I want the prestige of winning and I'll give it straight back to Fraser when I win it. No give it to either me or Peter. Yeah so basically either me or Chris.

Speaker A: 00:04:08

We're actually doing the game properly. Yeah me or Chris are either the winner or the loser. We've got it right here right. Okay so let's let's go through this. So let's start with you know the the calmer heads amongst us. Peter tell us about your rationale for what was it Biden at 60 percent. Tell us about this.

Speaker C: 00:04:25

ng he's never going to win in: 2016

Speaker A: 00:05:34

Yeah and actually everyone's gone for so I mean Nick is the same as you actually and and Chris would have been the same as you but let's go to Chris and your rationale. Yeah so so my rationale

Speaker B: 00:05:48

ections from from the sort of: 1936

Speaker A: 00:07:30

first for for yours because I think yours will be more interesting than mine. Yeah no I just predicted that just for kicks really. I have to go I have no idea. I just do not know. So you're really 50-50? Yeah I mean frankly I just don't know and just given what happened last time it's just it's just and yeah and some of the uncertainties around that we talked about with the coronavirus lockdown this mailing system whether the vacancy of the Supreme Court seat makes a difference all these sort of things and I must admit I copped out and I just did the usual Fraser thing just went oh I don't know this is really complicated. But you played the game though. I played the game. I played the game. So that's why I went 100% because I wanted to win. But yeah so just for fun I put Trump. But I don't think I necessarily think that I just don't know. Nick? Well I did I did a similar thing to Chris in that I actually but I sort of actually plotted the lead in September in the polls versus the eventual lead for the winner right and it's surprise it is surprisingly accurate actually even in September but with always with a narrowing so sorry not always but in on average with a narrowing. So if you have a 10-point lead in September on average you'll have a seven-point lead in by the time of the election in terms of how it predicts the turnout. So it's actually a really good predictor. It's only ever reversed once and I think that was Dewey Truman and it's quite a famous example of really cocked up polling. However remember we're talking about the turn of the eventual vote right the popular vote now bear in mind that in only two of the last three. Are we talking about the popular votes? Yeah in only two no well no I'm what I'm saying is I'm forecasting the popular vote. Oh okay. In only two of the three in only one of the last three Republican victories has there actually been they've have they actually won the popular vote right so obviously they are winning through the electoral college because America has this weird system. Yeah. So now but Biden is about seven points up at the moment so on average you're looking at four points come the day and as Chris said you know in the last election the figures I've got here are Hillary was about three points ahead in September and about 1.8 points ahead on the night so you know actually pretty fitting the model pretty well. So you have you know a pretty good I think he's got a very good chance of leading the popular vote but the question is by how much and whether that's going to be enough to tip the you know what seems to be a kind of built-in advantage for the Republicans and actually so the reasons why it might move towards Biden so reasons why he might be favoured over the polls why it might drift towards him is that there is a lot of very motivated anti-Trump people I think who are who are less motivated than you know pro-Trump people on average that's the protests at the moment are what whatever they're probably they may be hardening people probably who are going to vote for Trump may not you know support them anyway may harden them a bit but it will make him look a bit weak and a bit chaotic. Biden is a lot more popular with minorities than Clinton was and they are important in swing states and you know the Covid handling what a second wave you know happening now could well again weaken Trump's sort of you know feeling that sense of control however and this goes to back to Peter's weird Trump factor I think even more so than last time people are much less likely to tell you that they're going to vote for Trump because in the mainstream media you know he is really anathema so and I don't know how big that effect is and I suspect it's quite big so that's why I've gone down to 60 I mean 538 who are really reliable are saying 77 Biden and I just my instinct my gut says it's going to be close and so that's why I'm going 60 40 okay I mean most

Speaker B: 00:11:35

most polling has you know for simulations based on the popular vote looking at electoral college seats has Biden up by like a hundred or more electoral college votes and that just doesn't

Speaker A: 00:11:54

seem terribly terribly plausible well I think I mean the point is it's very swingy because you know the whole election comes down to really a few swing states and those states you know are close and so actually you know that that it's like a kind of highly balanced set of scales where you just need to have a tiny bit of extra weight on one side or the other and it'll entirely move in that direction so yeah I mean it's definitely the weirdness of the of the US system makes this harder to predict in a way yeah now I was going to ask you indeed you know because that data that you you mentioned was pretty compelling with you know that sort of there's an average of three percentage points dropped right um so therefore why just 60 40 but I think you've gone on to explain nicely why why you know the uncertainty is there now yeah and and whatever happens we will look back and think what a moron I was why didn't I take this thing into account yeah but also yeah you will go back and I mean look here's a prediction neither you nor Peter are going to win this our little competition you but you're not going to lose it either that's the world the all the problems of the world in a nutshell one of you guys is going to be the big celebrity who says you're 100 sure and the people who are actually accurately assessing how uncertain they were we're going to we're going to be lost in the noise yeah with all the fame that brings you yeah yeah crashing and burning in in glory or or covered in the glory the victor's laurels one or the other yeah you're going to have to be whispering in one of our ears that we're mortal um okay right well um see you in November okay so here we are now it's actually a few weeks after the election and uh we're now recording this remotely because new lockdown so let's have a look at our predictions Chris I think uh if memory serves you went all out for Trump is that

Speaker B: 00:13:44

right yeah that's that's absolutely right Fraser yes of course um yeah no I um I think you owe me a bottle of bourbon I think uh um it's fair to say uh but um yeah actually you've you've noticed there's no require you can keep it because I don't trust you but yeah yeah there's only half

Speaker A: 00:14:05

of it left I see um excellent I like the way he's got that to hand as well he always gets the podcast days yeah 10 o'clock in the morning I always have it ready he needs it to get through

Speaker B: 00:14:15

yeah um yeah no so as you'll recall um I I officially went for 100 percent uh Biden uh but that was a feature I I just wanted to quickly revisit the way we we did those those um those judgments yeah because we we sort of set it up as a as a market uh you know where where you kind of backed it to the percentage that you you thought it was going to be the case but actually the competition turned out we only made those rules up immediately before the podcast but the competitions it turned out was really just a binary choice between Biden and and Trump and only two people could compete in that that choice uh so I I artificially inflated my my bid so that I knew I'd get the Biden ticket even though I didn't actually think that there was a hundred percent chance um so I just think that was that was a sort of interesting part of the um of the process but having having done that and then um sort of you know being in this situation where I was the one holding the Biden ticket I very much felt all in at that point so psychologically going through the election uh or in the run-up to the election I was really quite you know quite sweaty palms kind of uh what's what's going on and this is why I never never gamble because it's uh psychologically because you love it

Speaker A: 00:15:43

that's why you don't gamble you especially don't gamble over booze no no yeah okay look so so just to recap something you said there and I think I did the same thing which is you were favoring Biden but because of the binary nature and because it was a winner takes all thing um you just thought well there's no point just going a bit Biden let's go all in which basically means you go either gonna our our bet you're either gonna win it or lose it unlike these others it was it was me

Speaker B: 00:16:08

versus you whereas yeah Nick didn't have a didn't have a stake in the competition so it wasn't like they now win 60 percent of the bourbon or right collectively yeah they get nothing so but but that's that aside yeah come going through the um the actual uh election when there were some sort of key moments where I was like really sweating it and one bizarrely was um when Trump got got Covid I am I immediately immediate thought was crikey that could have been Biden right if Biden got Covid it was it was game over you know Trump would have definitely won the election you know but and and then and then the other moment was uh when Trump sort of emerges not dead from Covid I thought oh no you know more powerful than ever some say well yeah that's right um I thought oh no now he's going to emerge as the hero and um then you had the sort of the Joe Biden corruption kind of family son story that I thought again could scupper things um and then obviously the the election result dropped or rather the the you know highly contested uncertain election result dropped the next morning you know and and at that point I thought Trump has definitely definitely won this so throughout my psychology was every term not oh this looks you know hooray but Biden looks like he's going to win I was always thinking uh that Trump was was going to

Speaker A: 00:17:43

win because I backed backed Biden okay Peter or Nick um can you talk us let's go through with yourself Peter first um what do you want to add at this stage I mean thinking about what your predictions were and how you felt as things played out well I didn't have any kind of

Speaker C: 00:17:59

psychological stress that Chris had because I completely forgotten we'd made any predictions uh when the election was going on oh okay um yeah sorry Fraser you're not that important to me

Speaker A: 00:18:09

um but uh Peter can you recall what you might have said because it was back in September I think we

Speaker C: 00:18:15

imagining a much more kind of: 2000

Speaker A: 00:19:17

and Nick what's your take on all this and how you felt well I remember saying Biden 60 and I I sort of feel like that was probably about right um I what I one one thing I remember doing was looking at the September polls and saying okay how much do the September polls tell us about who's going to win going back you know since the beginning of polling and um it turned out that nobody had ever been ahead in September and then lost the election but as Peter said I recall uh when we were recording the first part of this podcast is the problem with Trump is it's all a bit weird anyway and Trump weirdness can you know can overturn any any trend you you want to look at yeah and I think so I think I mean my view was that I I thought the the polls were going to be skewed towards Trump more so than usual um and I I you know I kind of had this feeling that unless Biden was it would be 50 50 unless Biden was really ahead in the polls you know it was really kind of seven eight percent up now um uh over the if you look if you kind of just do a really simple regression of the September polls and the outcome um you find that um historically you know the winner stays the winner but on average the lead narrows by 30 percent in terms of the popular vote now that doesn't even if so Biden being ahead in the popular vote however obviously doesn't get him uh the election victory because of the you know U.S. electoral system yeah that's which we covered in a previous podcast um now uh so so at the time the polls were about 51 43 he had about eight points in the bag and that's why I was 60 40 because um you know I didn't I I sort of thought well that's only I reckon he's not out of the woods unless he's about eight percent up now in fact the result was 51 47 in the end so the undecideds you know were were much more Trump type than uh than than uh Biden so so the in the simple model I was predicting a lead of six um points in the popular vote but it came out as four points which if you if you kind of you're factoring the Trump the Trump bias issue basically what I'm saying is it actually more or less exactly what I would have expected but I but but you know I think I think that the uh the noise and and even though Peter is now seeing it as a bit more sewn on I I think a few little things like what Chris is suggesting and even a few little things on the night that we weren't to know you know could have could have changed things even if it was um you know just people's people's last minute decision making might have been affected by

Speaker B: 00:22:03

something yeah yeah no I was just going to say that um retrospectively um because before we'd begun the you know before we went uh on air I had my sort of prediction uh and then like I say just a few minutes before we announced the rules of what the competition were going to going to be um and so my assessment beforehand had been 60 40 uh Biden and I had all my you know lines of of argument about the polling and so on and so forth um but then changed it for the purposes of the competition but even that 60 40 once I was a hundred percent and I was the one who who was effectively backing backing Biden um I feel like it it was much much close I I feel like it could have much more easily gone in in um Trump's favor I feel 60 40 kind of almost overcooks it a little bit I think yeah you know Biden was was the favorite but but you know when I look back on events it feels like very marginally to me and so I think I've got a particular viewpoint here

Speaker A: 00:23:09

well praise you're our one diehard Trump supporter aren't you you're the 100% Trump man so how are you experiencing the election well I can yeah I can talk you through that because I think as it stands as we've talked about this so far I look like the big loser here more so than usual because I went all in for Trump have you donated to the Trump legal campaign yet not yet not yet but um you send them half a bottle of bourbon yeah but here's the thing as you know I am something of a student of the cognitive engineering way of of thinking of analyzing right yeah not the not the best student but you are a student well he studies it and then ignores everything that he's studied and just well here's the thing so when we did that last one it was just ahead of I think it was the first uh presidential debate um and I thought you know what these Aleph boys they know what they're talking about okay and so many times in the past you know I've learned you know um from the wisdom that you know the pearls that you've that you have cast before me and so I thought right what I'm going to do is I'm going to put my money where my mouth is because I had been saving up um for I wanted to buy a a double bass okay and a banjo because I'm about to learn those instruments right which is a fairly substantial amount of money and I thought I'm going to bet on this okay the way you had talked it through and you were talking about how um comparing to Hillary and the lead and how much that tends to get eroded a little bit before the um actual election it to me I was convinced by what you had said it seemed to make sense and so yeah so I went to a betting exchange and had a look at the odds on there they have them in a decimal form um which and I think Biden was uh 1.7 to win okay so let's say um you put on one pound you'll get 70 pence back really addition oh okay in addition to your one pound that you've put on and I'm not far off 60 40 yeah it's about that and so I thought yeah I'll put some money on this um and I thought well if I put 100 quid on I'll get um you know 70 quid back that's not really worth it um and so I thought sod it so because I've saved up all this money I put a thousand pounds on this right so I put a thousand pounds and now just to be clear I think the most I've ever bet in my life before is like a fiver on the grand national this is hugely exciting yeah it's true and not just made up it's definitely true absolutely true absolutely true and so

Speaker C: 00:25:30

and so um and your gambling habit has taken off and that's how you're that's why you're now

Speaker A: 00:25:35

wearing a scarf because you've got that that's right so and so as you so so I thought I need to do this for the first debate because I thought um Biden would really wipe the floor with Trump and he didn't but Trump if you remember did that sort of whole tantrum sort of thing and just outraged everyone and sure enough the odds um on a on a on a Biden win um shortened and so I put my money in at the right time in in a way maybe months before it'd been better anyway so all the time my behavior in I think there was still about two weeks three weeks to go and so every day about once an hour I was checking my app on my phone to see where the odds were going and actually they were weird because they were just going all over the place and apparently in elections they just tend to lengthen lengthen lengthen or shorten shorten shorten but they were just going up and down all over the place but it was all rather exciting and then but they were gradually shortening on a Biden win and then the night of the election itself I wasn't going to stay up all night but I do remember waking up in the middle of night and just seeing that um you know these talks of a democrat um whitewash were just absolutely not the case and I think at um about five in the morning when I checked um because here's the thing you can cash out at any time you want on this so in the run-up I could have cashed out and sort of got myself a hundred quid let's say or 200 quid or something but suddenly in the middle of that night I was like 500 quid down if I wanted to cash out wow um yeah and so um a lesser person without been the time to pile more cash in in theory and so some people did actually yeah um and so I saw that at about so I was about 500 quid down as it were um at five in the morning I went off and did my swimming um and I came back and by the time I came back it was about maybe 100 quid down and I thought do you make 400 quid during your swim I did and then of course as you can imagine I was avidly watching it through the morning um and then it gradually gradually got to where it looked like you know Trump wasn't going to do it and so it was um getting into I could cash out a 500 quid for example um and so yeah so that's what happened however so did you did you cash out at the do you what I mean you basically won the bet have you been paid yet but here's the thing here's the thing what's bloody annoying is because of I think it's not necessarily because Trump has has not conceded but I guess it's partly because of that because it's still officially not been called yet every time I check my app and I check this morning it's still saying that it's in play okay is it sort of currently worth sort of one you know one seven 699 pounds or something right it is yeah so it is pretty much about that it's worth about 660 odd quid and I just feel known I've got in this this far I've gone this far I'm gonna go all the way yeah and so I'm gonna wait even it takes to bloody January to get my 770 quid which I've sweated bullets over well that's great I feel really pleased for you and uh and I'm glad that we you know aren't liable for some thousand pound loss that you've made you know that's great this is the thing because in the morning uh or even the middle night I was there going bloody Aleph Insights they said this is gonna be clear cut and yeah no I was gonna write to my lawyer get them to sue you and you know yeah it's it's not it isn't the case uh that the market knows best for these kinds of things you know uh I mean the market for um the market for for kind of political events is very heavily driven by you know people's assessments and narratives and judgments and they can be skewed and and so I think you know it is right to at least start with the statistical approach and and then you know make a comparison with uh you know with what the odds are showing and you can you know you can be you can be better than the market but the point is that there isn't the volume of those kinds of events to make a living out of it but um uh but yeah no that's really good yeah I mean it's interesting what you said about the odds fluctuating wildly because that's um I mean one of the problems one of the things that the good judgment project identified as sort of characteristics of good forecasters is that they tend not to be they tend to adjust when news comes in but they don't over adjust whereas a lot of people are like driven by what what just happened yesterday um you should you know put that in a context of the longer term picture um so so yeah yeah it sounds like you you yeah there could have been money to made to be made on the night as well but yeah no absolutely but I think the the uh yeah thinking about the um uh you know that question what does it tell us about how good people are forecasting um obviously that's can you use a single forecast to tell you that and um there are approaches to scoring for you know forecasting ability so things the one of the standard ones is the Breyer score um which you can think about as kind of scoring you on a number of things which are your reliability which is sort of how closely your forecasts tend to match the observed frequency of things so if you take all the things where you said they were 80 likely has 80 to 80 percent of them happen obviously you'd want that to be the case then you have resolution which is how close your forecasts are to uh zero or one sometimes called discrimination and then there's a sort of adjustment for uncertainty essentially which is uh um to do with you know how variable the thing you're looking at is so for example if if there's something that only happens one percent of the time um you then you know that's you're not going to get as many points for saying it isn't going to happen so broadly but actually you know proper if you take a sort of proper Bayesian approach to doing this um you you could you can actually just look at obviously so using that approach the Breyer score approach you need quite a large corpus of forecasts to be able to find out how good someone is but you but you can you can use a single forecast to be quite informative and and you know at the moment it looks like you and Chris are the the most the best forecasters in the room on the strength of if we knew nothing else about your forecasting behavior you guys would come out top Chris we would rate as obviously sort of a better forecaster because his his forecast was 100 percent and it came true so you now you the next piece of data might dramatically change that but you might turn out to be still useful and informative but the exact other way around so you might just be like someone who's you know everything they they predict the opposite happens but someone like that would actually be really useful someone who's reliably wrong would be as useful as someone who's reliably right you know I thought there was going to be a compliment in there somewhere but no well as I say I think we'll have to we'll have to we'll have to run and wait for the next one yeah yeah yeah okay um so before we wind up

Speaker C: 00:32:19

anything anyone want to say well just well just building on Nick's point that it's lucky that we're Bayesians because if we're frequentists to get a statistically significant indication of of accuracy we'd need several hundred or thousand elections and I just haven't got time for that

Speaker A: 00:32:35

yeah yeah yeah um well all I can say is you know welcome to the world super forecaster Fraser um yeah um well and and and so I'll post this uh bottle of uh Jack Daniels of finest bourbon to you Chris wipe the spittle off it first okay we'll wrap up there thank you as always for listening to the Cognitive Engineering Podcast I'm Fraser McGruer we've been here with Chris Wragg, Nick Hare, Peter Coghill of Aleph Insights. Until next time, goodbye!