Online Rating Systems

Speaker A: 00:00:00

Hello and welcome to the Cognitive Engineering Podcast produced by Tell Me Studios for Aleph Insights. In this series of podcasts we take a look at interesting topics and discuss what we think they tell us about analysis and decision making. I'm Frase McGruer and I'm here with Chris Wragg and Peter Coghill of Aleph Insights and this week we're discussing online rating systems and to what extent we can trust them. So starting off with yourself Peter, I think you've got an example to lead us in on. Off we go.

Speaker B: 00:00:33

So a thought was triggered in my mind in response to a recent breaking story where a forensics company which provides forensic support to a number of police constabularies in the UK is under investigation for falsifying or incorrectly handling forensic evidence which is likely to mean that some court cases maybe need to be re-held and it's feared that in many cases they don't have the original sample so they can't be re-tested so there may be miscarriages of justice as a result. So this was an inherently interesting topic but the thought occurred well I looked up this company on Glassdoor which is an interesting website if you've ever come across it, it's where people can put reviews about the companies that they work for. So it's often full of people, recent leavers who have been cheesed off for one reason or another giving scathing reviews but actually gives you an interesting insight albeit slightly biased perhaps into companies that you're either dealing with or you might be applying for work with because people are often specific about what's wrong or what's right with a company and many reviews are quite balanced. I've given reviews there for employers when I've actually quite, I've had nothing bad to say and I wanted to make a positive contribution. So it's a useful resource but it got me thinking more widely well is this

Speaker A: 00:02:23

resource a useful thing? Well hold on so and did you see a link between what you saw on Glassdoor

Speaker B: 00:02:29

and this firm? Yeah so I didn't do any comparative analysis with similar firms but I found that most of the reviews, well over two thirds, were extremely scathing about the company for one reason or another. Either the sort of the style of management or the working conditions, there were a lot of clearly unhappy people that had some association with this, who reported to have some association with this company and that got me thinking well is this a useful resource? If I was to, if I was conducting due diligence on companies would this be a useful resource and to what extent could I trust the reviews to give me a balanced insight into how this company is run and if knowing that they're likely to be biased in some way how can I how can I correct for that

Speaker A: 00:03:23

bias? And so just to be clear are we talking about, I mean in this case we're talking about almost these bad reviews on Glassdoor as being signifiers of other problems in the business, let's say, because they're not necessarily directly related, because sorry did you say that the firm

Speaker B: 00:03:42

had handled data poorly or what? I'm not sure, I'm not sure on, I don't want to allege anything, but there was questions asked about their handling of judicially sensitive

Speaker A: 00:03:56

information and samples. So I mean but what angle do we want to explore here? Is it that kind of signifier angle or is it just a more general one of you know how useful online ratings are? I think

Speaker B: 00:04:10

a little bit of both. I think perhaps if you've got a company which has got lots of unhappy people, you know is sick in some way, it's likely, it's probably not an unreasonable leaf of logic to think well if they run themselves badly then they're probably likely to do a bad job and so it's fairly you know and if you've got companies where everyone's happy and motivated and you expect that company a lot more likely to do a good job and provide good products. Okay let's

Speaker C: 00:04:43

bring you in at this point Chris. Yeah well I think you know I think there are two questions here that these systems post. The first is to what extent can these systems be taken as a reliable sample of the population whose views we're interested in? So to what extent can they be trusted to tell you what the average person in a particular pool thinks about something? And the second question is once you've made that decision, let's say they are representative, what does that information then tell you about your particular question? So it might be you know if you're looking at if you're looking at restaurant reviews or something like that it might be that you find out that lots of people think a restaurant the food in a restaurant is too spicy. So you know that's something that's been said but it might be you really like spicy food. So actually it's about translating not just first of all do people generally think the food is too spicy, secondly what does that mean for you and the decision you're trying to trying to make in your sets of preferences? On the first side of things I think you know these systems can be very susceptible to manipulation so over manipulation. So there's an example of the film The Promise for example which is a film about the historic film about the Armenian genocide and its rating on a movie rating site initially was very very low and there's a strong suspicion that this was orchestrated in some way by people who are particularly averse to the notion of the Armenian genocide being historically recognized. So that there was a campaign to downrate this movie so people wouldn't go and watch it and I was quite interested in going to watch this film so I opened up the reviews and I thought oh it's only got two stars that's interesting looked at the first you know 20 or 30 reviews all of which were giving it five stars and saying this is this is brilliant you know but what a travesty that this has been manipulated in some way. So you know there is these things can be susceptible to general manipulation or you know there's the issue of the fact do they represent the population even if they're not being tampered with through some campaign. How many people leave reviews for things? Well generally it's people at the extremes people who have a negative experience or people who have a very positive experience who are moved to leave a review but the vast majority of people for whom it was so-so perhaps don't bother to leave a review or generally you know don't see the point in leaving a review. Yeah sorry go on. Well I was just going to say so you know we need to take that into account the representativeness of the types of people leaving reviews on these.

Speaker A: 00:07:57

Okay I mean so to sort of sum up where we are so we have access now because of the nature of the internet we've got access now to crowd-sourced reviews or rating systems and one of the potential downsides of that is either they can be sabotaged or the and the the people leaving reviews there as you say can be a naturally sort of strong feeling in one way or another and so I guess the question is is that even taking that into account is is looking at the alternatives and so let's say as in previously you have to rely on what you know a bunch of critics from you know different national newspapers might say about a given play for example that you're interested in seeing and are we still in a better position? I think we I guess the question is are we still in a sort of even taking into account all the things that can go wrong we're still better

Speaker B: 00:08:58

off is that right? Well I think so as long as you're able to accommodate handling all that information so it might take you 10 minutes to read the Guardian review on a play but it might take you much longer than that to read a significant number of reviews about the same play on on a review sites so you you have more data to work with which is great and but you you could present its own problem having more data it takes more time and also you have to you have to develop your own sense your own spidey sense to try to filter out the the the chaff try to moderate those reviews which are clearly extreme and try and pick out the specific bits that are that are key

Speaker A: 00:09:47

indicators for you. In one way in some way we've talked one way that they seek to do this is that your your raters can be rated themselves right and so for example on when that happens on Amazon I actually don't particularly trust the I actually I actually pay very little heed to that that you know some people are star raters or that other people's find their reviews more useful does

Speaker B: 00:10:10

anyone else sort of pay much credence to those or not? Well you know it certainly in sites that I used quite a lot like Stack Overflow they've got quite a well-developed kudos system where you you have to work quite hard to earn more kudos so the people with high ratings are have demonstrated themselves to be quite knowledgeable in seeing a number of topics and and so you can you can generally trust their word and because in order to get there they they it'd be very difficult to get to a high level of kudos without surreptitiously with an agenda in mind you have to be a sort of sort of fully signed up member to the ethos of this platform which is about collectively improving

Speaker A: 00:11:02

everyone's capacity for doing so. So one way of doing it is that you rate the raters and are

Speaker C: 00:11:08

there systems that work it looks like there are? Yeah I mean I think they're where the difficulty comes and and this is the case for most things which we review where the difficulty comes is for things which are subjective in your enjoyment of them so take films for example you know what mass rating systems tend to do is give you lowest common denominator experiences and so you know the advantage the disadvantage of the trusted single reviewer or set of elite reviewers you know who used to be you know times columnists or whatever is that they it's easy to to it's far easier to corrupt one person and you know bribe one person than it is to to set up you know an entire crowd of people to to do something so they're probably were more susceptible to that that sense of favoritism or or bias and you know those individuals wielded a great deal of power for you know a restaurant critic could could make or break a new restaurant's existence and that's that's probably not democratic or fair but where an individual reviewer has advantages and at some point I think we need to develop a hybrid approach which which gets the best of both is Roger Ebert the the famous US film critic said you know most people don't don't watch a film consciously they don't think about a film consciously they go to a film and then they give it you know three stars two stars or whatever they're not necessarily expert in in in reviewing films or indeed in critically examining films and sometimes what you build up with a a reviewer is a sense of trust in their opinion so you get oh yeah you know this particular reviewer has got my number they know my you know my mores and they will be able to pick something which which I like I trust them on these kinds of issues and so while somebody who is rated by most people as being trustworthy that doesn't necessarily mean they match they match your set of tastes and your subjective view so I suppose that the hybrid approach is um is where you move towards a system where you have I would like to see ratings from people like like me people who are you know I know like the same types of things as me or represent the same uh perspective on the particular the the particular commodity as as I do and I think that's probably the smarter way of doing things and probably the way things are things are going

Speaker A: 00:14:02

and um yeah nicely explained and I like that um that still caters sort of for individuality um so I mean we started to talk there in terms of entertainment films but is there any sense we can talk about this in a different way because we started out by talking about a company and how competent it was um so can we broaden this out stuff that's more important than films um and bringing it back around to a similar kind of example yeah it relates to a

Speaker B: 00:14:33

common uh well-studied phenomenon um where the prevalence of information and so social sites on on the internet allows for people to coalesce around into echo chambers where they have where they have circular conversations that perpetually reinforce their preconceived ideas uh and um this is I believe I think is a is a growing serious social issue where um uh we we historically we've had sort of um physical segregation of of different of different peoples um whereas now uh we have we have uh the same thing going on in in on the in cyberspace where people people's views are incompatible and not shared and people aren't learning and making the best of each other but they are they're they're um retrenching into into stovepipes and and and which which don't um don't cross-pollinate with with good ideas or are in active conflict with each other so I'm thinking uh you know Ghalis Fora um the English Defence League forums these these are the places where they're they're probably some very decent moderate people on there but because they have a a slight bias in one direction they end up strengthening that bias uh for the worse of

Speaker C: 00:16:05

for the worst of all. I think I think it it boils down to really um what what things are appropriate to seek ratings for and so you know we we use them to help reduce uncertainty and to help us uh help support us making decisions in choice about some different options so like where rating sites are very useful is for things like uh what what product should I buy or what service should I buy so you know we have lots of um if you want to get a builder you can benefit from other people's experience of you know who a good builder is by looking at a particular sort of rating uh rating site um and under those circumstances you you know you may only employ builders you know once every 20 years or something uh it it is sensible to consult you know other people's opinions uh in order to to do that and those kinds of um ratings can can be can be very useful uh but it's it's about something around which we don't we don't have direct experience of and we can't make a judgment where I think it's more where we veer towards um the territory that that Peter's talking about is where you use those rating sites or where you use others opinions and uh reviews of things um to shape your views and opinions on things which perhaps you should be making your own mind up on to some extent so for example uh you know what political party should I support you know you could view that as I'm I'm buying a political brand and I want you know some reviews on this um but you you those kinds of decisions uh you you know you've seen the same political messaging as as other people uh and you should be able to form your your opinions based on that there's there's less uncertainty about what it is this thing is going to um what this thing actually is you can you can see you know all the all the various options and make your own your own decisions so I suppose you know uh the the consumer of ratings uh ratings platforms needs to needs to think about what what am I using what kinds of things am I using this to rate or or

Speaker A: 00:18:29

help shape my decision making process on but okay but I still go back to this question of trust how can we be sure or mitigate for trusting in these systems yeah well as I said earlier you know that

Speaker C: 00:18:42

there are sometimes campaigns to um uh to to um uh to manipulate these things and and then you get you know you you sometimes get uh individuals maliciously acting in these systems where um you know uh uh you get sock puppeting where somebody will pose as a client of their their rivals and say the service was terrible or you might get an author reviewing their own book very favorably as as has happened in a couple of famous famous cases and then they've got caught out and it's been been very embarrassing for them so I suppose um you know we we acknowledge the usefulness of these systems but but the the potential for um for mistrust in them is is is quite great so how do you build how do you build systems that spot deception or or manipulation uh and and that's something which you know which which we need to think about and be uh be be clever about because there's there's an element of gaming any any system you set up which is rigid which says okay you know uh um here are our set of rules for identifying impostors or um people manipulating the system can quite quickly be gamed by by the people people themselves so um you know this is

Speaker B: 00:20:05

quite a complex complex problem okay and aside from external rules which look for deception I think there's a lot that uh if you are designing a community there's a there's a a lot of careful design is needed in designing the incentivization structures to dissuade people with particular agendas from manipulating and gaming the site and rewarding people giving fair impartial um and meaningful information rich opinions about things what about um no okay so

Speaker A: 00:20:39

one thing I'm curious about is um to what extent I mean I am definitely a consumer of online rating systems but I very rarely if ever um have contributed um I'm trying to think if I have p so I don't think I do or I have which is rather kind of means I'm a sort of a free rider on this

Speaker B: 00:20:59

but um I bet you have Peter I bet you know I do quite frequently I use TripAdvisor as a resource for discovering restaurants and things particularly if I'm uh on holiday um I'll use it to scope out potential places to go and um try to leave a review if I if it's been even if it's been adequate I like to try and give an honest review about what I found was good and and I think I

Speaker C: 00:21:25

think this uh this is the key lesson the the greatest um the the greatest way we can uh sort of mitigate um these systems from being manipulated is to be active participants in them I mean you you said you're you're a voyeur of of the of the system effectively uh and you've never left one that probably means lots of people like you haven't left one and therefore when you're reading the reviews they're from people not not like you uh whereas when you've taken part in something you just build up more more trust in it because you you have a sense of well I've reviewed these people may well be like like me and therefore I I trust their their reviews more so I suppose the end point is uh you know make sure you you contribute and you don't just sit there lurking looking at everyone else's hard work I like that uh that's something I can take away

Speaker A: 00:22:20

from this and in fact um I need to take a lesson in uh in one of my own lessons because I'm always complaining about with my mum right my mum is 84 about to turn 84 and she's a voracious user of Facebook and she she absolutely loves it but what she does is she sits there for hours and hours and meticulously trawls through posts and then connections to those posts and you know she goes quite deep down the different connections and stuff but what she never ever ever does is comment or post anything herself or like anything and um and so yeah so I need to be less like my mum and start and in fact we all do and and sort of I'm sure your mother's a lovely lady she is a lovely lady as well she is a lovely lady but I but I think this is I think

Speaker B: 00:23:08

just to really round things off I think this is part of a sort of social maturation which is going on the internet and all these social tools are all brand new we've never had anything quite like it before and it's good we're we as a as a race are going through a uh as a species are going through a process of working out what is what is right and what is ethical and what's the most appropriate way of using it and I think maybe that sort of being more being not just a passive user of it but being an active member of the community um is is is probably one of the mature things we need to do and if if uh I've seen it in specific examples where actually your your membership of a certain community is contingent on you being an active member and if you if you aren't an active member then you then you're booted out and you those benefits are withdrawn from you.

Speaker A: 00:24:00

There we go okay nice note to wrap up on okay I'm Fraser McGruer we've been here with Chris Wragg and Peter Coghill of Aleph Insights you've been listening to the Cognitive Engineering Podcast thanks as always for listening until next time bye-bye.