Why is AI not an authority on fairness?

- Hello everyone. I am Chris Hyams, CEO of Indeed. My pronouns are he and him, and welcome to the next episode of "Here To Help." For accessibility, I'll offer a quick visual description. I'm a middle-aged man with dark-rimmed glasses. I'm wearing a black t-shirt. And behind me is books, records and my red sparkle drum kit. At Indeed, our mission is to help people get jobs. This is what gets us out of bed in the morning and what keeps us going all day. And what powers that mission is people. "Here to Help" is a look at how experience, strength, and hope inspires people to want to help others. My special guest today is Hannah Fry. Dr. Fry is a professor in the Mathematics of Cities at University College - London. She is a mathematician, a bestselling author, an award-winning science presenter, and the host of numerous popular podcasts and television shows. In her day job, she uses mathematical models to study patterns in human behavior and has worked with governments, police forces, health analysts, and supermarkets. Her TED Talks have amassed millions of views and she has fronted television documentaries for the BBC, Bloomberg, and PBS. She has also hosted podcasts for Google's DeepMind and the BBC. Hannah cares deeply about what data and math reveal to us about being human. I've been looking forward to this conversation for some time, Hannah. Thank you so much for joining me today.

- Oh, thank you for having me. What a delight to be here.

- At Indeed, we think about jobs and especially job titles all day long, and you have really one of the best job titles I've ever seen: Professor in the Mathematics of Cities. What exactly does a professor in the Mathematics of Cities do?

- Well, so my training is very technical, lots of, you know, differential equations and calculus and all of the things that people got frustrated about in college. But the way that I apply it is to large scale patterns of human behavior. So it's movements of people through crowds, through transport networks, but also the movement of things that people do. So pandemics being one of them, but also the other contagion effects like social contagions, things like social unrest, lots of those different kind of things where you have patterns that move around in space and time.

- You've done research into phenomena like behaviors of rioters or patterns of burglary. Can you give the listeners just a taste of what some of this research reveals?

- Yeah, absolutely. So, okay, so the burglary stuff is quite interesting actually, because if you look at the places where burglars target, there are these quite clear patterns that appear. So burglars tend to go for houses which are within their awareness space, it's called. So essentially if a burglar walks past your house regularly, say they're on their way to work or they're on their way into town or something, if they go past your house, it's going to be more likely to be a target than if it's a house that's right at the very end of a cul-de-sac or a dead-end road, I think is the phrase that you use in the States. But then there's other subtleties too, because burglars tend to prefer, you want somewhere that is not too busy because you don't want people to be sort of acting as local guardians. But then at the same time, if you go somewhere that's very, very quiet, if you're kind of thinking in the mindset of a burglar, if you go somewhere very, very quiet, then you are going to really stick out like a sore thumb. So you can look at all of these different aspects of a city, the street network, how the effect of one burglary impacts subsequent burglaries, and you can use this just to try and help to mitigate against these kind of crimes. So for example, there was one big study in the UK where people would put leaflets through the doors of people who they believed would be targeted in the immediate future and actually managed to reduce the number of burglaries as a result. People just being more careful and sort of a period of more vigilance, perhaps.

- You've dedicated most of your professional career to studying these patterns of human behavior. You started out studying fluid dynamics. What was the inspiration to make the leap from the physical world to the human side, yeah?

- To be honest, it was a bit of a fluke. So what I really wanted to do my entire life was I really wanted to work in Formula 1, right? I was obsessed with motorsport. And I think I just didn't realize when I was growing up, I just didn't realize that engineering was a thing that girls could do. It just genuinely never occurred to me. But I was a really good mathematician. I was really strong at math, so I kind of continued on that particular route. And then when the opportunity came up to do a PhD in Aerodynamics, Fluid Aerodynamics, as we call it, mathematicians call it, you know, I went full into it and then ended up I landed a job as an aerodynamicist in Formula 1. I got the dream job. And actually, by the time I got there, this is in about 2010 when I finished my PhD. By then, all of the really delicious mathematical ideas had been embedded into pre-written code. And so the job had changed, you know, and it was essentially, I was sitting there and I would like set up runs on the computer and I would come in in the morning and I would pick the picture that had the least amount of blue in it. And so I think I was like, "I have not spent eight years at university to do this little mathematics. I want the juicy math." So, but this was just at that moment, 2010, where big data was just beginning to really gain a proper foothold. You know, the internet of things was just starting. Sensors were being embedded across, you know, across urban spaces, but also about people's persons at the beginning of the smartphone. So there was just the very beginnings of this explosion in trying to apply these same modeling techniques, but looking at human behavior rather than physical systems. And so I just scurried back to university and did a postdoc for that. Mostly, I was chasing interesting problems. I think that's what I was doing.

- So it does seem like you went completely in the opposite direction, and among the variety of topics that you have really delved into and certainly have gotten attention to people is the mathematics of love. You have a incredibly popular TED Talk. You wrote a book about this. Can you share some of the insights that you learned about looking at, trying to quantify the most, maybe what we would think of as unquantifiable aspect of who we are?

- Yeah. Yeah. I mean, okay, so this, I have to confess, this was never a serious research area of mine. It was more of a private joke that got terribly out of hand. Essentially what happened was, I mean, I am like so nerdy, okay? I mean, you cut me open, I would bleed mathematics, right? That's sort of the situation we're in. And so when I was single and I was dating, obviously, I had all of these tactics and strategies to optimize my own dating life because, you know, what self-respecting nerd wouldn't? So I put them together in this TED Talk and then subsequently went on to write a book about them. But I do think actually, lying underneath that sort of superficial thing on the surface, I do think that it proves a really important point, which is that I think that that those two words, of love and maths, they don't really feel like they sit next to another. But I think that I wanted to prove that even when you take the thing that feels hardest of all to describe, even then, if you look at it from that mathematical perspective, it still has something it can offer you. My favorite example of this is when you look at the arguments between couples who are in a long-term relationship with one another. It's a fascinating study, this is by the psychologist John Gottman, an American psychologist. So what he's been doing is he gets couples and he puts them in a room together and he videotapes them, and he essentially asks them to have a conversation about the most contentious issue in their relationship. And then he's worked our way to go through and to score everything that happens in that conversation. So every time somebody smiles, they get positive score. Every time somebody stonewalls their partner, they get a negative score and so on. But what I really like about this is once you have that quantitative view, there are these really unusual and maybe even counterintuitive results that come out. So one of them is that they discovered something called the negativity threshold. And this is essentially how annoying one person has to be before they provoke an extreme response in their partner. Now, I mean, I think if you were to guess, so not having looked at the data, if you were to guess, okay, like which couples are the ones that are still going to be together in, you know, 5, 10, 20 years? Surely, surely, I would've thought, it'll be the ones where it takes a lot for them to get annoyed. But when you look at it from this data-driven perspective, actually, it turns out that the opposite is true. So the couples who have the best chance at long-term success, they're actually the couples where that negativity threshold is really low. So what they're doing is they are continually repairing and resolving very, very tiny issues in their relationship. So if something happens, they don't like it, they say, "Hey, not cool." And then they sort of learn to respect each other by actually being open and honest with each other about what their true boundaries are. And I think the thing I like about that so much is that okay, of course there's important stuff about the way that the language that you use, the way that you approach things with your partner. But I just, I think there's something really elegant and beautiful about that, that even when you look at our relationships written in numbers, there's something really profound in there about who we are as people and about how we interact with one another, just there in the data.

- You've spent so much of your time thinking about math and numbers and data and algorithms and what they can tell us about who we are as people. And then you had a very deeply personal experience with encountering in the healthcare system, math and numbers, and you ended up chronicling this experience in a BBC documentary that as we were speaking before, I mentioned my wife and I just watched last night called "Making Sense of cancer With Hannah Fry" and is incredibly powerful and moving, your personal journey and the questions of patient-centered care and how these statistics might or might not actually help in making these decisions about your life. So can you tell a little bit about this experience?

- Hmm, yeah, of course. So it was January, 2021, a year into the pandemic. And I was 36 years old and I was diagnosed with cervical cancer. And the big thing about my particular case was that we knew that it got into the lymphatic system, but we weren't sure about whether it had established in any of the nodes or not. So if it hadn't established in any nodes, then my chances of survival were really good. And, you know, 90% plus. If it was in one node, it would drop to 60% chances of survival and if it was in two or more nodes, the odds would turn against me, essentially, I'm more likely to die than to survive. And what I realized in receiving that information, you know, this wasn't just, for me, it didn't just feel like a cancer diagnosis. It felt like I'm someone who has spent my entire life thinking about risk and uncertainty and probability and numbers and their interpretations. And it was, these were the numbers. That was the cold, hard fact of it. But they were meaningless. They didn't help me at all. They gave me nothing. And I searched and I searched and I searched, and I became obsessed at looking for statistics to help me understand what was going to happen. And as hard as I looked, I couldn't find anything that gave me what it was that I wanted, which is, "This is what's going to happen to you." But I mean, as I think a lot of people do when they get a diagnosis like that, I started writing a diary, and then because I make TV programs for a living, a natural extension of that was to start videotaping. And so once I came through the other side and kind of spoiler, I actually, I got very, very lucky and I'm absolutely fine. There've been no evidence of disease for two years now. But once I came out the other side, I realized that there was something about this, something about the way that we handle cancer care, you know, as a society where we are really bad at dealing with uncertainty at the best of times, but what we're doing is we're putting people in a position where they have to make a decision about their own life. And the only thing that we are giving them to help is population-level statistics. And it's so woefully inadequate to support them through that, you know, often the worst moment of their life and these very difficult and very quick decisions that they need to make. Because I think I wasn't the only one who felt like they didn't understand what the numbers actually meant for them. I think that's actually quite a common thing.

- So you spend your time thinking about populations and what data means in populations. And one of the key lessons from statistics is that they're incredibly useful on large sets of data and populations, and when it comes to an individual or a single trial, they almost become meaningless. And you obviously have known that from a numeric perspective. How has that impacted the way that you think about the math that you do?

- Oh, dramatically. Dramatically. I mean, as it is, intellectually, I know I know this stuff to be true. Intellectually, I know that there is a difference between 90% and 87%, for example. But emotionally, I can't tell that. I have no difference in my mind. I mean, if it's 90%, it's equal to 100, right? And 10% equals 0. That's sort of the only way. And I think that our minds do that, right? We filter things into binary. And so I think what's changed about the way that I do my work subsequently is that, I mean, I was already sort of going down this route anyway a little bit with some of the work that I was doing around the impact of algorithms on society. But I've really, really sort of doubled down since the diagnosis. I think that it's not enough just to make good mathematical models. It's not enough to just analyze data unless you are also really clearly communicating about uncertainty, because uncertainty is unavoidable, right? There is irreducible randomness out there. And I think that you have to be aware that the people who are making these mathematical models or doing the analysis, those people who understand the caveats, I think you have to be aware that those caveats disappear the further down the chain that you get. So that in the end, what people are looking at when they're actually making decisions is always going to be a sort of binary. It's 90% or 10%, it's yes or no.

- What could create a more human-centered healthcare system that doesn't involve a course in statistics for everyone? Like, how can the system evolve to help people make better decisions for themselves?

- I think you just take the numbers out of it. I think you have somebody who understands risks and understands uncertainty who sits down with a patient and says to them, "Okay, what is it that you want? What is it about your life that you most want to preserve and what are you willing to give up to hold onto that" And I think that that is a conversation that gives you the answer of whether treatment or not is the right answer, but is one that doesn't necessarily involve any numbers. Because just, I think it's worth just saying this actually, 'cause this is something I hadn't appreciated until I had cancer myself. I think if you are lucky enough to go through your adult life with very little interaction with the healthcare service, you sort of get this false idea of what medicine is for. You know, you go and you've got like a rash so you get a cream and you get rid of it, right? Or like an ingrown toenail, you can cut it and whatever. It's like you have a thing, you do a thing to it, the thing goes away. But actually with cancer, you are dealing with an invisible enemy that may or may not be there. You may, I mean, often people having chemo may already be cured of cancer. You are dealing with treatments that may or may not work. There is uncertainty in every conceivable direction, and you are balancing probabilities and long-term risks and chances of survival. It's a very, very, very difficult space to navigate. And I think leaning too heavily on numbers in that situation, especially, numbers that are based on population-level statistics, I think is a very tricky thing to do.

- Well, I'd love to move on to talk a little bit about artificial intelligence, which you have written and talked and lectured and podcasted quite a bit about. It has been around for a very, very long time, and it touches nearly every aspect of our lives. However, interest in AI as a topic has exploded. ChatGPT seems to be the culprit in this. And so I'm curious what your thoughts are as someone who spends a lot of time talking to practitioners, but also talking to non-practitioners about this and about the, what is it about ChatGPT and large language models that you think has captivated people and gotten everyone so wrapped up?

- I think it's because you can anthropomorphize it. I really think it's that. I think it's that, I mean, when you look in the heart of these things, if you like, get into the guts of these algorithms, I mean they're just, it's clever number manipulation. That's really all it is, you know, these things that I still don't think that any algorithm has ever demonstrated a conceptual understanding of what it's manipulating. And I think ChatGPT comes close in that it can, you know, relate tokens across different spheres. But I think that we are still not at a point where these things understand ultimately what they're spouting. But I do think that when you have an algorithm that can say, you know, that can respond to you as a human would, that can take on sort of a character that you can tease and you can play with and you can, you know, manipulate, I think suddenly, it becomes, it sort of taps into that imagination that we have, that inclination that we have to deify these things, to imagine that they are some kind of superior superhuman being. And so I think that that's really a big part of it. I think the other thing is that actually, I mean, there was a big leap forward with machine learning in 2012, but it was all very technical. It was all very, you had to really understand how to use Python and to manipulate datasets to be able to use it to your advantage. And then I think we were seeing like a little trickle down of some of those elements. But I do think that with generative AI, I think we're really seeing the point at which a lot of the workings, the guts of it as I'm describing it, is now invisible to the user. You know, it means that it doesn't matter whether you have a supercomputer or a smartphone, you still have access to the same fundamental model. You can still do the same things with it. And so I think the usefulness of it has switched because so much more of it has become hidden from view. Yeah, I mean, I use it all the time. I dunno about you. I'm like on it all the time.

- So at Indeed, obviously we spend all of our time thinking about the world of work and the impact of AI on jobs and how we work. And I'm curious, in your writing and your thinking, what do you see as the impact of AI in the future of work?

- I mean, I do think there is going to be a seismic shift. I think that is unavoidable. I do think that in terms of people, I think certain types of work and certain types of jobs are genuinely under threat. But I do also think that the idea that, you know, that AI will render human involvement obsolete, I think that's for the birds. There was a programmer who wrote on Twitter that I, and I really liked the way that he put it. He said, "I've just played with ChatGPT and I've realized that 90% of what I now do, I'm worthless, but the remaining 10% has just increased in value by a factor of a thousand." And I sort of think there's some truth to that. You know, I think that there are some, you know, I did a photo shoot earlier today and the photographer and I were talking about exactly this point. It was for a thumbnail for a new podcast I'm doing. And he was like, you know, "You could kind of get rid of me in this. I don't really need to be here. You could just generate your image, you know, using Midjourney from previous photographs of you." And I do think that's true, but I also think that there is something about communicating the human experience from one person to another, which ultimately we prize above anything else.

- So one of the things that we're particularly focused on, given our perch of really at the sort of the center of the global economy is how bias creates significant barriers to employment for marginalized communities and how we can still have racist, sexist, ethnocentric outcomes in AI, even from good-intended practitioners just because of the data and society and other things. So if you can talk a little bit about that.

- Oh yeah. Absolutely. There's one paper that I really like about this, and it's called "Women Also Snowboard." And it was a paper where it's using image recognition. So you use a photograph and then automatically generate a label for what's in the photograph. And this group of academics, they noticed that that time and time again, if the gender of the individual who's being pictured wasn't clear from the image, the AI would make a guess. And it would guess along gender stereotypes. So if somebody was snowboarding and had a helmet on, it would always guess that it, well, not always, but it would predominantly guess that they were male. Likewise, if there was somebody in a kitchen or nursing or whatever, and you couldn't tell their gender, then it would guess that it was female. And what they decided to do is they decided to trace back to see where this bias originally came from. And they found that actually it was in the training set. So the way that these algorithms, the algorithms don't just sort of like emerge understanding what a snowboard is, you know, they are trained by humans who have gone through and labeled data. And if you go back, and often by the way, the people who are paid to train these datasets are very poorly paid, often from, you know, developing countries. There's sort of a whole idea about, this is sort of new version of colonialism where you are kind of extracting from certain places to use it in others. But they found that there was still this bias in the training set, so that people were imprinting their own biases about what people do when they were labeling the data. But intriguingly, what they found was that when this bias was passed through the algorithm, it exacerbated things. So these algorithms, they're not this mirror that holds up a view of what society looks like to us. They distort it. It's like going to the circus, right? And you know, those kind of crazy mirrors that you get, some things will be expanded, some things will be retracted. But unless you go in there with this absolute perfect dataset that is completely free from all bias, you are of course going to end up with unfairness in the final outcome. Unbiased is impossible. You have to accept that, whatever you do, however hard you try, there are still always going to be problems. And so that means committing to a level of intellectual humility about your own work. And I think committing to continually hunting for and resolving biases and unfairness as they arise and making sure that you are creating routes for appeal. You cannot suppose that these algorithms are an authority of fairness because they absolutely are not and it is dangerous, dangerous, dangerous to assume that they are.

- So there's a couple of DeepMind researchers that published a paper on this, and I don't know if they coined this phrase, but I actually hadn't heard the phrase before, "Decolonial AI." What are some of the principles of what a "Decolonial AI" might look like?

- Well, I think that the very first thing is you've got to make sure that you have got a fair representation sitting around the table. You know, I think that we can't continue with this, I mean, I say we can't, we probably will, but like it would be good to not continue with this situation where you have one particular group of people who are designing the future of the world for everybody.

- We have no discipline that exists today around fairness, and yet we are as professionals building these systems based on data that comes from humans who are deeply flawed and we need, and so I sort of made this call for, we need a new discipline of fairness assurance, and we need to educate the professionals who are working in these fields about ethics. And I actually made this case that we should create a Hippocratic Oath for computer scientists and data scientists. In doing research for our conversation, I saw that actually at almost exactly the same time in August or so of 2019, you made a very public call for this Hippocratic Oath for professionals in this field. Can you tell me about your idea, but also sort of what was the reception to you presenting this?

- Yeah, absolutely. One of the first projects that I worked in when I moved over to use mathematical models on human behavior was a collaboration with the police, where we were looking at a particular set of riots that happened in the UK in 2010. And they were very large scale, they went on for five days. Things really got very out of control. There was lots of arson, lots of people were harmed, lots of looting. And the police wanted to understand what they could have done differently to really bring about a swifter resolution to that unrest. So they had really given us a lot of their data and asked us to look for patterns in it. So we published that paper and that was all fine. And then a couple of years later, I was in Berlin and I was giving a talk about this to a big audience in Berlin. And I was standing on stage and I was giving this very enthusiastic presentation. And I was saying, you know, "How brilliant it is that we are now living in a world where data and algorithms can help the police to control an entire city's worth of people." Right? Okay. I can see from your smile, just how naive I was at the time. 'Cause especially in Berlin, right? I mean, it didn't even occur to me that if there's one city in the world where you don't say that, it's probably going to be in Berlin. Anyway, as a result during the Q&A, I mean, they destroyed me and quite rightly so. But that, for me, it was such an important moment for me in my career. It's the reason why I went to go and write "Hello World." And the reason why I've been thinking about the impact of algorithms on society ever since. Because the thing is, that if you are trained as a mathematician or a computer scientist or a physicist or whatever in these technical subjects, you don't have to worry about the ethics of a particle traveling down a wire. But what we're doing is we're taking those same people and we are asking them to work on very, very technical problems that actually impact humanity and the future of the planet. And I think that doing that without asking them to pause and think about ethics and bias and fairness, but also about uncertainty and about the edge cases, that that mean that some people get trapped in a system unfairly. You know, I think unless you are building that in from the ground up, I think that we really are doing our future selves a disservice.

- As a final question I ask everyone, looking back over the last few years, and usually I'm asking that in the context of COVID, but you've also had had quite an extraordinary last few years with everything that you've been through and that the world has been through. What, if anything, has left you with hope for the future?

- I do think that we are standing at this extraordinary moment in history where we now have the compute power, the algorithms and the data to really, really change our understanding of the world, of physical systems, of chemical systems, of the universe, of everything. And I think that there are some really, really exciting things that are happening in the world of science. There are really positive things to look forward to.

- Hannah Fry, thank you so much for taking your time to talk with us today and thank you so much for everything that you do to make the world a little more understandable and illuminated.

- Well, thank you very much for having me.