Why should you foster curiosity over certainty?

March 28, 2023

Join us for a fascinating conversation with Indeed’s own Director of Data Science, Hannah Lindsley. Chris and Hannah will discuss her journey to data science, the Theseus’ paradox, and how we can ensure that data science is used to promote social justice and equity.

- Hello everyone. I am Chris Hyams, CEO of Indeed. My pronouns are he and him, and welcome to the next episode of "Here To Help." For accessibility, I'll offer a quick visual description. I am a middle-aged man. I'm wearing dark-rimmed glasses, a blue T-shirt and a black zip-up sweater. And behind me is my red sparkle drum kit and on the wall some books, LPs, and photos. At Indeed, our mission is to help people get jobs. This is what gets us out of bed in the morning and what keeps us going all day and what powers that mission is our people. Here to help us look at how experience, strength and hope inspires people to want to help others, and helping others is also about looking at the world with a new lens and with every guest we aim to challenge old assumptions with new ideas. My guest today is Hannah Lindsley. Hannah is the Director of Data Science here at Indeed. She studied linguistics at the University of Texas and has worked in data science and engineering at HomeAway, Iodine Software, CognitiveScale, and Neuric Technologies. Hannah's brief bio reads, "I want to spend my career doing really hard things "with people I believe in." And I've had the pleasure to get to work closely with Hannah over the past year and I'm certain this intention will come through clearly in today's discussion. Today, we'll be diving into aspects of data science at Indeed, we'll try to better understand how data and how we look at it shapes our understanding of the world and we'll explore some of the limitations and challenges in accurately representing diverse perspectives. Hannah, thank you so much for joining me today.

- Hello, thanks for having me. I'm Hannah. I don't have strong pronoun preferences. I'm wearing a long-sleeved gray shirt and I have brown hair. I'm in the middle of a very sparse white-walled room.

- Fantastic, well, let's start where we always start these conversations. How are you doing right now?

- I'm doing well, I'm a little nervous. Excited though to chat with you.

- Great, well, let's dive right in. Let's start with you are a Director of Data Science, what does that mean and what would you say you do here?

- Yeah, that's a good question. What do I do here? I lead data science teams who optimize the employer experience. So data science is all about, I guess, helping scale decision making for companies, and for me, in particular, a big focus area is shaping how we make money as a company.

- So, let's talk about that. We have a set of core values in our business and the first two really were foundational to the business. The first is that we're a marketplace, we connect job seekers and employers, but we put job seekers first in everything that we do. And the second core value is that we have a pay-for-performance business model. And there's a lot of questions that come up about why is a business model a core value, but how would you talk about how Indeed makes money and the role that it plays in how we make decisions?

- Yeah, this is actually pretty different for me, at least, from a lot of the other companies that I am connected with and certainly that I've worked with, because making money here is so much more than just the contracts that we write, or how much people are willing to pay for our services. Let's be honest, money is one of the most powerful incentives on the planet. And we're trying to build our incentives such that they're aligned with our customers and also with our mission to help people get jobs. An unused subscription is a great example. So how many gyms make how much money off of people that don't go to the gym, maybe even ever. So a great strategy for revenue growth for a gym might be to directly target customers they know won't come into the gym, but we don't want to get paid when our customers don't get what they need, because not only do we believe that that's like how to build a sustainable and healthy business and grow for the future, but also just on a personal level, we're pretty driven. I would say almost everybody that I meet at Indeed is pretty driven altruistically by the mission to help people get jobs. But the reason my team and I have a job is because we aren't perfect at this. Sometimes we get paid when we're not delivering value. We are trying to systematically eliminate revenue that doesn't help our customers. This is why we're changing our business model from this sort of like pay-per-day or subscription-like model, to a pay-per-outcome model and giving enormous amounts of thought into how do we price these things.

- You mentioned that data science is essentially a tool for decision making. So let's talk about in the context of pricing and how Indeed makes money, what is the role of data science, and how would you describe data science, maybe to someone who's heard that term but doesn't really know what it means?

- Every single job seeker and every single employer are looking for, basically, something that will change their lives or change their business, right? Looking for someone to hire or someone to hire them, a job, a good fit. And there are, what, 300 million customers I think we have between job seekers and employers in the marketplace. We do not have 300 million employees to help guide each one through the process, and they each deserve that level of attention. Data science is something that we can utilize to bridge the gap, at least in part, and scale up to learn in aggregate from individual behavior, learn in aggregate, how can we serve an individual? And the way that most of us probably interact with data science on a day-to-day basis, I'm going to guess is probably advertising in general, like the way that any of us see it on the internet or on television or whatever. Basically an algorithm is reading your past behaviors like what you've read on the internet, sites you visited, things you've clicked on. And then it learns with high likelihood how to predict what you're going to click on next. And then they show you something you're very likely to click on, that next YouTube video in the stream. And there's something slightly demotivating about being a professional clickbait developer. And that's why I'm so passionate about my job today and how we make money. I want to be able to harness the power of data science to help people at scale, but I don't want to build clickbait. And that's really where it goes back to that job seeker first principle.

- That was a nonlinear path starting from when you graduated college, but actually what I think of as the Hannah origin story is a really interesting one, and I want to spend some time kind of digging into it, 'cause I think it says a lot about what you bring to the work that you do today, but also will give us some threads to pull on sort of how you think about data and what it means in the world. So if you can back up and share a little bit about your upbringing and your early education, please.

- Yeah, I was homeschooled in the '90s. Homeschooling was, probably at that time, pretty different maybe than the like COVID homeschooling that we all did with our kids, a little bit. It was a very, very sheltered environment. My education was largely self-directed, but my parents were super passionate about education. So there was this heavy emphasis on logic, philosophy, economics, political science, worldviews of the world. And it was a very fundamentalist religious experience as well. And I had extremely black and white views of the world. And I was very, very passionate, these were like my beliefs, it wasn't like just the culture that I was in, it was mine. And I was so certain about things. I had purpose, I had talent, I had the evidence on my side. So I was right, and I was very effective, just all the time, 100%. But I also loved to learn, and by the time I was 14, I started my first classes at Austin Community College. And by the time I was 16, I had about a year toward my degree. And so I moved over to UT full-time to study linguistics. And for the first time in my entire life, I was confronted with this very noisy cacophony of ideas and perspectives and opinions and new truths, beliefs that I'd never considered. And I learned what it felt like to wake up one day and realize that I couldn't believe what I used to anymore because my whole life had been focused so much on the way things should be. Like take these facts, take things that are likely true about the world, and then tell people how they should live, and decide how the world should look. And going into linguistics, language, you can say that language should or shouldn't look a certain way, but one of the beautiful things about language is that it is one of the purest expressions of change over time, like it's so natural. And there is no should in language. I don't know how to explain the complete shock, the complete culture shock of having this sort of insular world and then moving into this space with so many human beings. Just so many. And their brains were all as clockwork and incredible as mine, you know? And they came to radically different conclusions than I had.

- What was the aftermath of that? Because it's a disorienting experience to realize that the way that you've been living is based on a faulty view. How did you respond to this experience?

- I don't think I responded particularly well, particularly at the beginning. So while I had this duality of experiences at school where it was that love of learning, consistently being surprised thing. But then also I started having panic attacks, the realization that I could be so very wrong about something that I absolutely knew with every fiber of my being was true, not just disorienting, I don't even know how to describe it. It was terrifying. I felt like I didn't belong at school because nobody could understand the context in which I was raised, right? Nobody could understand where I was coming from. I didn't meet anybody while I was there. I also felt like I didn't belong in my body. The girl I used to be owned that thing, it was completely unmooring. Yeah, it wasn't great to begin with.

- So how did you put yourself back together?

- There is this sort of thought experiment, a vintage Hellenic thought experiment called the Ship of Theseus. Theseus is, mythically, the king who built Athens, and he's credited with this thought experiment that philosophers used to discuss, which is say there's a ship, a brand new ship, it's just about to set sail and it's pristine, it's beautiful, it's highly, highly glossed and ready to go. So over the next few years, sailors are using it, they're going back and forth on their journeys. Over time, some of the boards start rotting. They replace them and then more time passes and many, many, many boards are replaced. And in fact, all the boards on the ship are replaced several times over. And the question that is contemplated and debated in ancient Greece was, is this ship the same ship or is it a totally new ship? This framework and the Ship of Theseus was something, I think my brother was the one who first raised this to me, because he was right alongside me when I was going through this transition. He was like, yeah, this sounds like the primary question for you is Theseus's paradox. And over the course of my life, I've had a whole bunch of planks ripped up. What am I going to do with this? Am I going to be the same person or am I different? I made a decision, particularly after living with and learning from something like severe anxiety that I was going to embrace the old me and the new me together. And I wasn't going to blame my past self for my current self disagreeing with it. And that I would be one person who would be both right and wrong and probably neither and make my home between two cities. This has been really fundamental for me in how I see disagreements between any two people, because I know that we can all wake up tomorrow and have a different point of view.

- Thank you for sharing all of that. I'd love to talk a little bit, we are deep in March right now. March is Women's History Month, and both the world of technology, but also the world that you came from, gender and gender roles tend to have some sort of structure to them that I think people expect that also have maybe lack sort of solidity and permanence to them. Can you talk a little bit about what all of this brings up for you?

- Yeah, so I mentioned that I was raised in a religious environment and the church, and not just a single church, but the broader community of the church, in the sect that I was raised in, often believed that women shouldn't work generally at all. And certainly that if they did, they couldn't hold certain roles of leadership, in particular over men. So I had this one part of my upbringing and it was discussed quite a bit. This was very, very prevalent for all ages of girls and boys. But then separately in my household, my mom and my dad, and I have three brothers and one sister, it was absolutely not like that. I actually, my parents did not say things like, well, as my daughter, as opposed to as my son then, they had incredibly high standards of education, of work ethic, of progress and thought for all of their children. And, honestly, I had this incredible privilege of not thinking about it very much in my safe space, in my home with the people who loved me. But then going out into the broader community where people disagreed, I felt totally safe to just say what I disagreed with and kind of interact in that way with that broader narrative. In fact, I can't think of my parents ever making some distinction like this, except one time with some advice that my mom gave me that has stuck with me, I think about it most days actually. My mom said that she wouldn't make the boys, my three brothers, go to college if they didn't want to, it would be their choice. But that Leah and I, my sister, we wouldn't have a choice. We had to go to college, that was her expectation. And she said it was because we were very likely to have children. And children can change a woman's life, in particular, in a very different way than it can change men's. And she wanted us to be empowered with the knowledge that we had choices and we choose our own lives. And so that was really powerful for her because she went to college, and she wanted to give that to her kids. And I'm very grateful for that advice because not one year after I started working, I got pregnant unexpectedly. And that was an incredibly intense time. For one thing, I was working with a company, I was the fourth person at this company. And so we only had four of us, and guess what? That means you don't have to have FMLA. So I took two weeks unpaid time off to have a baby and then came back in full force, highly emotional, not particularly hormonally recovered at all. And I remember just crazy experiences. I was breastfeeding my son, and so I would have to pump at work, right? And there were no locking doors in the office except for the server closet. This is funny in hindsight, it didn't feel funny at the time. But basically, a server would go down because we weren't particularly stable, and people would be banging on the door, and I'd be crying and they're trying to, and I'm like, "I'm not making enough milk for my baby! "I'm a terrible mom!" And just incredibly, I don't know, again, funny now, not so funny at the time. And one of the guys that I worked with, a kid really, he was just out of school. He thought it was gross to have milk in the refrigerator with his food. And so I got a separate refrigerator, kind of the antics and adventures, I suppose, that a lot of women, I think, are familiar with, particularly from that time period. But all that to say, when I moved more into data science, because that was, I guess, I didn't work with women for years, not a single one. But when I moved more into data science, away from engineering and platform development and ontology building as well at the early days, things started to change, in part because women of many other countries, I'm thinking China and Russia in particular, had a much higher representation from women. And so these centralized data science teams that I was a part of would often be more like 50-50, much closer to it. And today, I think, certainly for several years now, and particularly at Indeed, I have that same privilege that I had when I was growing up in my parents' home that I honestly don't think about my gender much at work at all. It's not that I never do, and it's not that I never experience situations where I'm like, "Hmm, I wonder if that was targeted at me "because of something." But almost every day, it's something that I just don't think about.

- So getting back into data science from this, the way that we approach things here and maybe actually coming from your experience of being certain about the world and then having to embrace uncertainty, not just as I was wrong about that, but if I was wrong about that, maybe I'm wrong about everything. So our approach to things is experimentation, is actually trying to start from a perspective of, I actually don't know what the outcome is here. We might have a lot of smart people. We might have access to an incredible amount of data. We might have a lot of insight we've built up over time, but until we try something, we don't know for certain. So can you talk about experimentation?

- Yeah, experimentation is what science is about, honestly. Kind of putting aside the bias we have, the desire we have for something to go the way that we want it to go and say, "Okay, we're going to test it and we're going to see "and we're going to be open to the results "even if we don't like them." And because of that, experimentation gives voice to many, many, many ideas, even ones, especially ones, that we wouldn't pursue on our own. And this is how we get that surprise of progress. Progress isn't predetermined, we don't know what's going to be progress. We don't know what it's going to look like, we don't know what's going to work. There is no progress without failure. And progress often isn't the way that we envision it ourselves. I need people who think very, very differently about the world or else I'm going to be stuck with my own limitations. Like if I ran this whole product, it would be much worse than if we get the benefit of all the different perspectives and ideas in this framework of experimentation to say, well, let's see, let's see if it works. To me, I've said this to my team a lot, diversity is meaningless. Diversity on teams has no point without the autonomy given to these very, very, very disparate people to try things that their leaders wouldn't try. So using that experimentation as a framework for trying out many ideas, I actually think is part of something that speaks into women at work or any underrepresented people group or any individual to feel like they have a place to be and an impact that they can make.

- So as we think about then the implication, so data science has the word science in it and there might be some sort of implication or some ideas that people have about science and its infallibility, but clearly that's not the case. There are limitations to all sciences. What are the limitations of data science that are necessary to not just be aware of, but actually embrace in how we utilize it?

- Well, we've said that data science is decision-making at scale. So whose decisions and how good they are is definitely something that can limit us. If machine learning models, for example, determining which jobs people should have, were trained on historical winners in the job space, I wouldn't have a job. Historical data and generalization, essentially, can close us off to new ways of thinking. This whole thing about experimentation being a framework for empowering diverse people, well, the opposite can be true without a lot of intentionality.

- So what is the intentionality? How do we avoid these pitfalls?

- So I know a lot of people talk about some of the really good work going on in regulation and auditing and observing the results and ensuring that particular models are trained in certain ways. But I think for me, there's another way that we don't discuss as much that I think is really interesting, particularly in light of the stuff we've been talking about. Machine learning is a model of the world or some part of the world, some problem. It's kind of like a model of truth or a belief set almost. And just like one person can't establish the truth for everybody, trust me, I've tried, one model of the world is probably not sufficient. And so we get a lot of benefit from actually having multiple models, multiple points of view, that same protection that we get from diversity helps us find a closer approximation of the truth. We actually do this at Indeed, where we have many, many different teams saying, "We think this job and this job seeker "are a great fit." And then another team says, "This job and this job seeker." And another team says, "This job and this job seeker." They have different machine learning models that all actually come together in concert to determine what are the best jobs to show to this job seeker, rather than representing just a single perspective.

- We are unfortunately nearing the end of our time together. I could keep talking about this stuff all day here, but let me wrap up by asking the same final question that I ask everyone here, which is that we've been through an extraordinary three years from the start of the pandemic. And certainly from our seats at Indeed, we have seen a whole lot of the world and the impact and great suffering and great collaboration and achievement and all of these things, and obviously, we've had our own sets of personal experiences during this. And so my question is, in light of all that and looking back over this time, what have you seen or experienced that has left you with some hope for the future?

- The discomfort and surprise of progress, that cycle of, I don't want this, I don't want this, I don't think this is the right way. And then it turns out if people are open to it and curious by it, the discovery of things that work that we wouldn't choose ourselves, that gives me enormous hope. And seeing people change their minds definitely gives me hope. And my mind consistently being changed by others, much as I dislike it in some ways, it is very hopeful.

- Fantastic, well, Hannah Lindsley, thank you so much for joining me today. Thank you for sharing your experience. And again, a unique story, but universal experience, I think, in many ways. And thank you for sharing that. Thank you for bringing that sort of authenticity and honesty to this conversation. And thank you for everything you do for Indeed and helping people get jobs all over the world.

- Thank you for having me, it was a pleasure.

Share