Moneyball at the keyboard: Lessons on how to Scout Talented Developers
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Jonas, Adam
The central premise of Moneyball is that the collected wisdom of baseball insiders is subjective and flawed. Like baseball, the tech industry has a poor history of evaluating talent by favoring biased perspectives over objective analysis. As a baseball scout turned web developer and team lead, I will explore how the lessons I learned in my former career can enable us all to make better decisions on how to grow our teams and surface undervalued skills.
 This is one of those offbeat talks, so we're going to expect a lot of energy. Do the wave or something. We'll see. Welcome to Moneyball and the Keyboards, Gotting Talented Developers. I really do appreciate that you're joining me at this time slot. I know there's some awesome talks, and so I appreciate you being here because without you there's no talk. I'm Adam Jonas, I'm the Managing Director of Engineering at the Flatiron School, where our primary goal is to inspire people to fall in love with code. We train some of the best junior developers around, placing them at awesome companies like Kickstarter, Etsy, Intel, and the New York Times. I am not a teacher. My team has been working on transforming our internal learning management system into a full-fledged education platform. It's called Learn.co. We're really proud of it. I'd love you to check it out. But it's that work that really inspired me to start putting together these slides because I spent a lot of time thinking about how we could create some sort of metrics around how we could possibly surface talented students. So as I often do, I fell back to a sports analogy. Today we're going to talk about baseball. We're going to use baseball in the ways that talent is identified in baseball as an example of how we can improve our own mental model for thinking about talent and software. So why am I in particular talking about baseball? It's because thanks to the Flatiron School, software is actually my second career. I worked in a few different capacities in scouting player development for the Brewers, the Twins, and then ran an academy and spent most of my 20s doing this, living in and around Latin America and specifically the Dominican Republic. I moved back to the US in 2010 and tried my hand at founding a company that helped drafted players better improve their bargaining position and make more informed decisions about their futures. And while this venture went terribly, horribly wrong, it did pique my interest in code, which eventually landed me at the Flatiron School. And so here's what we're going to go over today. I'm going to tell you a little bit about the money ball philosophy and why it created such a shift in baseball. Then we're all going to go to scout school together and see how professional baseball talent has been traditionally scouted. And after that, we're going to see how talented students are identified at the Flatiron School. And finally, based on what we've learned, we're going to discuss three ways to better understand developer talent and how to better evaluate it going forward. If you're still awake at that point, we'll hopefully do some Q&A at the end. Cool. One more thing before we dive in, and there's lots to cover, but I think it's natural to sort of think about these sorts of conversations, specifically when thinking about hiring. But we evaluate talent all the time. We evaluate our coworkers because it has a lot to do with compensation. We evaluate our coworkers because it has a lot to do with clout on our team. We evaluate open source contributions and pedigrees and everything that comes across on a resume. We evaluate speakers and whether they're any good. We put value judgments on a lot of things. And so I think we're wired to do that, and that's totally okay. But I challenge you to just not limit your thoughts on this subject to just the interview process because it's a lot bigger than that. On with the show. So you might be familiar with Moneyball. It was a book written in 2003 by Michael Lewis and then turned into a movie with the always-handsome Brad Pitt in 2011. And the essence of Moneyball, while specifically focusing on baseball, is really about objectifying what was previously thought as subjective. And so the subject of Moneyball is the Oakland Athletics. I can see one of those hats right there. Awesome. And the fact that they have smaller revenues and constrained resources against other teams they're competing against on that right side of the graph, the New York Yankees, the Boston Red Sox, et cetera. And it's because of those constraints that Oakland is forced to search for undervalued players in the market. And they end up turning to these underutilized tools of statistical analysis. And so what did we learn from the Moneyball approach? We learned that industry outsiders who had not been indoctrinated with the traditional way of doing things could identify inefficiency in the old ways. And we learned that we could use better talent analysis tools to determine a player's objective value. So for an example, what's the difference between a player who gets to first base via single and a player that gets to first base via walk? Intuitively, we favor the person who took the action, the person that swung the bat and burned their way to first base. But from a team's contribution perspective, it's about the same. You got a guy on first base, that's about all it is. And so when we think about these contributions, we often will overvalue what we think is being earned or somehow has more merit to it. And this is exactly what was happening in baseball. We saw players with high batting averages were being overvalued and players with high walk percentages were being undervalued. And so the Moneyball approach took advantage of this, this gap between perception and reality, and it was more accurately able to value a player's contributions to the team. So this is Moneyball. It's this objective view, this data-driven view, this high-level view, this detached view. But it's not perfect because humans are complicated. And so let's turn to a little bit more of a humanistic approach by all heading to Scout School together. Scout School is a very real thing. It's sponsored by Major League Baseball, and I was lucky enough to attend in 2005. And you have to be invited by a team, and what you do is you spend two weeks spending the mornings going over the theory of scouting and the afternoons literally applying what you learned in the field, writing up reports and seeing different levels of the game. And so the first thing you learn is that all players are evaluated in the five following categories called tools. Hitting for average, hitting for power, running, fielding, throwing. This is it. This is what everything everyone is evaluated on. And players that possess elite talent in every tool, well, that's pretty rare. Willie Mays stands out as sort of the prototypical five-tool player in that he could do it all. But most players have strengths and weaknesses, and it's these strengths and weaknesses that help determine their role and the ways that they can contribute to the team. There's rubrics for all of these tools as well, and in Scout School they give it to you in this neat little binder. If only it was that easy. But let's go over the running tool in particular. And these grades range from two to eight as they do in all categories. And it's measured by the time that the bat hits the ball on contact to the time that they arrive at first base as measured by a stopwatch. Now, you can imagine there's some complicating factors here. First of all, a righty is actually physically further from first base than a lefty. Also you would imagine that baseball players being the people that they are are not going to run 100% every single time to first base. And so we need to look for the right situations to actually take down that time. Lazy Pop Flies, Gimmy Singles to left field, home run trots, probably not the right time to do that. And so while traditional scouting has this reputation of being purely subjective and sort of antithetical to the money ball approach, you can see how there's some standardization here. You can see that there's possibly some consistency. And I think you'd be surprised with how consistent the grades are from experience scout to scout. So the only way that this could work is if it were written down all the time. And baseball has done an amazing job of leaving this robust audit log for us to examine decisions over the years. This is an actual scouting report. You can see the rubric is there on the left. It's not particularly hardened because you're actually talking about all the tools. You also have the five star categories that we just went over, as well as some other categories to give you a little bit more context. So this lets us know why we're hiring this person or why we're passing on this person. It also lets us know where the candidate we see as we evaluate the candidate, where are they today versus where they could possibly project to the future. Same player, a little bit more exposition. And this provides a ton of data. Not only data on the candidate themselves, but data for us. This is not only about how we see the person that we're evaluating. It's also about the evaluator. And so this means that we're a lot less likely to succumb to hindsight bias where we claim that we saw it all along. At the end of the day, a scout's job is to predict the future. And it's hard. And it opens scouts to a ton of criticism. But leaving a record can be your proof and ultimately your redemption for naysayers. I want to focus on three particular statements of this. This is the good face. This is the first one. So the good face, pretty subjective, right? Why is that even in a scouting report? And the reason is, is that scouting, like a lot of other things, is pattern matching. And so this scout felt that he had seen faces like this in the past. They could possibly be on an all-star poster or the face of a franchise. And so he thought that this was data that he would include in the report. Will be in the big leagues by age 21. Pretty bold. You're going to go out on a limb for, this happens to be an 18-year-old, and say that they're going to make it to the highest level possible by the time they can legally drink. All the basic tools to be an outstanding shortstop prospect, best raw tools of any position player I've ever scouted, should only get better with maturity and experience, tools and makeup to be a star. This is in fact a report on Derek Jeter, who was drafted sixth overall in 1992. And he in fact did debut at age 21. And he was a 14-time all-star, which clearly he fulfilled the promise that was predicted for him by the scout. And so if you think that this Derek Jeter could possibly grow up to be this Derek Jeter, then you're going to want credibility for that. You're going to want someone saying that I wrote this down, this is what I put myself out there all those years ago, and guess what? I was right. Let's look at three other examples. Trevor Hoffman, future Hall of Fame pitcher, Albert Pujos, the best hitter of his generation, David Ortiz, cultural icon and three-time World Series champion. All these players were missed. Trevor Hoffman was selected 288th overall as a shortstop and then converted to a pitcher after two seasons, approving that he could in fact not hit. Albert Pujos was drafted 402nd overall, meaning that every single team had 13 opportunities to select the best player and failed. He was so good in fact, he was the fastest player of his draft year to make it to the major leagues. And he went on to win Rookie of the Year at age 21. In 2002, which is the season that Moneyball takes place, David Ortiz was let go by the Minnesota Twins for no compensation and he was picked up by the Red Sox the following winter and hit almost all of his 500 home runs for them. And so how did this happen? How did everyone miss these guys in an industry so focused on evaluating every single player? It's because they weren't seen in the proper context. Their talent was hidden by their circumstances. Hoffman was at the wrong position. Pujos was at a junior college that had never produced a major league or Teese had a myriad of nagging injuries which disguised his potential. And so let's take a step back and acknowledge that talent and context are inextricably late. There is fact no such thing as general talent. Derek Jener earned more than $400 million in his 20 year major league career, but it's not like you'd let him touch your VRC file or do your taxes. His talent, your talent, my talent, everyone's talent, they only exist in the context in which they're applied. I want to extend this a little bit further. Within the game, players are expected to contribute in ways determined by their roles. And so because of the defensive demands of a shortstop, fielding, throwing and running are considered the most important tools for prospects. If we look back at Derek Jeter's report, you can see he comes in at a 65 on fielding, a 60 on arm strength and a 70 on running, and is hitting in power great out to 50s major league average. And so he's the exact profile for what the scout is looking for in a shortstop, and you can tell by his gushing report. Then there's the first baseman. The first baseman only works 100 feet away, where the defensive demands of that position are just not as critical. And so they're expected to contribute in other ways, namely to carry the offensive side of the coin, power, hitting, that's what they're known for. And so it's really determined, your value is only determined by the role and the way that you can contribute to a team. And these are generous. These are people that are expected to contribute in multiple ways. There's also the specialist. In this case, the specialist is the pitcher. We only care about one tool of theirs. That's their arm. How hard can they throw? How well can they throw? How many outs can they get? This is a picture of Jim Abbott. And Jim Abbott had a 10-year major league career. He threw no hitter for the New York Yankees, and he won a gold medal for Team USA. What makes Jim Abbott a little bit different is that he was born without his right hand. And so if you met Jim Abbott on the street and tried to shake his hand, it might be difficult to imagine that he is a professional baseball player, let alone had this highly successful career. But it's because of the specialized nature of his role and his unique abilities that he was able to have the sustained success. And so given that context is so important, what's your context? Are you looking for a generalist to work on a team? Are you looking for individual contributors to work alone? Is this a small startup where flexibility and adaptability are valued or a large organization where optimization is most important? And so given that talent doesn't live in a vacuum, getting a handle on your unique set of circumstances and the circumstances that your team faces is critical to understanding what talent really is. Let me tell you a little bit about my context and the way that we identify students at the Flatiron School. And I think I sort of imagine this as a Petri dish. You can tell by my awesome graphic. The Flatiron School has contact with thousands of students a year. We have a 6% acceptance rate. And so over the course of the last three years, we've accepted hundreds of students. And all of them that have been job seeking, pretty much to a person, has in fact gotten a job. And so we have lots of conversations with employers. And when I started talking about this to the admissions department, I think I was sort of overwhelmed with the parallels to my time in the Dominican Republic, where a lot of players had very little training. And how do we know that they have talent before they really understand how to play the game? And so when we're tasked with trying to understand who has talent and who doesn't, it's really difficult because you have very little context to go on. And so what's their ceiling? How far can they go? That's hard. Remember the five tools of baseball? These are the five tools of the Flatiron School. Hireability, technical background, aptitude, passion, and culture. And so before anyone has written a line of code, this is what we're looking for. I want to go over these really quickly. Hireability. Our school is designed for a very specific outcome. And so when I say a successful student, I'm talking about a student that can go from no experience to employable in a really short period of time. They can also get their first job at a high starting salary, and they're promoted quickly, and their employer, of course, is giving us a lot of positive feedback on them. And so to be able to deliver on this promise that we're making to our students, we need to know that that outcome is actually something that they want. And so that's a big part of the admissions process. We also need to know that they can hold a conversation, that they can talk about times that they work together in a team, that they can project enthusiasm and energy, because all job interviews have some of those components in them. Technical background. This is exactly what it sounds like, how much have they done in the past? Aptitude. And you might see how some of these tools can sort of bleed together, sort of like a fast runner that can get down the line. That might boost their hit tool, because they can beat out some ground balls. And so how well do you think that this person can learn is what we're actually after an aptitude? How well can they integrate new information? We happen to use tic-tac-toe as our code challenge, and that's a highly Google-able challenge. Lots of our students are just pulling down solutions from the Internet, and so part of what we do is suss out the difference between what they actually wrote versus what they pulled from somewhere else. And I think that's okay. I think that's okay, because that's how I work. Not every line of code, not every way I attack a problem is from inside my own head, and so we should be using the tools and the approach that we expect to be using after they graduate. And so what we're looking for here is pattern matching. Can they see sameness and differences? What's the tiniest next step that they could take to improve? Here's our rubric. It ranges from one to five, starting with low effort and little comprehension, going all the way up to clean object-oriented code that's tested and clearly demonstrates that she's mastered the problem. We have exemplars for all these answers. Again, like baseball scouts, I think you'd be surprised by the consistency from instructor to instructor in their grades. Passion. So there's this sort of sycophie in nature to what we do, because when we run into code, sort of the learning never ends. We push the boulder up to the top of the mountain, and then we think we've got it, and then we realize we're just at the foot of the next mountain. And so we find that our successful students really enjoy the process. They even enjoy, I guess it'd be generous to say, the mental gymnastics. It's really the mental punishment of learning new things. And so we need to see that students have overcome difficult things in the past, and they've come out on the other side that they're really willing to push through those challenges. Culture. This is a tricky one. We don't admit students. We admit classes. And those classes need to be balanced, of course, in terms of gender and ethnic diversity, but they also need to be balanced in terms of background and diversity of perspective. If we have a bunch of finance folks who pass all of our criteria, we may ask some of them to defer, because they just sort of come from the same place. And we've seen that having a monoculture really impedes learning, motivation, and ultimately, collaboration. I'm going to throw this in, but it's a little anecdotal. There around 80% of our top students have some sort of creative outlet, whether that be professional or as a hobby. And so these are poets. These are musicians. These are photographers and painters. And it doesn't seem to really matter what the medium is, but they just seem to approach problems differently. And so this is something that we've discovered. We're still working on it, but it seems to exist. So here are some actual ratings. These are the actual ratings of our last class. Of course, without the names. And so by a show of hands, what do you think was the best predictor of success in terms of speed defining a job, starting salary, and employer satisfaction? Was it culture fit? No people. Was it passion? Some people. Was it technical background? Two brave souls. Higher ability. Awesome. And aptitude. Cool. It was in fact passion. And it was by a large margin. Passion was the best predictor of whether a candidate was going to succeed, and it wasn't even close. What didn't matter? Technical background. Technical experience had almost no correlation with how fast a student was hired or what their starting salary was or how fast they were promoted. But students eventually caught up with the experienced ones and then even surpassed them. And so it would be great if this rubric sort of worked for everyone, but unfortunately we aren't all operating within the same context. And so you need something that's tailored to your situation. And so what's important to you? What are you thinking about? And there are some places that you might look to determine your tools. Performance reviews, although those can really be a minefield if you guys aren't doing them right. Student statements, value statements. Sometimes those are updated, sometimes they aren't. One-on-ones and retros. What's your team actually thinking about and talking about? And so ultimately you can't be successful in evaluating talent without knowing what you value and what's important to you. Okay. Everything we've been talking about has been pretty abstract. Let's discuss three things that will help you better think about talent starting today and maybe discover some of those diamonds in the rough. Admit we have no idea where talent comes from. Control for sample size bias and rethink cultural fit. Okay. So Trevor Hoffman was drafted as a shortstop. He's converted to this pitcher. Is it outside the realm of possibility that you might come across someone who was in a generalist role and if they could just focus on the one thing they did particularly well they could excel. What about Albert Pujols? Albert Pujols got drafted from Maple Woods Community College. If you got a resume from Maple Woods Community College, how would you feel about that? Is that something that you would give a lot of attention to? Maybe the physical injury analogy doesn't really work here in David Ortiz's case. What if someone had a number of personal circumstances that were affecting their performance? What if they were going through a divorce? What if they had a three-month-old who was conducting some sort of twisted sleep deprivation experiment on him? This is my guy. He is really messing with me. But I do miss him. Anyway, so we're so bad at separating context from personal attribution that there's a term for it, it's called fundamental attribution error. And so baseball is really bad at this too. There are 600 players playing in the major leagues at one time and 72 of those players play in the All-Star game. So that's the top 12%. Only a third of those are the slam dunk first round prospects that everyone saw coming. And so in reality if a prospect whether that be in baseball or in software is projected to be a star, guess who else knows? Everyone. And if you're Google and if you're Facebook and if you're Amazon, you can afford to throw a ton of money at everyone and hope that they develop. But for the rest of us that need to live in reality, we need to be looking for candidates that fall through the cracks. And they may fall through the cracks for a number of reasons. But ultimately they've been viewed through the conventional lens where we judge their talent and that might emphasize their weaknesses and obscure their strengths. And so this is really what Moneyball boils down to. It's looking where others aren't. The second thing is sample size bias. So we're wired to make snap judgments. It's no matter how thoughtful or kind we think we might be, we still do it. And our goal is to create as many touch points as possible without necessarily dedicating more time. It took baseball a really long time to come to the realization that one game was just a snapshot. And because the software industry is much, much younger by comparison, I think we're still conceited enough that we can, we think that we can judge someone's talent based on one interview. But let's admit that we're worse at this than we actually think we are. So creating a fuller picture of the process, creating a fuller picture is essential part of the process. And this is how we currently do it. We got whiteboard challenges. We got phone screens. We have open source commits. We have looking at their blogs and resumes. I personally like looking at blogs and open source commits and any sort of evidence that demonstrates consistent long term behavior. Pairing has also been mentioned as one of the better methods since it's actually simulates the job that someone might be looking to get or working to work in. But ultimately these are all data points and they need to be treated as such. And in my opinion, what we're really after is, how much does she love the work? Is he creative? And will she make your team better? Traditional interview screens can do this. We just have to be asking the right questions. As can whiteboard challenges or any other data points that we use as long as we're taking the macro view as opposed to the sort of one game snapshot perspective. Cultural fit. Cultural fit is hard. And I think it's become a little bit of a mess. What we're starting to screen for is much less of a match with organizational values and much more of a match for personal fit. And so I think it's natural to bond over shared experiences and pedigrees and if you went to the same college you can play the name game and that helps us feel connected. But cultural fit has become this new form of discrimination all in the name of employee enjoyment and fun. And so what are we actually looking for here? Are we looking for someone that we want to have a beer with or are we looking for someone that we want to work with? To turn this around we're going to need some serious self-reflection because the research says that more diverse teams outperform less diverse ones. For jobs involving complex decisions and creativity, more diverse teams outperform less diverse ones. We're more confident in homogeneous groups because that makes us feel comfortable but they just don't perform as well. And so to make this happen we need to confront our biases head on. It's scary. It's really scary to admit that we might have personal biases or institutional biases but the data is likely there if you have the courage to take a look at it. We're comfortable with what we know for what we've seen in the past. Faces like ours, backgrounds like ours, skills like ours. But is comfort really enough? Is that good enough criteria? This is Jackie Robinson signing his first contract to become the first black player in the major leagues. Now Jackie was signed by the Dodgers for one reason and one reason only and that was to win. He of course went on to have a great career, a Hall of Fame career but from a hiring perspective the Dodgers got access to a talent pipeline that no one else was willing to take advantage of. And so yes it was the right thing to do from a social consciousness perspective but it was also a huge competitive advantage. Hiring people from non-traditional backgrounds takes a lot of guts but it's one way to take advantage of a developer hiring market inefficiency. These players offered incredible value to the organizations that took chances on them. And yes, high priced folks might be worth their resources but you just can't afford all the most expensive players all the time unless you are the New York Yankees or Boston Red Sox. And so talent needs an opportunity from you. Being more objective by writing things down, establishing rubrics for rating guidelines, introspecting on our own scouting abilities, not letting cultural screens be prohibitive, just generally getting away from going with our gut means that we are going to be a more inclusive industry and ultimately push us towards a more complete meritocracy. Thanks. How else can we look at technical background? Yeah, so actually my team, we have a different set of tools. I knew this was going to come up. It is communication, ownership, adaptability, velocity and quality. And so with some, given the small team that we have and the speed that we are moving, those are the things that we look for. We have our own rubrics and I do grade people out when I interview them. But this is what has worked for the school and again, it is really based on your context. Yeah, oof. I don't. No, no, I don't. I think we should, I think it is sort of built in because a lot of, we do hire quite a bit out of our own school too because they get to go through the program and then we know who awesome developers are because they just graduated. But in terms of hiring from the outside, yeah, I think that is a reasonable way to approach it and it is not something that I currently do. Sure. So the question was when evaluating culture fit, how different is this person than the rest of the organization? Yeah. Yeah, I think we just have to be really careful of making culture fit like this prohibitive screen. Like there is, I mean, we use Greenhouse as our software within the school and the culture screen is one interview. And we also do the sort of Gladiator style like yay or nay thing which I think works out pretty well. But it is really up to the person doing the screen. Oftentimes that is our recruiting manager. For my team in particular, I think I have, it is up to me to make those calls. But when I am looking to round out my team, a lot of it is thinking about what we don't have. So is this a person that is going to make our team better? Are they creative? How much do they love this and how much are they going to develop? But going back to this sort of present versus future, passion just drives everything. And so if that is the case and they really want to get better, they are going to get better. Yeah, it is somewhere in between, right? So we, I was a student in the first class of Flatiron and they didn't really know if it was going to work. It was sort of just like 20 people in a loft and the founder just telling us stuff. It worked for me. So that works. But now we have an actual system and we are using the product that I am working on to sort of scale that out. And we have seen things like before we had the system in place versus after they can consume twice the amount of material. And so we are able to cover some things that we just didn't cover before. And so we give them much more of an education. But yeah, I think it is buyer beware, right? Boot camps are a thing. They weren't really when I was going through the process. But they have changed a lot of people's lives. So that seems to be good. But people that are getting oversold, like did they do enough research or could they not get into a more prestigious program? It's hard. Yeah. We have about as long term as me. I was in that first class. So yeah, we are up to a little over three years. We do. We are really proud to have an audited jobs report where we go to every single student. And this is a third party consulting party that we pull in and actually they contact every student, talk about their salary, talk about their prospects. As far as I understand we are the only school that is doing that. So yes, we are taking that data very seriously. I guess what I'm wondering is that maybe you could start the same thing that made you good for your team? Yeah. I don't know if that's true. But given my answer to the previous question is I use a different rubric for hiring on my team. I think it really is context dependent. And so if you're talking, you know, if we're putting someone at Intel, well they're going to clearly have a different hiring standard than we would for screening our students. Thank you everyone. I really appreciate being here. Thank you. Thank you. Thank you.
 This is one of those offbeat talks, so we're going to expect a lot of energy.  Do the wave or something.  We'll see.  Welcome to Moneyball and the Keyboards, Gotting Talented Developers.  I really do appreciate that you're joining me at this time slot.  I know there's some awesome talks, and so I appreciate you being here because without  you there's no talk.  I'm Adam Jonas, I'm the Managing Director of Engineering at the Flatiron School, where  our primary goal is to inspire people to fall in love with code.  We train some of the best junior developers around, placing them at awesome companies  like Kickstarter, Etsy, Intel, and the New York Times.  I am not a teacher.  My team has been working on transforming our internal learning management system into  a full-fledged education platform.  It's called Learn.co.  We're really proud of it.  I'd love you to check it out.  But it's that work that really inspired me to start putting together these slides because  I spent a lot of time thinking about how we could create some sort of metrics around  how we could possibly surface talented students.  So as I often do, I fell back to a sports analogy.  Today we're going to talk about baseball.  We're going to use baseball in the ways that talent is identified in baseball as an example  of how we can improve our own mental model for thinking about talent and software.  So why am I in particular talking about baseball?  It's because thanks to the Flatiron School, software is actually my second career.  I worked in a few different capacities in scouting player development for the Brewers,  the Twins, and then ran an academy and spent most of my 20s doing this, living in and around  Latin America and specifically the Dominican Republic.  I moved back to the US in 2010 and tried my hand at founding a company that helped drafted  players better improve their bargaining position and make more informed decisions about their  futures.  And while this venture went terribly, horribly wrong, it did pique my interest in code, which  eventually landed me at the Flatiron School.  And so here's what we're going to go over today.  I'm going to tell you a little bit about the money ball philosophy and why it created  such a shift in baseball.  Then we're all going to go to scout school together and see how professional baseball  talent has been traditionally scouted.  And after that, we're going to see how talented students are identified at the Flatiron School.  And finally, based on what we've learned, we're going to discuss three ways to better  understand developer talent and how to better evaluate it going forward.  If you're still awake at that point, we'll hopefully do some Q&A at the end.  Cool.  One more thing before we dive in, and there's lots to cover, but I think it's natural  to sort of think about these sorts of conversations, specifically when thinking about hiring.  But we evaluate talent all the time.  We evaluate our coworkers because it has a lot to do with compensation.  We evaluate our coworkers because it has a lot to do with clout on our team.  We evaluate open source contributions and pedigrees and everything that comes across  on a resume.  We evaluate speakers and whether they're any good.  We put value judgments on a lot of things.  And so I think we're wired to do that, and that's totally okay.  But I challenge you to just not limit your thoughts on this subject to just the interview  process because it's a lot bigger than that.  On with the show.  So you might be familiar with Moneyball.  It was a book written in 2003 by Michael Lewis and then turned into a movie with the always-handsome  Brad Pitt in 2011.  And the essence of Moneyball, while specifically focusing on baseball, is really about objectifying  what was previously thought as subjective.  And so the subject of Moneyball is the Oakland Athletics.  I can see one of those hats right there.  Awesome.  And the fact that they have smaller revenues and constrained resources against other teams  they're competing against on that right side of the graph, the New York Yankees, the Boston  Red Sox, et cetera.  And it's because of those constraints that Oakland is forced to search for undervalued  players in the market.  And they end up turning to these underutilized tools of statistical analysis.  And so what did we learn from the Moneyball approach?  We learned that industry outsiders who had not been indoctrinated with the traditional  way of doing things could identify inefficiency in the old ways.  And we learned that we could use better talent analysis tools to determine a player's objective  value.  So for an example, what's the difference between a player who gets to first base via  single and a player that gets to first base via walk?  Intuitively, we favor the person who took the action, the person that swung the bat  and burned their way to first base.  But from a team's contribution perspective, it's about the same.  You got a guy on first base, that's about all it is.  And so when we think about these contributions, we often will overvalue what we think is being  earned or somehow has more merit to it.  And this is exactly what was happening in baseball.  We saw players with high batting averages were being overvalued and players with high  walk percentages were being undervalued.  And so the Moneyball approach took advantage of this, this gap between perception and reality,  and it was more accurately able to value a player's contributions to the team.  So this is Moneyball.  It's this objective view, this data-driven view, this high-level view, this detached  view.  But it's not perfect because humans are complicated.  And so let's turn to a little bit more of a humanistic approach by all heading to Scout  School together.  Scout School is a very real thing.  It's sponsored by Major League Baseball, and I was lucky enough to attend in 2005.  And you have to be invited by a team, and what you do is you spend two weeks spending  the mornings going over the theory of scouting and the afternoons literally applying what  you learned in the field, writing up reports and seeing different levels of the game.  And so the first thing you learn is that all players are evaluated in the five following  categories called tools.  Hitting for average, hitting for power, running, fielding, throwing.  This is it.  This is what everything everyone is evaluated on.  And players that possess elite talent in every tool, well, that's pretty rare.  Willie Mays stands out as sort of the prototypical five-tool player in that he could do it all.  But most players have strengths and weaknesses, and it's these strengths and weaknesses that  help determine their role and the ways that they can contribute to the team.  There's rubrics for all of these tools as well, and in Scout School they give it to  you in this neat little binder.  If only it was that easy.  But let's go over the running tool in particular.  And these grades range from two to eight as they do in all categories.  And it's measured by the time that the bat hits the ball on contact to the time that  they arrive at first base as measured by a stopwatch.  Now, you can imagine there's some complicating factors here.  First of all, a righty is actually physically further from first base than a lefty.  Also you would imagine that baseball players being the people that they are are not going  to run 100% every single time to first base.  And so we need to look for the right situations to actually take down that time.  Lazy Pop Flies, Gimmy Singles to left field, home run trots, probably not the right time  to do that.  And so while traditional scouting has this reputation of being purely subjective and  sort of antithetical to the money ball approach, you can see how there's some standardization  here.  You can see that there's possibly some consistency.  And I think you'd be surprised with how consistent the grades are from experience scout to scout.  So the only way that this could work is if it were written down all the time.  And baseball has done an amazing job of leaving this robust audit log for us to examine decisions  over the years.  This is an actual scouting report.  You can see the rubric is there on the left.  It's not particularly hardened because you're actually talking about all the tools.  You also have the five star categories that we just went over, as well as some other categories  to give you a little bit more context.  So this lets us know why we're hiring this person or why we're passing on this person.  It also lets us know where the candidate we see as we evaluate the candidate, where are  they today versus where they could possibly project to the future.  Same player, a little bit more exposition.  And this provides a ton of data.  Not only data on the candidate themselves, but data for us.  This is not only about how we see the person that we're evaluating.  It's also about the evaluator.  And so this means that we're a lot less likely to succumb to hindsight bias where we claim  that we saw it all along.  At the end of the day, a scout's job is to predict the future.  And it's hard.  And it opens scouts to a ton of criticism.  But leaving a record can be your proof and ultimately your redemption for naysayers.  I want to focus on three particular statements of this.  This is the good face.  This is the first one.  So the good face, pretty subjective, right?  Why is that even in a scouting report?  And the reason is, is that scouting, like a lot of other things, is pattern matching.  And so this scout felt that he had seen faces like this in the past.  They could possibly be on an all-star poster or the face of a franchise.  And so he thought that this was data that he would include in the report.  Will be in the big leagues by age 21.  Pretty bold.  You're going to go out on a limb for, this happens to be an 18-year-old, and say that  they're going to make it to the highest level possible by the time they can legally drink.  All the basic tools to be an outstanding shortstop prospect, best raw tools of any position player  I've ever scouted, should only get better with maturity and experience, tools and makeup  to be a star.  This is in fact a report on Derek Jeter, who was drafted sixth overall in 1992.  And he in fact did debut at age 21.  And he was a 14-time all-star, which clearly he fulfilled the promise that was predicted  for him by the scout.  And so if you think that this Derek Jeter could possibly grow up to be this Derek Jeter,  then you're going to want credibility for that.  You're going to want someone saying that I wrote this down, this is what I put myself  out there all those years ago, and guess what?  I was right.  Let's look at three other examples.  Trevor Hoffman, future Hall of Fame pitcher, Albert Pujos, the best hitter of his generation,  David Ortiz, cultural icon and three-time World Series champion.  All these players were missed.  Trevor Hoffman was selected 288th overall as a shortstop and then converted to a pitcher  after two seasons, approving that he could in fact not hit.  Albert Pujos was drafted 402nd overall, meaning that every single team had 13 opportunities  to select the best player and failed.  He was so good in fact, he was the fastest player of his draft year to make it to the  major leagues.  And he went on to win Rookie of the Year at age 21.  In 2002, which is the season that Moneyball takes place, David Ortiz was let go by the  Minnesota Twins for no compensation and he was picked up by the Red Sox the following  winter and hit almost all of his 500 home runs for them.  And so how did this happen?  How did everyone miss these guys in an industry so focused on evaluating every single player?  It's because they weren't seen in the proper context.  Their talent was hidden by their circumstances.  Hoffman was at the wrong position.  Pujos was at a junior college that had never produced a major league or Teese had a myriad  of nagging injuries which disguised his potential.  And so let's take a step back and acknowledge that talent and context are inextricably late.  There is fact no such thing as general talent.  Derek Jener earned more than $400 million in his 20 year major league career, but it's  not like you'd let him touch your VRC file or do your taxes.  His talent, your talent, my talent, everyone's talent, they only exist in the context in  which they're applied.  I want to extend this a little bit further.  Within the game, players are expected to contribute in ways determined by their roles.  And so because of the defensive demands of a shortstop, fielding, throwing and running  are considered the most important tools for prospects.  If we look back at Derek Jeter's report, you can see he comes in at a 65 on fielding, a  60 on arm strength and a 70 on running, and is hitting in power great out to 50s major  league average.  And so he's the exact profile for what the scout is looking for in a shortstop, and you  can tell by his gushing report.  Then there's the first baseman.  The first baseman only works 100 feet away, where the defensive demands of that position  are just not as critical.  And so they're expected to contribute in other ways, namely to carry the offensive side of  the coin, power, hitting, that's what they're known for.  And so it's really determined, your value is only determined by the role and the way  that you can contribute to a team.  And these are generous.  These are people that are expected to contribute in multiple ways.  There's also the specialist.  In this case, the specialist is the pitcher.  We only care about one tool of theirs.  That's their arm.  How hard can they throw?  How well can they throw?  How many outs can they get?  This is a picture of Jim Abbott.  And Jim Abbott had a 10-year major league career.  He threw no hitter for the New York Yankees, and he won a gold medal for Team USA.  What makes Jim Abbott a little bit different is that he was born without his right hand.  And so if you met Jim Abbott on the street and tried to shake his hand, it might be difficult  to imagine that he is a professional baseball player, let alone had this highly successful  career.  But it's because of the specialized nature of his role and his unique abilities that  he was able to have the sustained success.  And so given that context is so important, what's your context?  Are you looking for a generalist to work on a team?  Are you looking for individual contributors to work alone?  Is this a small startup where flexibility and adaptability are valued or a large organization  where optimization is most important?  And so given that talent doesn't live in a vacuum, getting a handle on your unique  set of circumstances and the circumstances that your team faces is critical to understanding  what talent really is.  Let me tell you a little bit about my context and the way that we identify students at the  Flatiron School.  And I think I sort of imagine this as a Petri dish.  You can tell by my awesome graphic.  The Flatiron School has contact with thousands of students a year.  We have a 6% acceptance rate.  And so over the course of the last three years, we've accepted hundreds of students.  And all of them that have been job seeking, pretty much to a person, has in fact gotten  a job.  And so we have lots of conversations with employers.  And when I started talking about this to the admissions department, I think I was sort  of overwhelmed with the parallels to my time in the Dominican Republic, where a lot of  players had very little training.  And how do we know that they have talent before they really understand how to play the game?  And so when we're tasked with trying to understand who has talent and who doesn't, it's really  difficult because you have very little context to go on.  And so what's their ceiling?  How far can they go?  That's hard.  Remember the five tools of baseball?  These are the five tools of the Flatiron School.  Hireability, technical background, aptitude, passion, and culture.  And so before anyone has written a line of code, this is what we're looking for.  I want to go over these really quickly.  Hireability.  Our school is designed for a very specific outcome.  And so when I say a successful student, I'm talking about a student that can go from no  experience to employable in a really short period of time.  They can also get their first job at a high starting salary, and they're promoted quickly,  and their employer, of course, is giving us a lot of positive feedback on them.  And so to be able to deliver on this promise that we're making to our students, we need  to know that that outcome is actually something that they want.  And so that's a big part of the admissions process.  We also need to know that they can hold a conversation, that they can talk about times  that they work together in a team, that they can project enthusiasm and energy, because  all job interviews have some of those components in them.  Technical background.  This is exactly what it sounds like, how much have they done in the past?  Aptitude.  And you might see how some of these tools can sort of bleed together, sort of like a fast  runner that can get down the line.  That might boost their hit tool, because they can beat out some ground balls.  And so how well do you think that this person can learn is what we're actually after an  aptitude?  How well can they integrate new information?  We happen to use tic-tac-toe as our code challenge, and that's a highly Google-able challenge.  Lots of our students are just pulling down solutions from the Internet, and so part of  what we do is suss out the difference between what they actually wrote versus what they  pulled from somewhere else.  And I think that's okay.  I think that's okay, because that's how I work.  Not every line of code, not every way I attack a problem is from inside my own head, and  so we should be using the tools and the approach that we expect to be using after they graduate.  And so what we're looking for here is pattern matching.  Can they see sameness and differences?  What's the tiniest next step that they could take to improve?  Here's our rubric.  It ranges from one to five, starting with low effort and little comprehension, going all  the way up to clean object-oriented code that's tested and clearly demonstrates that she's  mastered the problem.  We have exemplars for all these answers.  Again, like baseball scouts, I think you'd be surprised by the consistency from instructor  to instructor in their grades.  Passion.  So there's this sort of sycophie in nature to what we do, because when we run into code,  sort of the learning never ends.  We push the boulder up to the top of the mountain, and then we think we've got it, and then we  realize we're just at the foot of the next mountain.  And so we find that our successful students really enjoy the process.  They even enjoy, I guess it'd be generous to say, the mental gymnastics.  It's really the mental punishment of learning new things.  And so we need to see that students have overcome difficult things in the past, and they've  come out on the other side that they're really willing to push through those challenges.  Culture.  This is a tricky one.  We don't admit students.  We admit classes.  And those classes need to be balanced, of course, in terms of gender and ethnic diversity,  but they also need to be balanced in terms of background and diversity of perspective.  If we have a bunch of finance folks who pass all of our criteria, we may ask some of them  to defer, because they just sort of come from the same place.  And we've seen that having a monoculture really impedes learning, motivation, and ultimately,  collaboration.  I'm going to throw this in, but it's a little anecdotal.  There around 80% of our top students have some sort of creative outlet, whether that  be professional or as a hobby.  And so these are poets.  These are musicians.  These are photographers and painters.  And it doesn't seem to really matter what the medium is, but they just seem to approach  problems differently.  And so this is something that we've discovered.  We're still working on it, but it seems to exist.  So here are some actual ratings.  These are the actual ratings of our last class.  Of course, without the names.  And so by a show of hands, what do you think was the best predictor of success in terms  of speed defining a job, starting salary, and employer satisfaction?  Was it culture fit?  No people.  Was it passion?  Some people.  Was it technical background?  Two brave souls.  Higher ability.  Awesome.  And aptitude.  Cool.  It was in fact passion.  And it was by a large margin.  Passion was the best predictor of whether a candidate was going to succeed, and it wasn't  even close.  What didn't matter?  Technical background.  Technical experience had almost no correlation with how fast a student was hired or what  their starting salary was or how fast they were promoted.  But students eventually caught up with the experienced ones and then even surpassed them.  And so it would be great if this rubric sort of worked for everyone, but unfortunately  we aren't all operating within the same context.  And so you need something that's tailored to your situation.  And so what's important to you?  What are you thinking about?  And there are some places that you might look to determine your tools.  Performance reviews, although those can really be a minefield if you guys aren't doing them  right.  Student statements, value statements.  Sometimes those are updated, sometimes they aren't.  One-on-ones and retros.  What's your team actually thinking about and talking about?  And so ultimately you can't be successful in evaluating talent without knowing what  you value and what's important to you.  Okay.  Everything we've been talking about has been pretty abstract.  Let's discuss three things that will help you better think about talent starting today  and maybe discover some of those diamonds in the rough.  Admit we have no idea where talent comes from.  Control for sample size bias and rethink cultural fit.  Okay.  So Trevor Hoffman was drafted as a shortstop.  He's converted to this pitcher.  Is it outside the realm of possibility that you might come across someone who was in a  generalist role and if they could just focus on the one thing they did particularly well  they could excel.  What about Albert Pujols?  Albert Pujols got drafted from Maple Woods Community College.  If you got a resume from Maple Woods Community College, how would you feel about that?  Is that something that you would give a lot of attention to?  Maybe the physical injury analogy doesn't really work here in David Ortiz's case.  What if someone had a number of personal circumstances that were affecting their performance?  What if they were going through a divorce?  What if they had a three-month-old who was conducting some sort of twisted sleep deprivation  experiment on him?  This is my guy.  He is really messing with me.  But I do miss him.  Anyway, so we're so bad at separating context from personal attribution that there's a term  for it, it's called fundamental attribution error.  And so baseball is really bad at this too.  There are 600 players playing in the major leagues at one time and 72 of those players  play in the All-Star game.  So that's the top 12%.  Only a third of those are the slam dunk first round prospects that everyone saw coming.  And so in reality if a prospect whether that be in baseball or in software is projected  to be a star, guess who else knows?  Everyone.  And if you're Google and if you're Facebook and if you're Amazon, you can afford to throw  a ton of money at everyone and hope that they develop.  But for the rest of us that need to live in reality, we need to be looking for candidates  that fall through the cracks.  And they may fall through the cracks for a number of reasons.  But ultimately they've been viewed through the conventional lens where we judge their  talent and that might emphasize their weaknesses and obscure their strengths.  And so this is really what Moneyball boils down to.  It's looking where others aren't.  The second thing is sample size bias.  So we're wired to make snap judgments.  It's no matter how thoughtful or kind we think we might be, we still do it.  And our goal is to create as many touch points as possible without necessarily dedicating  more time.  It took baseball a really long time to come to the realization that one game was just  a snapshot.  And because the software industry is much, much younger by comparison, I think we're  still conceited enough that we can, we think that we can judge someone's talent based on  one interview.  But let's admit that we're worse at this than we actually think we are.  So creating a fuller picture of the process, creating a fuller picture is essential part  of the process.  And this is how we currently do it.  We got whiteboard challenges.  We got phone screens.  We have open source commits.  We have looking at their blogs and resumes.  I personally like looking at blogs and open source commits and any sort of evidence that  demonstrates consistent long term behavior.  Pairing has also been mentioned as one of the better methods since it's actually simulates  the job that someone might be looking to get or working to work in.  But ultimately these are all data points and they need to be treated as such.  And in my opinion, what we're really after is, how much does she love the work?  Is he creative?  And will she make your team better?  Traditional interview screens can do this.  We just have to be asking the right questions.  As can whiteboard challenges or any other data points that we use as long as we're taking  the macro view as opposed to the sort of one game snapshot perspective.  Cultural fit.  Cultural fit is hard.  And I think it's become a little bit of a mess.  What we're starting to screen for is much less of a match with organizational values  and much more of a match for personal fit.  And so I think it's natural to bond over shared experiences and pedigrees and if you  went to the same college you can play the name game and that helps us feel connected.  But cultural fit has become this new form of discrimination all in the name of employee  enjoyment and fun.  And so what are we actually looking for here?  Are we looking for someone that we want to have a beer with or are we looking for someone  that we want to work with?  To turn this around we're going to need some serious self-reflection because the research  says that more diverse teams outperform less diverse ones.  For jobs involving complex decisions and creativity, more diverse teams outperform less diverse  ones.  We're more confident in homogeneous groups because that makes us feel comfortable but  they just don't perform as well.  And so to make this happen we need to confront our biases head on.  It's scary.  It's really scary to admit that we might have personal biases or institutional biases but  the data is likely there if you have the courage to take a look at it.  We're comfortable with what we know for what we've seen in the past.  Faces like ours, backgrounds like ours, skills like ours.  But is comfort really enough?  Is that good enough criteria?  This is Jackie Robinson signing his first contract to become the first black player in  the major leagues.  Now Jackie was signed by the Dodgers for one reason and one reason only and that was to  win.  He of course went on to have a great career, a Hall of Fame career but from a hiring perspective  the Dodgers got access to a talent pipeline that no one else was willing to take advantage  of.  And so yes it was the right thing to do from a social consciousness perspective but it  was also a huge competitive advantage.  Hiring people from non-traditional backgrounds takes a lot of guts but it's one way to take  advantage of a developer hiring market inefficiency.  These players offered incredible value to the organizations that took chances on them.  And yes, high priced folks might be worth their resources but you just can't afford  all the most expensive players all the time unless you are the New York Yankees or Boston  Red Sox.  And so talent needs an opportunity from you.  Being more objective by writing things down, establishing rubrics for rating guidelines,  introspecting on our own scouting abilities, not letting cultural screens be prohibitive,  just generally getting away from going with our gut means that we are going to be a more  inclusive industry and ultimately push us towards a more complete meritocracy.  Thanks.  How else can we look at technical background?  Yeah, so actually my team, we have a different set of tools.  I knew this was going to come up.  It is communication, ownership, adaptability, velocity and quality.  And so with some, given the small team that we have and the speed that we are moving,  those are the things that we look for.  We have our own rubrics and I do grade people out when I interview them.  But this is what has worked for the school and again, it is really based on your context.  Yeah, oof.  I don't.  No, no, I don't.  I think we should, I think it is sort of built in because a lot of, we do hire quite  a bit out of our own school too because they get to go through the program and then we  know who awesome developers are because they just graduated.  But in terms of hiring from the outside, yeah, I think that is a reasonable way to approach  it and it is not something that I currently do.  Sure.  So the question was when evaluating culture fit, how different is this person than the  rest of the organization?  Yeah.  Yeah, I think we just have to be really careful of making culture fit like this prohibitive  screen.  Like there is, I mean, we use Greenhouse as our software within the school and the culture  screen is one interview.  And we also do the sort of Gladiator style like yay or nay thing which I think works  out pretty well.  But it is really up to the person doing the screen.  Oftentimes that is our recruiting manager.  For my team in particular, I think I have, it is up to me to make those calls.  But when I am looking to round out my team, a lot of it is thinking about what we don't  have.  So is this a person that is going to make our team better?  Are they creative?  How much do they love this and how much are they going to develop?  But going back to this sort of present versus future, passion just drives everything.  And so if that is the case and they really want to get better, they are going to get  better.  Yeah, it is somewhere in between, right?  So we, I was a student in the first class of Flatiron and they didn't really know if  it was going to work.  It was sort of just like 20 people in a loft and the founder just telling us stuff.  It worked for me.  So that works.  But now we have an actual system and we are using the product that I am working on to  sort of scale that out.  And we have seen things like before we had the system in place versus after they can  consume twice the amount of material.  And so we are able to cover some things that we just didn't cover before.  And so we give them much more of an education.  But yeah, I think it is buyer beware, right?  Boot camps are a thing.  They weren't really when I was going through the process.  But they have changed a lot of people's lives.  So that seems to be good.  But people that are getting oversold, like did they do enough research or could they  not get into a more prestigious program?  It's hard.  Yeah.  We have about as long term as me.  I was in that first class.  So yeah, we are up to a little over three years.  We do.  We are really proud to have an audited jobs report where we go to every single student.  And this is a third party consulting party that we pull in and actually they contact  every student, talk about their salary, talk about their prospects.  As far as I understand we are the only school that is doing that.  So yes, we are taking that data very seriously.  I guess what I'm wondering is that maybe you could start the same thing that made you  good for your team?  Yeah.  I don't know if that's true.  But given my answer to the previous question is I use a different rubric for hiring on  my team.  I think it really is context dependent.  And so if you're talking, you know, if we're putting someone at Intel, well they're going  to clearly have a different hiring standard than we would for screening our students.  Thank you everyone.  I really appreciate being here.  Thank you.  Thank you.  Thank you.
Moneyball_at_the_keyboard_0_47.png Moneyball_at_the_keyboard_48_263.png Moneyball_at_the_keyboard_264_371.png Moneyball_at_the_keyboard_372_983.png Moneyball_at_the_keyboard_984_2207.png Moneyball_at_the_keyboard_2208_3035.png Moneyball_at_the_keyboard_3036_3575.png Moneyball_at_the_keyboard_3576_5975.png Moneyball_at_the_keyboard_5976_10355.png Moneyball_at_the_keyboard_10356_12311.png Moneyball_at_the_keyboard_12312_15851.png Moneyball_at_the_keyboard_15852_18371.png Moneyball_at_the_keyboard_18372_19007.png Moneyball_at_the_keyboard_19008_19391.png Moneyball_at_the_keyboard_19392_19835.png Moneyball_at_the_keyboard_19836_24023.png Moneyball_at_the_keyboard_24024_25631.png Moneyball_at_the_keyboard_25632_28799.png Moneyball_at_the_keyboard_28800_28991.png Moneyball_at_the_keyboard_28992_33851.png Moneyball_at_the_keyboard_33852_35051.png Moneyball_at_the_keyboard_35052_35507.png Moneyball_at_the_keyboard_35508_37163.png Moneyball_at_the_keyboard_37164_42083.png Moneyball_at_the_keyboard_42084_42839.png Moneyball_at_the_keyboard_42840_44495.png Moneyball_at_the_keyboard_44496_44555.png Moneyball_at_the_keyboard_44556_44699.png Moneyball_at_the_keyboard_44700_44795.png Moneyball_at_the_keyboard_44796_45179.png Moneyball_at_the_keyboard_45180_45515.png Moneyball_at_the_keyboard_45516_45839.png Moneyball_at_the_keyboard_45840_46115.png Moneyball_at_the_keyboard_46116_46271.png Moneyball_at_the_keyboard_46272_46631.png Moneyball_at_the_keyboard_46632_46691.png Moneyball_at_the_keyboard_46692_46859.png Moneyball_at_the_keyboard_46860_47027.png Moneyball_at_the_keyboard_47028_47339.png Moneyball_at_the_keyboard_47340_47411.png Moneyball_at_the_keyboard_47412_47507.png Moneyball_at_the_keyboard_47508_47651.png Moneyball_at_the_keyboard_47652_48395.png Moneyball_at_the_keyboard_48396_48467.png Moneyball_at_the_keyboard_48468_49403.png Moneyball_at_the_keyboard_49404_50747.png Moneyball_at_the_keyboard_50748_51407.png Moneyball_at_the_keyboard_51408_51503.png Moneyball_at_the_keyboard_51504_51551.png Moneyball_at_the_keyboard_51552_51755.png Moneyball_at_the_keyboard_51756_51864.png