<jupyter_start><jupyter_text>An analysis of the *Mendoza Line* in MLB batting statistics.-----The *Mendoza Line* is common U.S. slang referring to the threshold for seriously below average performance.The term originated in baseball, referring to the batting average of shortstop <NAME>.For those unfamiliar with the origin of the term, there is good background in the [wikipedia entry on the Mendoza Line] and [this column] from the St. Louis Post-Dispatch.  The term has made Mendoza's last name famous since it was first coined in 1979, but we should verify the figure and analyze where this level of performance falls in the spectrum of other major league batters.In addition, we'll look at how batting averages over time compare to this figure.The data used in this analysis comes from SeanLahman.com's [baseball database](http://www.seanlahman.com/baseball-archive/statistics/).[wikipedia entry on the Mendoza Line]: https://en.wikipedia.org/wiki/Mendoza_Line[this column]: http://www.stltoday.com/sports/baseball/professional/branded-for-life-with-the-mendoza-line/article_cff05af5-032e-5a29-b5a8-ecc9216b0c02.html  Table of contents:1.  Set up2.  Data      2.1  Sources      2.2  Data wrangling and initial observations      2.3  Data quality check3.  Exploration and analysis      3.1  How bad was this average in the years leading up to 1979?      3.2  What percent of batters are below the Mendoza Line over time?4.  Conclusions      4.1  Limitations and areas for further investigation  1. Set up-----Load the required libraries:<jupyter_code>import numpy as np
import pandas as pd
import platform
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import norm, percentileofscore
%matplotlib inline<jupyter_output><empty_output><jupyter_text>For readers and reviewers, the versions of the major software components are:<jupyter_code>print('python version:', platform.python_version())
print(pd.__name__, 'version', pd.__version__)
print(np.__name__, 'version', np.__version__)<jupyter_output>python version: 3.6.1
pandas version 0.20.1
numpy version 1.12.1<jupyter_text>2. Data----- 2.1 SourcesAs noted earlier, the data used comes from SeanLahman.com's baseball database. Specifically, I used this [dataset](http://seanlahman.com/files/database/baseballdatabank-2017.1.zip) which was updated February 26, 2017 with statistics through the 2016 season. While the zip repository contains 27 different .csv files covering various statistics, we're only going to use a subset:1. Master.csv --> player names and biographical data2. Batting.csv --> batting stastics3. Appearances.csv --> positional info  2.2 Data wrangling and initial observationsImport each of the .csv files into a pandas DataFrame object:<jupyter_code>directory = 'core/'
master_df = pd.read_csv(directory + 'Master.csv')
batting_df = pd.read_csv(directory + 'Batting.csv')
appearances_df = pd.read_csv(directory + 'Appearances.csv')<jupyter_output><empty_output><jupyter_text>Look at the master table to make sure it loaded correctly:<jupyter_code>master_df.head()<jupyter_output><empty_output><jupyter_text>First, let's see if we can find <NAME> in our database...<jupyter_code>mendozas = master_df.loc[master_df['nameLast'] == 'Mendoza']
mendozas<jupyter_output><empty_output><jupyter_text>Judging by the first names and the dates played compared to the biographical info in the background reading, it's pretty easy to find our man in the third row, born in Chihuahua, Mexico in 1950. Let's save his player ID in a variable **mendoza_ID** so we can look up his stats.<jupyter_code>mendoza_ID = mendozas[mendozas['nameFirst'] == 'Mario']['playerID'].values[0]
mendoza_ID<jupyter_output><empty_output><jupyter_text>Now, let's look up Mario Mendoza's batting statistics.  First, let's look at the batting dataframe:<jupyter_code>batting_df.head()<jupyter_output><empty_output><jupyter_text>The columns in the batting_df dataframe have the following labels:<jupyter_code>#playerID       Player ID code
#yearID         Year
#stint          player's stint (order of appearances within a season)
#teamID         Team
#lgID           League
#G              Games
#AB             At Bats
#R              Runs
#H              Hits
#2B             Doubles
#3B             Triples
#HR             Homeruns
#RBI            Runs Batted In
#SB             Stolen Bases
#CS             Caught Stealing
#BB             Base on Balls
#SO             Strikeouts
#IBB            Intentional walks
#HBP            Hit by pitch
#SH             Sacrifice hits
#SF             Sacrifice flies
#GIDP           Grounded into double plays<jupyter_output><empty_output><jupyter_text>Let's examine Mendoza's numbers:<jupyter_code>mendoza_batting_df = batting_df[batting_df['playerID'] == mendoza_ID]
mendoza_batting_df<jupyter_output><empty_output><jupyter_text>Create a quick summary of Mendoza's hits and at bats per year, and calculate his batting average **('BA')** - note the convention is to round this to three decimals places:<jupyter_code>def calculate_BA(batting_df):
    return (batting_df['H']/batting_df['AB']).round(3)
mendoza_data = pd.DataFrame.from_items([('BA', calculate_BA(mendoza_batting_df)),
                             ('H', mendoza_batting_df['H']), 
                             ('AB', mendoza_batting_df['AB'])])
mendoza_data.index = mendoza_batting_df['yearID']
mendoza_data<jupyter_output><empty_output><jupyter_text>Let's look at his typical batting average in the years up through (and including) 1979 when the phrase was coined:<jupyter_code>end_year = 1979
start_year = mendoza_data.index.values.min()
print('Average {} - {} batting average: {:4.3f}'.format(start_year, end_year, 
      mendoza_data[(mendoza_data.index) <= end_year]['BA'].mean()))<jupyter_output>Average 1974 - 1979 batting average: 0.200<jupyter_text>The Mendoza Line quantified and verified: he was a .200 hitter Now, this "average of averages" would give equal weighting to his batting averages from each year regardless of the number of at bats.  Let's redo the previous calculation using the actual hits and at bats from each season:<jupyter_code>print('Cumulative {} - {} batting average: {:4.3f}'.format(start_year, end_year,
    float(mendoza_data[(mendoza_data.index) <= end_year]['H'].sum()/mendoza_data[(mendoza_data.index) <= end_year]['AB'].sum())))<jupyter_output>Cumulative 1974 - 1979 batting average: 0.201<jupyter_text>Looks like the cumulative batting average over that period was almost consistent with the average of his batting averages, so the initial figure wasn't skewed by any outlier years.  How did he fare from 1979 through the end of his career in 1982?<jupyter_code>final_career_year = mendoza_data.index.values.max()
print('{} - {} batting average: {:4.3f}'.format(end_year+1, final_career_year, 
      float(mendoza_data[(mendoza_data.index) > end_year]['H'].sum()/mendoza_data[(mendoza_data.index) > end_year]['AB'].sum())))<jupyter_output>1980 - 1982 batting average: 0.235<jupyter_text>He was a little better those last few years, but unfortunately the saying had already become a cultural idoim and the "Mendoza Line" was memorialized as a batting average of **0.200**.<jupyter_code>MENDOZA_LINE = 0.200<jupyter_output><empty_output><jupyter_text>2.3 Data quality checkWe've imported the csv files into three dataframes for our analysis1. master_df --> player names and biographical data2. batting_df --> batting stastics3. appearances_df --> positional infoThe master_df was only needed to find our info for <NAME> as we aren't using biographical data elsewhere in our analysis, so we don't need to scrub this dataset as it has already served its limited purpose. However, we should investigate the batting and appearances datasets to check for data issues.<jupyter_code>batting_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 102816 entries, 0 to 102815
Data columns (total 22 columns):
playerID    102816 non-null object
yearID      102816 non-null int64
stint       102816 non-null int64
teamID      102816 non-null object
lgID        102079 non-null object
G           102816 non-null int64
AB          102816 non-null int64
R           102816 non-null int64
H           102816 non-null int64
2B          102816 non-null int64
3B          102816 non-null int64
HR          102816 non-null int64
RBI         102392 non-null float64
SB          101516 non-null float64
CS          79360 non-null float64
BB          102816 non-null int64
SO          94978 non-null float64
IBB         66251 non-null float64
HBP         100006 non-null float64
SH          96478 non-null float64
SF          66782 non-null float64
GIDP        76706 non-null float64
dtypes: float64(9), int64(10), object(3)
memory usage: 17.3+ MB<jupyter_text>We can see in the information above that it looks like there are a good number of missing data points from the batting records.  This data set goes back to 1871 and it's not surprising that some data may not have been tracked in the same way historically.  However, our analysis will only be covering from 1975 onward, a relatively modern period.  We can check that subset of the data:<jupyter_code>batting_df[batting_df['yearID'] >= 1975].info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 50151 entries, 52665 to 102815
Data columns (total 22 columns):
playerID    50151 non-null object
yearID      50151 non-null int64
stint       50151 non-null int64
teamID      50151 non-null object
lgID        50151 non-null object
G           50151 non-null int64
AB          50151 non-null int64
R           50151 non-null int64
H           50151 non-null int64
2B          50151 non-null int64
3B          50151 non-null int64
HR          50151 non-null int64
RBI         50151 non-null float64
SB          50151 non-null float64
CS          50151 non-null float64
BB          50151 non-null int64
SO          50151 non-null float64
IBB         50151 non-null float64
HBP         50151 non-null float64
SH          50151 non-null float64
SF          50151 non-null float64
GIDP        50151 non-null float64
dtypes: float64(9), int64(10), object(3)
memory usage: 8.8+ MB<jupyter_text>Great - it looks like there is no missing batting data in this period. Now, let's verify the same on the appearances data:<jupyter_code>appearances_df[appearances_df['yearID'] >= 1975].info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 50131 entries, 52630 to 102760
Data columns (total 21 columns):
yearID       50131 non-null int64
teamID       50131 non-null object
lgID         50131 non-null object
playerID     50131 non-null object
G_all        50131 non-null int64
GS           50131 non-null float64
G_batting    50131 non-null int64
G_defense    50131 non-null int64
G_p          50131 non-null int64
G_c          50131 non-null int64
G_1b         50131 non-null int64
G_2b         50131 non-null int64
G_3b         50131 non-null int64
G_ss         50131 non-null int64
G_lf         50131 non-null int64
G_cf         50131 non-null int64
G_rf         50131 non-null int64
G_of         50131 non-null int64
G_dh         50131 non-null float64
G_ph         50131 non-null float64
G_pr         50131 non-null float64
dtypes: float64(4), int64(14), object(3)
memory usage: 8.4+ MB<jupyter_text>Similarly, it looks like there are no missing data points in this subset of the data either. Again, it makes sense that the data sets from 1975 forward would be clean as baseball was very popular during this entire period and keeping detailed statistics had long been part of baseball, even pre-dating the period in question.  3. Exploration and analysis----- 3.1 How bad was this average in the years leading up to 1979?In order to quantify how mediocre a performance batting .200 was in 1979 when the phrase was coined, I want to look at typical batting averages in this time period. To do this, I need to adjust the batting_df dataset in a few different ways:* Look only at data in the 5 year window from 1975 - 1979* Remove pitchers* Remove players without at least 50 at bats in a season *(which could be stints with multiple teams in the same season)* First, create a new dataframe with just the batting data from 1975 to 1979 (inclusive)<jupyter_code>def stat_window(df, start_year, end_year):
    search = str(start_year) + ' <= yearID <= ' + str(end_year)
    return df.query(search)

start_year = 1975
end_year = 1979
batting_window = stat_window(batting_df, start_year, end_year)
print(len(batting_window), "batting data records from {} - {}".format(start_year,
                                                                            end_year))
batting_window.head()
batting_window.info()
players_set = set(batting_window['playerID'])
print(len(players_set), "unique players with batting records during this period")<jupyter_output>1436 unique players with batting records during this period<jupyter_text>Next, remove pitchers from the dataset.  Pitchers are defined as players with more than one appearance as pitcher during a season. One appeance is used as the threshold to allow for fielders who might pitch rarely during an extra innings situation. This could lead to slight errors on edge cases of fielders who routinely pitched or players who switched positions during their career, but this would be very rare case during the time period being analyzed.<jupyter_code># Create a set of all players with more than one game pitched in a stint or season
min_G_p = 1
all_pitchers = set(appearances_df[appearances_df['G_p'] > min_G_p]['playerID'])

# remove these players from the batting dataframe
batters_set = set(x for x in batting_window['playerID'] if x not in all_pitchers)
print(len(batters_set), "unique non-pitchers in {} - {}".format(start_year, end_year))
def remove_position(df, position):
    non_position = [x not in position for x in df['playerID']]
    return df[non_position]

batting_window = remove_position(batting_window, all_pitchers)
print(len(batting_window), 'batting data records with pitchers removed')
print(len(set(batting_window['playerID'])), 
      "unique players, should match unique non-pitchers in cell above")
batting_window.head()<jupyter_output>2729 batting data records with pitchers removed
815 unique players, should match unique non-pitchers in cell above<jupyter_text>Next, remove players without at least 50 at bats in that year. The intent here is to try to remove "noisy" data points from players who didn't have at least 50 at bats in a season, which might included short-term call-ups from the minor leagues, injured players, etc.  However, we must allow for players to achieve this minimum in a combination of 'stints' across different teams in the same season.***To do this, we create a multi-index*** to sum the games played ('G') data by playerID and yearID (to aggregate seasons with multiple stints), so that we can look up our data by player, by year:<jupyter_code>def get_player_year_sum(df, field):
    
    grouped = df.groupby(['playerID', 'yearID'], as_index=False).sum()
    
    index_arrays = [grouped['playerID'], grouped['yearID']]
    multi_index = pd.MultiIndex.from_arrays(index_arrays, names = ['playerID', 'yearID'])
    return pd.Series(grouped[field].values, index=multi_index)

stat = 'AB'
player_year_stats = get_player_year_sum(batting_window, stat)
player_year_stats.head(10)<jupyter_output><empty_output><jupyter_text>Create a boolean array to check for minimum criteria (at bats) in the season:<jupyter_code>min_stat = 50
required_min = []
for x in batting_window.iterrows():
    if player_year_stats[x[1][0], x[1][1]] >= min_stat:
        required_min.append(True)
    else:
        required_min.append(False)

batting_window = batting_window[required_min]
print(len(batting_window), 'batting data records with minimum of {} {}'.format(min_stat, stat))<jupyter_output>2163 batting data records with minimum of 50 AB<jupyter_text>Now that we've cleaned up this data, we can analyze the distribution of batting averages.<jupyter_code>BAs_window = batting_window['H']/batting_window['AB']
BAs_window.describe()<jupyter_output><empty_output><jupyter_text>From the describe() statement above, the mean of the batting averages was .251, with a standard deviation of 0.47 **- so the Mendoza Line of .200 was about one standard deviation below the mean.**  We can also graph the distribution of batting averages to get a visual feeling for the distribution.<jupyter_code>BA_bins = [x/1000 for x in range(100,410,10)]
plt.rcParams['figure.figsize'] = 8, 5
BAs_window.hist(bins=BA_bins, normed = True, edgecolor='black')
plt.title('MLB batting averages: 1975 - 1979')
plt.axvline(x=0.200, color='black')
plt.text(.190, 7 , "Mendoza Line", rotation=90)
plt.xlabel('Batting average')
plt.ylabel('Frequency (percentage)')
plt.show()<jupyter_output><empty_output><jupyter_text>Calculating some statistics based on a normal distribution...<jupyter_code>mendoza_Z = (MENDOZA_LINE - BAs_window.mean())/BAs_window.std(ddof=0)
print("The Z score of a .200 batting average is {:4.2f}".format(mendoza_Z))
print("Assuming a normal distribution of batting averages, this would place .200 above",
      "only {:3.1f}% of batters".format(100*norm.cdf(mendoza_Z)))<jupyter_output>The Z score of a .200 batting average is -1.10
Assuming a normal distribution of batting averages, this would place .200 above only 13.6% of batters<jupyter_text>However, the normal distribution is only and approximation of the data.  We can look at the actual percentile rankings of the batting averages to calculate precisely what percentage of batters would fall below the Mendoza Line:<jupyter_code>BAs_window.quantile([0,.1,.2,.3,.4,.5,.6,.7,.8,.9,1])<jupyter_output><empty_output><jupyter_text>Eyeballing the deciles above would imply that and average of .200 would fall just north of the tenth precentile (where only 10% of observations would be below this point).  This is even worse than what the normal distribution would imply. We can use **percentileofscore** from the scipy.stats module to figure out precisely what percentage of scores were below .200:<jupyter_code>def mendoza_percentile(series):
    return percentileofscore(series, MENDOZA_LINE, kind="strict")

print("Given the actual distribution, a .200 batting average was above", 
      "only {:3.1f}% of batters".format(mendoza_percentile(BAs_window)))<jupyter_output>Given the actual distribution, a .200 batting average was above only 10.3% of batters<jupyter_text>Conculsion: 1. The term "Mendoza Line" refers to a performance of batting average of approximately .200, as verified by <NAME>'s actual batting average in the years before the term was coined.  2. This level of performance in the 1975-1979 time frame would have placed a batter in only the 10th percentile. Said another way, almost *90% of batters had a higher average* when we removed pitchers and players without a minimum number of at bats.   3.2 What percent of batters are below the "Mendoza Line" over time?In the 1975-1979 time frame, when the term Mendoza Line was coined, batting .200 put a player in roughly the 10th percentile of eligible batters (those with at least 50 ABs, excluding pitchers). I'd like to know how this level varied over time thereafter (from 1980 onward).   **Specifically, what percent of batters are below .200 each year?**  First, create a dataset with just the figures from 1980 forward<jupyter_code>start_year = 1980
end_year = batting_df['yearID'].max()

batting_window = stat_window(batting_df, start_year, end_year)
print(len(batting_window), "batting data records from {} - {}".format(
                            start_year, end_year))
batting_window.head()<jupyter_output>45453 batting data records from 1980 - 2016<jupyter_text>Again, remove the pitchers<jupyter_code>batting_window = remove_position(batting_window, all_pitchers)
print(len(batting_window), 'batting data records with pitchers removed')<jupyter_output>23510 batting data records with pitchers removed<jupyter_text>Next, remove players without at least 50 at bats in that year. Similar to the process above, we need to create a multiindex to allow for players to have different 'stints' across different teams in the same season.  Note that to qualify for awards like the batting title, the minimum level of appearances is much higher.  Create the at bats multiindex for the 1980 onward batting data:<jupyter_code>stat = 'AB'
player_year_stats = get_player_year_sum(batting_window, stat)
player_year_stats.head(10)<jupyter_output><empty_output><jupyter_text>And remove the players without less than 50 ABs in a year from our post-1980 batting dataframe<jupyter_code>min_stat = 50
required_min = []
for x in batting_window.iterrows():
    if player_year_stats[x[1][0], x[1][1]] >= min_stat:
        required_min.append(True)
    else:
        required_min.append(False)

batting_window = batting_window[required_min]
print(len(batting_window), 'batting data records with minimum of {} {}'.format(min_stat, stat))
batting_window.head(10)
#  helper function to return an array with the qualifying batting averages for any given year
def get_annual_BA(year):
    annual_data = batting_window[batting_window['yearID'] == year]
    return (annual_data['H']/annual_data['AB']).values

# create a dataframe with a column containing the qualifying batting averages for each year
# note that the columns will be of varying lengths, but pandas will pad the missing values with NaN

BA_dict = {x: get_annual_BA(x) for x in range(start_year, end_year+1)}
annual_BA_df = pd.DataFrame.from_dict(BA_dict, orient='index')
annual_BA_df = annual_BA_df.transpose()
annual_BA_df.head()<jupyter_output><empty_output><jupyter_text>Quick detour: Let's take a look at the 1980's to get a feel for how batting averages are distributed by year<jupyter_code>eighties = annual_BA_df.loc[:,:1989]
eighties.head()
eighties.plot.box()
plt.title('1980s MLB batting averages (min 50 ABs)')
plt.ylim(.1, .4)
plt.axhline(y=0.200, color='black')
plt.text(1.5, .21 , "<NAME>")
plt.xlabel('Year')
plt.ylabel('Batting average')
plt.show()<jupyter_output><empty_output><jupyter_text>As shown in the box plot above, the middle 50% of the observed batting averages fall in a band roughly .050 wide centered in the .250 - .260 area.  The maximum is generally in the .340-.360 area and the min is in the .150 - .170 area.  It looks like the .200 Mendoza Line remains a very low average.     But how low is low? Let's get the percetile represented by .200 in each year.<jupyter_code># helper function to eliminate NaNs from series before using to calculate percentiles
def no_NaNs(series):
    return [x for x in series if x > 0] 

min_year = batting_window['yearID'].min()
max_year = batting_window['yearID'].max()
years = [x for x in range(min_year, max_year+1)]
mendoza_pct_by_year = pd.Series([mendoza_percentile(no_NaNs(annual_BA_df[year])) for year in years], index=years)
mendoza_pct_by_year.plot()
plt.title('MLB batting averages below Mendoza Line annually, since 1980')
ave = mendoza_pct_by_year.mean()
plt.axhline(y=ave, color='black')
plt.xlabel('Year')
plt.ylabel('Percentage of batting averages below Mendoza Line')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08279843191315212, 'fraction_numerical': 0.045017877913238274, 'mean_word_length': 4.3773453787352326, 'pattern_counts': {'":': 0, '<': 128, '<?xml version=': 0, '>': 137, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '63210', 'license': 'MIT', 'n_tokens_mistral': 7849, 'n_tokens_neox': 6935, 'n_words': 2595, 'path': 'baseball-final.ipynb', 'repo_name': 'mkelty99/mendoza_line', 'size': '23247'}