<jupyter_start><jupyter_text>2. Setup and Joining We assume that all data was reported and recorded correctly, and that no players share the same name and pick.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
pd.set_option('display.max_columns', None)
player_avgs = pd.read_csv('player_avgs.csv')
nba_combine = pd.read_csv('nba_draft_combine_all_years.csv')
player_avgs.head()
nba_combine.head()<jupyter_output><empty_output><jupyter_text>We'll rename the `Draft pick` column in `nba_combine` to `Pk`, in order to facilitate joining the two datasets later:<jupyter_code>nba_combine.rename(columns = {'Draft pick': 'Pk'}, inplace = True)<jupyter_output><empty_output><jupyter_text>Then, we need to clean the players' names in the `player_avgs` dataframe, so that the part after the slash is not included.<jupyter_code>player_avgs.Player = [name.split('\\')[0] for name in player_avgs.Player]
player_avgs.Player[0:5]<jupyter_output><empty_output><jupyter_text>Now, we can merge the two datasets using `pd.merge`. We'll join them based on both player name and which pick they were, since we have a case where two players had the exact same name (<NAME>) but were drafted in different positions. There are no other duplicated names.<jupyter_code>combine_careers_raw = pd.merge(nba_combine, player_avgs, on = ['Player', 'Pk'], how = 'inner')
# combine_careers_raw.to_csv('combine_careers.csv')
combine_careers_raw.head()
# the duplicated player name is <NAME>
combine_careers_raw.Player.value_counts().head()
# they were drafted at different picks
# joining on both player name and pick position allows us to avoid cross-product joins
combine_careers_raw[combine_careers_raw.Player == '<NAME>'][['Player', 'Pk']]<jupyter_output><empty_output><jupyter_text>3. Basic Cleaning Let's look at the structure of our data. We see below that we have 368 rows and 44 columns, but we have quite a few missing values in several columns. There's also seems to be several redundant columns that could be removed (e.g., `Year_x` and `Year_y`).<jupyter_code>combine_careers_raw.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 366 entries, 0 to 365
Data columns (total 43 columns):
Unnamed: 0                  366 non-null int64
Player                      366 non-null object
Year_x                      366 non-null int64
Pk                          366 non-null float64
Height (No Shoes)           366 non-null float64
Height (With Shoes)         366 non-null float64
Wingspan                    366 non-null float64
Standing reach              366 non-null float64
Vertical (Max)              317 non-null float64
Vertical (Max Reach)        317 non-null float64
Vertical (No Step)          317 non-null float64
Vertical (No Step Reach)    317 non-null float64
Weight                      365 non-null float64
Body Fat                    365 non-null float64
Hand (Length)               328 non-null float64
Hand (Width)                326 non-null float64
Bench                       209 non-null float64
Agility                     312 non-null float64
Sprint            [...]<jupyter_text>A. Removing Unnecessary Columns We immediately notice that the first column is unnecessary, so we can remove it.<jupyter_code>combine_careers = combine_careers_raw.copy()
combine_careers.drop('Unnamed: 0', 1, inplace = True)<jupyter_output><empty_output><jupyter_text>We also notice that we have two year columns. Are there any discrepancies between the two? The answer is yes:<jupyter_code>year_diff = combine_careers[['Year_x', 'Year_y']][combine_careers.Year_x != combine_careers.Year_y]
combine_careers.iloc[year_diff.index][['Player', 'Year_x', 'Year_y']]<jupyter_output><empty_output><jupyter_text>A quick look at these players' NBA combine results shows that `Year_y`  is the correct column (for both [2010](https://www.nbadraft.net/2010-nba-draft-combine-official-measurements/) and [2017](https://www.nbadraft.net/2017-nba-draft-combine-measurements/) draftees). A brief check against the [official NBA Combine data](https://stats.nba.com/draft/combine-anthro/?SeasonYear=2016-17) shows that player statistics are still accurate, with some minor rounding differences. We'll go ahead and delete the `Year_x` column, and rename `Year_y` to `Year`.<jupyter_code>combine_careers.drop('Year_x', 1, inplace = True)
combine_careers.rename(columns = {'Year_y': 'Year'}, inplace = True)<jupyter_output><empty_output><jupyter_text>Other columns we can delete: `Lg` and `Rd` (which stand for league and draft round, respectively). These are redundant because all players are in the NBA league, and because [draft round is redundant information if we know the draft pick](https://www.nba.com/nba-draft-lottery-explainer) (picks 1-30 are in round 1, and picks 31-60 are in round 2).<jupyter_code># all players are in the NBA
combine_careers.Lg.value_counts()
# all picks 1-30 are in Round 1, and all picks 31-60 are in Round 2
combine_careers.groupby(['Rd', 'Pk']).Player.count().unstack()
combine_careers.drop(['Lg', 'Rd'], 1, inplace = True)<jupyter_output><empty_output><jupyter_text>Let's take a final look at our (relatively clean) dataset. We'll deal with all the missing values next.<jupyter_code>combine_careers.shape
combine_careers.head()<jupyter_output><empty_output><jupyter_text>B. Removing Players Who Never Played We can easily see that there are a number of players who participated in the draft combine, were drafted, but never played a single game.<jupyter_code>combine_careers[combine_careers.G.isnull()].Player<jupyter_output><empty_output><jupyter_text>Let's go ahead and delete these rows, since they won't help us learn about how athleticism affects career.<jupyter_code>combine_careers = combine_careers[combine_careers.G.notnull()]
# no more players who didn't play any games
len(combine_careers[combine_careers.G.isnull()].Player)
combine_careers.shape<jupyter_output><empty_output><jupyter_text>C. Simplifying Positions  Players who have in-between positions may be denoted in two different ways (e.g., "G-F" and "F-G"). However, for our purposes, we don't need to worry about this distinction, so we'll combine "G-F" and "F-G", and then "F-C" and "C-F" (there's really no such thing as a "G-C" in the NBA, nor in our data).<jupyter_code>combine_careers.loc[combine_careers['Pos'] == 'F-C', 'Pos'] = 'C-F'
combine_careers.loc[combine_careers['Pos'] == 'F-G', 'Pos'] = 'G-F'
combine_careers.Pos.value_counts()<jupyter_output><empty_output><jupyter_text>Let's write this dataset to CSV.<jupyter_code>combine_careers.set_index('Player').to_csv('cc.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10136657155729088, 'fraction_numerical': 0.023126595584922662, 'mean_word_length': 4.751295336787565, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '512281', 'license': 'MIT', 'n_tokens_mistral': 2259, 'n_tokens_neox': 2106, 'n_words': 671, 'path': 'produce_base_dataset.ipynb', 'repo_name': 'jrtran/stats131group7', 'size': '6686'}