<jupyter_start><jupyter_text>Data Quality CheckThis notebook will investigate the quality of the statcast data, such as missing values<jupyter_code>import pandas as pd
import numpy as np
import pathlib
import matplotlib.pyplot as plt
from IPython.display import Video


RAW_DATA = pathlib.Path.cwd().parent / 'Data' / 'raw' / 'Statcast_data.csv'
print(pathlib.Path.is_file(RAW_DATA))

#read in the data as a dataframe
baseball = pd.read_csv(RAW_DATA, index_col = 0)<jupyter_output><empty_output><jupyter_text>Look at data in general<jupyter_code>baseball.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 127400 entries, 0 to 127399
Data columns (total 20 columns):
 #   Column             Non-Null Count   Dtype  
---  ------             --------------   -----  
 0   player_name        127400 non-null  object 
 1   p_throws           127400 non-null  object 
 2   pitch_name         126918 non-null  object 
 3   release_speed      126922 non-null  float64
 4   release_spin_rate  125123 non-null  float64
 5   release_pos_x      126899 non-null  float64
 6   release_pos_y      126899 non-null  float64
 7   release_pos_z      126899 non-null  float64
 8   pfx_x              126899 non-null  float64
 9   pfx_z              126899 non-null  float64
 10  vx0                126899 non-null  float64
 11  vy0                126899 non-null  float64
 12  vz0                126899 non-null  float64
 13  ax                 126899 non-null  float64
 14  ay                 126899 non-null  float64
 15  az                 126899 non-null  float64
 16  sz[...]<jupyter_text>Look at cateorical (object) data types<jupyter_code>cat_dtypes = [col for col in baseball.select_dtypes('object').columns]
cat_dtypes
# p_throws
baseball['p_throws'].value_counts().sort_values(ascending=True).plot(kind='barh')
# check out the pitch_name categories
baseball['pitch_name'].value_counts()
#target feature
baseball['description'].value_counts().sort_values(ascending=True).plot(kind='barh')<jupyter_output><empty_output><jupyter_text>Takeaways: 1. Eephus is not a common pitch; it is extremely rare, and virtually no pitcher intentionlly has an Eephus pitch that would intend to use in a game.2. Knuckle Curve is considered to be essentially the same type of pitch as Curveball; they move very similarly to be considered the same pitch.3. The 'description' column has ball and blocked_ball categories, which are the same result: ball. Those features can be combined. Also note the imbalance between the categories.    Numeric Data1. Check extreme/impossible values (ex: negative velocity)2. Check missing values3.<jupyter_code>baseball.describe().T<jupyter_output><empty_output><jupyter_text>Some notes:- release_speed: no improper values. Some pitches are slow (59mph), and some pitches are very fast (104mph); both values not unheard of.- release_spin_rate: minimum and max values are interesting; may be an eephus pitch / extreme fastball, respectively. Will explore below.- release_pos_x: Range is supposed to be (-5, 5), so while the min value is less than -5, it is not too troubling.- release_pos_y: no improper values. Interesting range of values. Curious to explore further.- release_pos_z: no improper values.- pfx_x: no improper values- pfx_z: no improper values- vx0: no improper values- vy0: no improper values- vz0: no improper values- ax: no improper values- ay: no improper values- az: no improper values- sz_top: no improper values- sz_bot: no improper values- release_extension: no improper values<jupyter_code># check out the release_spin_rate extreme values
# what pitch type was the extreme slow spin rate pitch and who threw it?

slow_spin = baseball.loc[baseball['release_spin_rate'] < 500]
slow_spin<jupyter_output><empty_output><jupyter_text>Here is the specific pitch: Link: https://sporty-clips.mlb.com/1e34e71d-7067-4ca7-b364-7b59be93d955.mp4   Some spin rates seem off a bit. Explore further below<jupyter_code>player_pitch_spins = baseball[['player_name', 'pitch_name', 'release_spin_rate']].groupby(['player_name', 'pitch_name']).describe().sort_values(by=('release_spin_rate', 'mean'))
player_pitch_spins.head(20)
player_pitch_spins.tail(20)<jupyter_output><empty_output><jupyter_text>Release spin rate does have a lot of variance, but the values are valid... Analysis of Missing Values<jupyter_code>#look at the missing instances
for col in baseball.columns:
    missing = pd.isna(baseball[col]).sum()
    print(f'{col} has: {missing} missing values \n')
#grab a dataframe of every row with at least one missing value

baseball_missing = baseball[baseball.isnull().values.any(axis = 1)]

baseball_missing
#grab which player has greatest combination of pitches with a missing value and total missing values in all attributes. 

missing_groupby = baseball_missing.set_index('player_name').isnull().sum(axis=1).groupby(level=0).agg(['count','sum']).sort_values('count', ascending=False)
missing_groupby.columns = ['Instances', 'Total_missing']
missing_groupby.head(20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08933889219773675, 'fraction_numerical': 0.043478260869565216, 'mean_word_length': 4.204545454545454, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '528668', 'license': 'MIT', 'n_tokens_mistral': 1717, 'n_tokens_neox': 1543, 'n_words': 507, 'path': 'notebooks/data_cleaning.ipynb', 'repo_name': 'khickey25/baseball_pitch_prediction', 'size': '5037'}