<jupyter_start><jupyter_text>Experimental Data Analysis of NCAA Basketball Statistics  Authored By: <NAME> and <NAME>  Equations Used:* The covariance can also be used to assess correlation: $cov=\frac{\sum_{i=1}^{n}(x_i -\bar{x})(y_i -\bar{y})}{n}$* The correlation coefficient $r$ helps checks how good the linear relationship is: $r=\frac{cov}{\sigma_x\sigma_y}$* Calculate the slope using the means (where n is the number of data points): $m =\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n}(x_i-\bar{x})^2}$* Calculate the y intercept as$b = \bar{y}-m\bar{x}$* $Scoring Margin = \frac{PPG}{Opponents PPG}$* $Effective Field Goal Percentage= \frac{3PM*1.5 + 2PM}{3PA + 2PA}$* $2 Pointers Made = FGM - 3FG$  Process of Coding* The required columns were loaded into NCAA_statistics_parsed.csv after being calculated with the neccessary functions.* In order to get the data I needed, I read in all of my files using mypytable csv reader. Then, I got all of the columns I needed using the get column function I have in mypytable.* In order to make the graphs I needed, I used reusable functions I made that use matplotlib in plot_utils.py. * In order to get values such as correlation, slope intercept, and covariance, I have reusable functions that are in plot_utils.py.<jupyter_code>import matplotlib.pyplot as plt
import importlib
import os
import numpy as np
# some useful mysklearn package import statements and reloads

import mysklearn.myutils
importlib.reload(mysklearn.myutils)
import mysklearn.myutils as myutils

# uncomment once you paste your mypytable.py into mysklearn package
import mysklearn.mypytable
importlib.reload(mysklearn.mypytable)
from mysklearn.mypytable import MyPyTable 

import mysklearn.plot_utils
importlib.reload(mysklearn.plot_utils)
import mysklearn.plot_utils as plot_utils

ncaa_facts = os.path.join("input_data","NCAA_Statistics_Parsed.csv")
ncaa_stats = MyPyTable().load_from_file(ncaa_facts)<jupyter_output><empty_output><jupyter_text>Histograms 1. Scoring Margin Histogram* Skew Left<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
scoring_margin = ncaa_stats.get_column("Scoring Margin")
plot_utils.create_histogram("Scoring Margin",scoring_margin)<jupyter_output><empty_output><jupyter_text>2. Effective Field Goal Percentage* Symmetric<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
efg = ncaa_stats.get_column("eFG%")
plot_utils.create_histogram("eFG%",efg)<jupyter_output><empty_output><jupyter_text>3. Steals Per Game + Blocks Per Game* Skew Right<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
spg_bpg = ncaa_stats.get_column("SPG+BPG")
plot_utils.create_histogram("SPG+BPG",spg_bpg)<jupyter_output><empty_output><jupyter_text>4. Rebound Margin* Skew Left<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
rebound_margin = ncaa_stats.get_column("Rebound Margin")
plot_utils.create_histogram("Rebound Margin",rebound_margin)<jupyter_output><empty_output><jupyter_text>5. Winning Percentage* Symmetric<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
win_percentage = ncaa_stats.get_column("Win Percentage")
plot_utils.create_histogram("Win Percentage",win_percentage)<jupyter_output><empty_output><jupyter_text>Analysis* From looking at the distributions, it looks like there tends to be more data towards middle values and taper off when either going to higher or lower values. There seems to be an outlier in Steals per game + blocks per game in that there are higher values for lower frequencies.  Scatter Plots 1. Scoring Margin V.S. Win Percentage<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
import scipy.stats as stats
import numpy as np
scoring_margin = ncaa_stats.get_column("Scoring Margin")
win_percentage = ncaa_stats.get_column("Win Percentage")
plot_utils.draw_scatter_plot(scoring_margin,win_percentage,"Scoring Margin", "Win Percentage")<jupyter_output><empty_output><jupyter_text>2. Effective Field Goal Percentage V.S. Win Percentage<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
import scipy.stats as stats
import numpy as np
efg = ncaa_stats.get_column("eFG%")
win_percentage = ncaa_stats.get_column("Win Percentage")
plot_utils.draw_scatter_plot(efg,win_percentage,"eFG%", "Win Percentage")<jupyter_output><empty_output><jupyter_text>3. Steals Per Game + Blocks Per Game V.S. Win Percentage<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
import scipy.stats as stats
import numpy as np
spg_bpg = ncaa_stats.get_column("SPG+BPG")
win_percentage = ncaa_stats.get_column("Win Percentage")
plot_utils.draw_scatter_plot(spg_bpg,win_percentage,"SPG+BPG", "Win Percentage")<jupyter_output><empty_output><jupyter_text>4. Rebound Margin V.S. Win Percentage<jupyter_code>importlib.reload(myutils)
importlib.reload(plot_utils)
import scipy.stats as stats
import numpy as np
rebound_margin = ncaa_stats.get_column("Rebound Margin")
win_percentage = ncaa_stats.get_column("Win Percentage")
plot_utils.draw_scatter_plot(rebound_margin,win_percentage,"Rebound Margin", "Win Percentage")<jupyter_output><empty_output><jupyter_text>Analysis* Highest to Lowest correlations:    1. Scoring Margin and Win Percentage    2. Rebound Margin and Win Percentage    3. Effective Field Goal Percentage and Win Percentage        4. Steals Per Game + Blocks Per Game and Win Percentage* Based off of this information, I would hypothesize that a mixture of Scoring Margin and Rebound margin would produce most accurate results in determining the winning percentage of a team due to the high correlation rate that they hold.* We ultimately ended up deciding to cutout the scoring margin attribute due to the likelihood that it would overshadow the rest of the attributes. This was trivial because teams that scored more than their opponents won their games.  Determining Discretezation Bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 3)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 4 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 4)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 5 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 5)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 6 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 6)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 7 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 7)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 8 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 8)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 9 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 9)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>* 10 bins<jupyter_code>importlib.reload(plot_utils)
percentages = ncaa_stats.get_column("Win Percentage")
cutoffs = plot_utils.compute_equal_width_cutoffs(percentages, 10)
freqs = plot_utils.get_cutoff_frequencies(percentages, cutoffs)
plt.bar(cutoffs[:-1], freqs, width=cutoffs[1] - cutoffs[0], align="edge")
plt.title("Win Percentage Frequency Diagram by Bins")
plt.xlabel("Cutoffs")
plt.ylabel("Frequency")
print()<jupyter_output><jupyter_text>Analysis:* We decided to go with the four bins option due to the fact that it was not too many or too few. Instead of doing equal width cutoffs, we modalarized it to make it more advanced by that by overriding the cutoff points at which we discretized the data.  Summary StatisticsRunning some basic code on each of the data here gives us some useful summary statistics:<jupyter_code>import mysklearn.myutils
importlib.reload(mysklearn.myutils)
import mysklearn.myutils as myutils

ncaa_facts = os.path.join("input_data","NCAA_Statistics_Parsed.csv")
ncaa_header, ncaa_data = myutils.load_from_file(ncaa_facts)
ncaa_data = myutils.convert_to_numeric(ncaa_data)
ncaa_header = ncaa_header[1:]
ncaa_data = myutils.drop_column(ncaa_data, ncaa_header, 0)

for att in ncaa_header:
    my_col = myutils.get_column(ncaa_data, ncaa_header, att)
    print("Median of %s:" % (att), myutils.get_median(my_col))
    print("Mean of %s:" % (att), myutils.get_mean(my_col))
    print("SD of %s:" % (att), myutils.get_standard_deviation(my_col))
    print()<jupyter_output>Median of Scoring Margin: 1.0170782550727964
Mean of Scoring Margin: 1.0162707010296386
SD of Scoring Margin: 0.09926059680557543

Median of eFG%: 0.5023921619007314
Mean of eFG%: 0.5031580512596515
SD of eFG%: 0.03096279413298644

Median of SPG+BPG: 9.594117647058823
Mean of SPG+BPG: 9.672694427382291
SD of SPG+BPG: 1.6354833087724625

Median of Rebound Margin: 1.008747777204153
Mean of Rebound Margin: 1.01716812779002
SD of Rebound Margin: 0.1179884096948017

Median of Win Percentage: 52.0
Mean of Win Percentage: 50.55472529117821
SD of Win Percentage: 18.29008236426564
{'fraction_non_alphanumeric': 0.10402956819615974, 'fraction_numerical': 0.02740466961146669, 'mean_word_length': 7.742316784869977, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '661660', 'license': 'MIT', 'n_tokens_mistral': 4171, 'n_tokens_neox': 3782, 'n_words': 915, 'path': 'EDA.ipynb', 'repo_name': 'NCAA-Basketball-Predictor/Predictor', 'size': '11103'}