<jupyter_start><jupyter_text>Data Exploration and Introduction to Hypothesis TestingNow, we'll go through some of the basic statistical concepts and start talking about hypothesis testing and continue next week.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
from datetime import datetime
import seaborn as sns  # a visualization library based on matplotlib
from datetime import datetime

import warnings
warnings.filterwarnings('ignore')

sns.set(style="darkgrid")

%matplotlib inline
from os.path import join
d_path = "./data"
filename = "nba_players_modified.csv"

df = pd.read_csv(join(d_path, filename))
df.head()
df["Year"] = pd.to_datetime(df["Year"])  # convert back to datetime<jupyter_output><empty_output><jupyter_text>Descriptive Statistics**Measures of central tendency**___Mean:___ The mean is a descriptive statistic that looks at the average value of a data set. In the case of the mean, the "middle" of the data set refers to this typical value.$\mu = \frac{\sum_i^N X_i}{N}$___Median:___ The median attempts to define a typical value in the data set as well. It is the value in the middle when the data set is sorted.___Mode:___ Most frequent value in the data set.**Measures of spread**___Variance:___ Measures how far each number in the set is from the mean.$\sigma^2 = \frac{\sum_{i=1}^N (x_i - \mu)^2 }{N}$___Standard Deviation:___ The square root of variance. The main difference from variance is that standard deviation returns the values in the originl units.$\sigma = \sqrt{\frac{\sum_{i=1}^N (x_i - \mu)^2 }{N}}$<jupyter_code>df.describe()  # generates descriptive statistics that summarize 
               # the central tendency, dispersion and shape of a dataset’s distribution
               # excluding NaN values<jupyter_output><empty_output><jupyter_text>Now, let's observe the distributions of some columns.<jupyter_code>sns.distplot(df["G"].values, norm_hist=True)  # games played
plt.show()
sns.distplot(df["PTS"], norm_hist=True)  # points<jupyter_output><empty_output><jupyter_text>[Boxplots](https://www.wellbeingatschool.org.nz/information-sheet/understanding-and-interpreting-box-plots)**Median**The median (middle quartile) marks the mid-point of the data and is shown by the line that divides the box into two parts. Half the scores are greater than or equal to this value and half are less.**Inter-quartile range**The middle “box” represents the middle 50% of scores for the group. The range of scores from lower to upper quartile is referred to as the inter-quartile range. The middle 50% of scores fall within the inter-quartile range.**Upper quartile**Seventy-five percent of the scores fall below the upper quartile.**Lower quartile**Twenty-five percent of scores fall below the lower quartile.**Whiskers**The upper and lower whiskers represent scores outside the middle 50%. Whiskers often (but not always) stretch over a wider range of scores than the middle quartile groups.![](https://mathpullzone-8231.kxcdn.com/wp-content/uploads/boxplot-with-outliers.jpg)<jupyter_code>plt.subplot(3,1,1)
sns.boxplot(data=df[["G"]], orient="h")

plt.subplot(3,1,2)
sns.boxplot(data=df[["PTS"]], orient="h")

plt.subplot(3,1,3)
sns.boxplot(data=df[["3PA"]], orient="h")

plt.tight_layout()  # auto. positioning of subplots
plt.show()<jupyter_output><empty_output><jupyter_text>We can also analyse how attributes relate to each other.<jupyter_code>cols2plot = ["PTS", "MP"]

sns.pairplot(data=df, vars=cols2plot)
plt.plot()
cols2plot = ["MP", "PTS", "PF"]

# applying a filter on year
sns.pairplot(data=df[df["Year"] > datetime(2012,1,1)], vars=cols2plot, hue="Pos")
plt.plot()<jupyter_output><empty_output><jupyter_text>Extracting Some Insights<jupyter_code>top5scorers = df.groupby(by="Player").sum()["PTS"].sort_values(ascending=False)[:5]
print(top5scorers)
top5scorers.plot(kind="barh")  # does not match with the official list
                               # since we removed some of the veterans<jupyter_output><empty_output><jupyter_text>How does age affect points made?<jupyter_code>min_age = df["Age"].min()
max_age = df["Age"].max()

age_group = df.groupby(by="Age").sum()  # group by age and take sum of each column
pts_game = age_group["PTS"] / age_group["G"]  # find pts avg. at each age

ax = pts_game.plot(kind="bar",
                   color="steelblue",
                   rot=70)

for tick in ax.xaxis.get_major_ticks()[::2]:  # just to make it more readable
    tick.label1.set_visible(False)            # hiding some xticks

plt.ylabel("Avg. Points")
plt.title("Age vs Points")
plt.show()<jupyter_output><empty_output><jupyter_text><NAME> Shot Selection in Each Season<jupyter_code>df["2PAr"] = 1 - df["3PAr"]  # adding the two point shot rate column

player = "<NAME>"
player_df = df[df["Player"] == player]

player_df.set_index(player_df["Year"].dt.strftime("%Y"), inplace=True)  # change index to display
                                                                        # years as xticks

player_df[["2PAr", "3PAr"]].plot.bar(rot=60)
plt.title("Shot Selection in Each Season by {}".format(player))
plt.show()<jupyter_output><empty_output><jupyter_text>How did Golden State Warriors affect the league?There is an ongoing debate regarding the shot selections in NBA. Teams strated to focus more on three points shots and changed their game plans accordingly. Some of the experts credit Golden State Warriors as the initiater team, beacuse of players like Curry and Thompson. You can learn more about the discussion from this [link.](https://www.pinnacle.com/en/betting-articles/Basketball/three-point-shots-nba/VPN2MLSADYW5FPZL)Let's check if we can see such a phenomena in our dataset.<jupyter_code>base_year = 1990
gw_year = 2012

# checking 3-point shot attemps 
past_df = df[(df["Year"] < datetime(2012, 1, 1)) & (df["Year"] >= datetime(2007, 1, 1))]
gs_df = df[df["Year"] > datetime(2012, 1, 1)]

ax = sns.kdeplot(past_df["3PAr"].rename("2007-2012"), shade=True)
sns.kdeplot(gs_df["3PAr"].rename("2012-2017"), ax=ax, shade=True)

plt.show()<jupyter_output><empty_output><jupyter_text>The difference between two terms is visible. Now, let's apply a significance test to statistically approve this difference.<jupyter_code># extracting values
past_values = past_df["3PAr"].values
gs_era = gs_df["3PAr"].values
# two-sided test for the null hypothesis that 2 independent samples 
# have identical average (expected) values
_, p_value = stats.ttest_ind(a=past_values, b=gs_era, equal_var=False)
p_value<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11646341463414635, 'fraction_numerical': 0.01524390243902439, 'mean_word_length': 5.091922005571031, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 46, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '180053', 'license': 'MIT', 'n_tokens_mistral': 2188, 'n_tokens_neox': 2049, 'n_words': 647, 'path': 'recitation/rec3-data-exploration.ipynb', 'repo_name': 'berkayersever/data-science', 'size': '6574'}