<jupyter_start><jupyter_text>Smash BrosToday's extra nerdy dataset is stats from a tournament of the game *Super Smash Bros. Melee* for the Nintendo GameCube.  The stats were recorded during the tournament [Smash Summit 2017](https://liquipedia.net/smash/Smash_Summit/Spring_2017), these stats were collected by an older version the [Slippi](https://slippi.gg/faq) project.  The data was collected with this [R script](https://github.com/AdamSpannbauer/twitch_chat/blob/master/r_scripts/slippi_win.R) in 2017 (no promises the script still works).The data contains a player id column, `gamerTag`, and a binary variable indicating if the player won the set: `won`.  The rest of the variables are different statistics counted up throughout the set.<jupyter_code>python -m !pip install --upgrade pip
#!pip install --upgrade nb_black
#%reload_ext nb_black
import pandas as pd
import numpy as np

from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression

from scipy import stats

import seaborn as sns
import matplotlib.pyplot as plt

%matplotlib inline


data_url = "https://raw.githubusercontent.com/AdamSpannbauer/twitch_chat/master/data/slippi_data/ssbm_win.csv"
ssbm = pd.read_csv(data_url)
ssbm.head()<jupyter_output><empty_output><jupyter_text>Complete the below function to show the biggest offenders for NAs in a `pandas.DataFrame` (i.e. show the top n columns by number or percent of `np.nan` values in the column).* Give the `n` parameter a default value of `5`* Give the `rate` parameter a default boolean value* Fill in the blanks to match what the comments are saying<jupyter_code>def show_top_missing(ssbm, 5, rate=False):
    """Utility to print most 'sparse' columns by missing values"""

    # Create a boolean df indicating which values are np.nan
    is_na_df = ____

    # Aggregate the df:
    # * if rate is true, then calculate what percent of values are nan
    # * if rate is false, then calculate the count of nans
    if rate:
        missing_stats = ____
    else:
        missing_stats = ____

    top_missing = missing_stats.sort_values(ascending=False)

    # We might consider to return or print here
    # If we wanted to work with the output values, we'd want
    # to use return.  I just wanted to see a display of the
    # output so I chose print instead.
    print(top_missing.head(n))
# If you followed ALL the prompts for the function
# this code should run without error. If you get an 
# error about "missing required positional arguments"
# then you need to check that you set default values 
# for n and rate.

show_top_missing(ssbm)
ssbm.head()<jupyter_output><empty_output><jupyter_text>Create some visualization(s) in light of the goal (predicting win).  Based on the visualization, what do you think will be a good predictor?<jupyter_code>plot_cols = [
    "averageComboDamage",
    "numRecoveries",
    "numKillingPunishes",
    "averagePunishDamage",
    "neutralWinPercent",
    "openingsPerKill",
    "won",
]

sns.pairplot(ssbm[plot_cols], hue='won')
plt.show()

#categoric vs numeric
sns.boxplot(x='won', y='openingsPerKill', data=ssbm)
plt.show()
#suggests 6 kills predicts wins.
#categoric vs numeric
sns.swarmplot(x='won', y='openingsPerKill', data=ssbm)
plt.show()
#suggests 6 kills doesn't necessary predict. try to look at data unsummarized amap<jupyter_output><empty_output><jupyter_text>In most machine learning contexts, we will isolate our input features from our target variable.  Player ID won't be much use since we want our model to predict wins for every player, not just the 16 players in our data set.* So we'll drop `gamerTag` & `won` and assign it to `X`    * This `X` is a conventional way to specify that this variable is our input features* It's also conventional to put your target into a variable named `y`<jupyter_code># Store every column but 'gamerTag' and 'won' in X
X = ssbm.drop(columns=['gamerTag','won'])

# Store what you want to predict ('won') in y
y = ssbm['won']<jupyter_output><empty_output><jupyter_text>Let's go ahead and try to build our first model to predict this win.  Since the topic at hand is feature selection, we won't really discuss the model being used (yet); we'll just use it as a way to grade our features.  Note, this model building is also not following the best practice of doing a train/test split to evaluate accuracy.<jupyter_code>!pip install --upgrade --sklearn
# .fit is where sklearn learns
model = LogisticRegression()
model.fit(X, y)
# In practice score on a test set
model.score(X, y)
import sklearn
sklearn.__version__<jupyter_output><empty_output><jupyter_text>We see we ran into a `ConvergenceWarning`(note that it tells us some possible solutions to this issue).  Despite this issue we ended up predicting with ~95% accuracy.*Note, again, this is a **bad way to evaluate a model**, we should be testing our model on data the model has never seen before.* Feature SelectionWe might be able to avoid this warning by narrowing down our features.  Narrowing down features has a lot of benefits, including:* Aiding in computation issues (RE:above warning)* Aiding in accuracy    * Can be a direct result from helping first point* Aiding in interpretability    * A lot of times you're not only interested in predicting something accurately, but also explaining why you're good at making these predictions.  For example, in a business context you might want to accurately predict customer churn, but you're management would also really like to know what is indicating that a customer is going to churn.  With an easily interpretable model you can show the impact each input feature has on a customer's likelihood to churn.  With a lot of predictors this gets harder to parse out.    * There's also an issue known as [multicollinearity](https://en.wikipedia.org/wiki/Multicollinearity).  This intuitively means that some of your input features are encoding the same information (like having a feature for `temperature_celsius` & one for `temperature_farenheit`).  When your model has multiple variables representing the same thing it can confuse the model, the model might still be accurate, but it will have more trouble explaining to you why it's accurate.    In this case, a way we can smartly select a subset of predictors is with `SelectKBest` with the `f_classif` method.  If you remember, ANOVA outputs a value called $F$, this is what the `f` in `f_classif` represents.  All we're doing here is running an ANOVA for each of our variables and target.Our target variable is a category and all our predictors are numeric, so for each numeric feature we run an ANOVA using the target as the groups.  Below is an example using the feature `numKillingPunishes`.  To start we might visualize the differences between the 2 groups.  We see some difference, but we can quantify this difference using the `stats.f_oneway` function we've used before to perform an ANOVA.  We end up with an $F$ value of about `23.8`.Note, we didn't really check for normality here.  ANOVA is fairly robust against violations of the normality assumption. Also, in the case variable selection, we aren't trying to declare any significant difference, we're just looking for potentially useful differences. Perform an ANOVA for the `numKillingPunishes` feature grouped by `won`.<jupyter_code>sns.violinplot("won", "numKillingPunishes", data=ssbm)
plt.show()
won = ssbm[ssbm["won"] == 1]
lost = ssbm[ssbm["won"] == 0]

stats.f_oneway(won["numKillingPunishes"], lost["numKillingPunishes"])<jupyter_output><empty_output><jupyter_text>So we've done an ANOVA for one of our variables, we can use `SelectKBest` from `sklearn` to do the rest.  It will:* run the same ANOVA* store all the values of $F$* keep only the features with the top $k$ values of $F$We have to select the value of $k$, and there's not a general rule of thumb on what it should be. Below, I chose 5 cause I wanted to it to choose the 5 best because ¯\\_(ツ)_/¯.From the output, we can see the scores for each ANOVA that was run with `f_classif`, and we can confirm that we see the same value for `numKillingPunishes` as we did in our one variable example.<jupyter_code>selector = SelectKBest(f_classif, k=5)

# Use `.fit()` method so the selector can 'learn' from our data
selector.fit(X, y)

# Use `.transform()` method so the selector can apply
# what it learned in `.fit()`
k_best = selector.transform(X)<jupyter_output><empty_output><jupyter_text>* Use `SelectKBest` and `f_classif` to choose the best 5 predictors of `won`* Put these best predictors back into a dataframe with the original names<jupyter_code>selector = SelectKBest(f_classif, k=5)

# Use `.fit()` method so the selector can 'learn' from our data
selector.fit(X, y)

# Use `.transform()` method so the selector can apply
# what it learned in `.fit()`
k_best = selector.transform(X)

# We can see/rank which features were the best
score_df = pd.DataFrame({"feature": X.columns, "f_score": selector.scores_})
score_df = score_df.sort_values("f_score", ascending=False)
print(score_df.head())

# We can put back into a dataframe to see column names
best_df = pd.DataFrame(k_best, columns=ssbm.columns)
best_df.head()<jupyter_output><empty_output><jupyter_text>We could now move back to modeling this data using the same code as before, but now providing only the 5 'best' features.<jupyter_code>model = LogisticRegression()
model.fit(k_best, y)
# In practice score on a test set
model.score(k_best, y)<jupyter_output><empty_output><jupyter_text>With this restricted set, we didn't get a `ConvergenceWarning`. We do see a drop in accuracy, but this might actually be a sign that our first model was [*overfitting*](https://en.wikipedia.org/wiki/Overfitting).  Again, we can't take much from the score on our training data (see note below, and above, and everywhere).However, we didn't lose too much accuracy despite deleting 19 of our input columns.  This shows that we likely have a lot of 'dead weight' in our features, and we can likely predict the outcome without having all 24 features to wade through (which can be a burden for both our model and us during interpretation).*Note, again, this is a **bad way to evaluate a model**, we should be testing our model on data the model has never seen before.*  PCAIn PCA (principle components anaylsis) we capture the variance of our data into 'components'.  What does that mean? Well, we'll be predicting our outcome based on how the other variables' values change (or vary).  So our model is really wanting to pick up on how the input features vary in relation to the outcome we're predicting.  PCA captures all this 'varying' in our data by combining our features, and it can capture this variance with less features than we started with.Since we're really focused on how our features are varying, we `scale` them before doing PCA. Remember that `scale` will transform our inputs so that their means are 0 and their standard deviations are 1.  With all these standard deviations set to 1, none of the features will overpower the others in terms of their variance.  For example, the `totalComboDamage` has a variance of 188387.3 and `openingsPerKill` has a variance of 2.8, this is a big gap.  PCA is trying to maximize the amount of variation captured, and an easy way to shortcut this would be to focus in on features with larger variances; we want to put all our features on equal ground, and so we scale.Below we see the effects of scaling on our features. * Use the `StandardScaler()` to scale the data<jupyter_code>print("Feature variances before scaling:")
print(X.var().sort_values(ascending=False).head())

scaler = ____

# 'learn' from the data
scaler.____

# apply what the scaler learned
scaled_X = scaler.____<jupyter_output><empty_output><jupyter_text>Now we can move onto performing PCA.  To do this we can use `sklearn.decomposition.PCA`, and the main thing left for us to decide is how many components.  The reading focuses in on eigenvalues, here we'll use the variance explained and a scree plot.  I'd say the 'elbow' that we typically look for in a scree plot occurs after our 4th component.  So we might choose to set `n_components` to 4. * Create a `PCA` object and apply it to the scaled data* Plot the variance explained by each component* Create a correlation heatmap of the prinicple components<jupyter_code>pca = PCA()

# 'learn' from the data
pca.____

# apply what the pca object learned
pca_X = pca.____(scaled_X)

# What attribute holds the variance explained?
plt.plot(____, marker="o")
plt.title("Variance explained by each component")
plt.show()

sns.heatmap(____, vmin=-1, vmax=1)
plt.title("Correlation between components")
plt.show()<jupyter_output><empty_output><jupyter_text>* Use what we learned from the scree plot to choose an appropriate number of components to use for the model<jupyter_code>pca = PCA(____)

# 'learn' from the data
pca.____

# apply what the pca object learned
pca_X = pca.____(scaled_X)
model = LogisticRegression()
model.fit(pca_X, y)
# In practice score on a test set
model.score(pca_X, y)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0735071233707184, 'fraction_numerical': 0.003940588056986966, 'mean_word_length': 4.818783068783069, 'pattern_counts': {'":': 2, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '355150', 'license': 'MIT', 'n_tokens_mistral': 3906, 'n_tokens_neox': 3703, 'n_words': 1809, 'path': 'week5_eda/ssbm_features_blank.ipynb', 'repo_name': 'mgalaviz1/my_work', 'size': '13196'}