<jupyter_start><jupyter_text>SLU17 - Data Sufficiency and SelectionIn this notebook you will cover the following:- Feature Importance- Single Factor Analysis- Learning Curves<jupyter_code>import math
import pandas as pd
import numpy as np
import sklearn
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, learning_curve
from sklearn.metrics import roc_auc_score
from matplotlib import pyplot as plt
from base64 import b64encode, b64decode # just for grading purposes

%matplotlib inline
# load up a classification dataset

X = pd.read_csv('data/exercise_X.csv')
y = pd.read_csv('data/exercise_y.csv')['label']
# give X a quick look
X.head()
# looks like a balanced binary target
y.value_counts()<jupyter_output><empty_output><jupyter_text>Find the first obviously useless featureCan you determine which of the features contains all uniques and therefore cannot have any predictive power?<jupyter_code># Use this cell to determine which of the features serves as a categorical
# feature and contains all uniques

# set the variable feature_all_unique to the name of the feature
# that contains all uniques
feature_all_unique = None

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert b64encode(feature_all_unique.encode()) == b'aWQ='
### END TESTS<jupyter_output><empty_output><jupyter_text>Find the second obviously useless featureThis one doesn't contain all uniques but based upon some Single Factor Analysis you should be able to determine which feature isn't worthbothering with.<jupyter_code># use this cell to do some more SFA on other features to determine
# which of them is useless

# Use this cell to determine the other obviously useless feature
other_useless_feature = None

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert b64encode(other_useless_feature.encode()) == b'ZmVhdHVyZV84'
### END TESTS
# now drop the features that you determined to be useless

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert b64encode(str(sorted(X.columns)).encode()) == b'WydmZWF0dXJlXzAnLCAnZmVhdH<KEY>
### END TESTS<jupyter_output><empty_output><jupyter_text>Find the rest of the useless featuresSingle Factor Analysis isn't likely to do much in helping us to determinewhich of the rest of the features are useless. We'll need to use feature_importances in order to find the rest of these bad boys<jupyter_code># in order to use feature importances you will need to import a tree
# based algorithm. Let's use a decision tree classifier

# import your decision tree classifier here the exact same way that
# it is done in the learning notebooks

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert DecisionTreeClassifier, 'Import that bad boy from the sklearn.tree module'
### END TESTS
# Now let's train the classifier and get the feature importances

# Create your classifier, assign it to the clf variable and then
# train it on X and y
clf = None

# once the classifier is trained, set the feature importances here
# make it a pandas series with the index being the column names
# so that we can visualize the results
feature_importances = None

# set the random_state=1 and max_depth=5 or the tests won't pass!
# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert isinstance(clf, sklearn.tree.DecisionTreeClassifier), 'The classifier must be a DecisionTreeClassifier'
assert clf.random_state == 1, 'random_state must be 1'
assert clf.max_depth == 5, 'max_depth must be 5'
assert feature_importances.equals(pd.Series(clf.feature_importances_, index=X.columns)), 'feature_importances must be a pandas series with the X columns as the index'
### END TESTS
feature_importances.plot.barh();
# now remove 3 additional useless features

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert b64encode(str(sorted(X.columns)).encode()) == b'WydmZWF0dXJlXzInLCAnZmVhdHVyZV8zJywgJ2ZlYXR1cmVfNCcsICdmZWF0dXJlXzYnLCAnZmVhdHVyZV83J10='
### END TESTS<jupyter_output><empty_output><jupyter_text>The learning curveOkay now that we have gotten rid of all those useless features, let's focus on getting a sense for how much data we need in order to havereasonable performance.<jupyter_code># Now create a dataframe that has a single feature that is the
# cross validation score in order to help us understand
# how increasing amounts of data affect the performance

# HINT: just use the snippet from the Learning Notebook
train_scores_mean = None
test_scores_mean = None

# instantiate a classifier that you will inspect the learning rate of
clf = DecisionTreeClassifier(max_depth=5, random_state=1)

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
# round in order to compensate for implementation details

assert math.isclose(sum(train_scores_mean), 4.5, rel_tol=1e-2)
assert math.isclose(sum(test_scores_mean), 3.8, rel_tol=1e-2)
### END TESTS
learning_curve_df = pd.DataFrame({
    'Training Scores': train_scores_mean,
    'Test Set scores': test_scores_mean
}, index=train_sizes)

learning_curve_df.plot.line(
    title='Decision Tree Learning Curve'
);
# Now select the minimum training set size that this particular classifier
# seems to need before it's learning rate stabilizes

min_train_set_size = None

# YOUR CODE HERE
raise NotImplementedError()
### BEGIN TESTS
assert min_train_set_size <= int(b64decode(b'MzAw'))
### END TESTS<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06623616236162362, 'fraction_numerical': 0.00867158671586716, 'mean_word_length': 5.896946564885496, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '501745', 'license': 'MIT', 'n_tokens_mistral': 1665, 'n_tokens_neox': 1570, 'n_words': 637, 'path': 'units/SLU17_Data_Sufficiency_And_Selection/Exercise notebook - SLU17 (Data Sufficiency and Selection).ipynb', 'repo_name': 'manuboat/LDSSA', 'size': '5530'}