<jupyter_start><jupyter_text>Lambda School Data Science*Unit 2, Sprint 3, Module 3*-----**I copied my data set from Monday, since yesterday I played with the Condos dataset.**-----I got a baseline metric, some accuracy scores, ran tree classifications, and calculated and visualized ROC curves.I still need to - Try xgboost.- Get your model's permutation importances. Define ML problemsYou will use your portfolio project dataset for all assignments this sprint. AssignmentComplete these tasks for your project, and document your decisions.- [ ] Choose your target. Which column in your tabular dataset will you predict?- [ ] Is your problem regression or classification?- [ ] How is your target distributed?    - Classification: How many classes? Are the classes imbalanced?    - Regression: Is the target right-skewed? If so, you may want to log transform the target.- [ ] Choose your evaluation metric(s).    - Classification: Is your majority class frequency >= 50% and < 70% ? If so, you can just use accuracy if you want. Outside that range, accuracy could be misleading. What evaluation metric will you choose, in addition to or instead of accuracy?    - Regression: Will you use mean absolute error, root mean squared error, R^2, or other regression metrics?- [ ] Choose which observations you will use to train, validate, and test your model.    - Are some observations outliers? Will you exclude them?    - Will you do a random split or a time-based split?- [ ] Begin to clean and explore your data.- [ ] Begin to choose which features, if any, to exclude. Would some features "leak" future information?If you haven't found a dataset yet, do that today. [Review requirements for your portfolio project](https://lambdaschool.github.io/ds/unit2) and choose your dataset. Question: **Is this mushroom poisonous?**Data set: https://archive.ics.uci.edu/ml/datasets/mushroom**This is NOT one of the most popular UCI datasets.**  Ingest Data<jupyter_code>%%capture
import sys

# If Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/gptix/Project2/master/'
    !pip install category_encoders==2.*

# If local:
else:
    DATA_PATH = '../data/'
import pandas as pd

# I extracted these column names from the text file 
# https://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/agaricus-lepiota.names
# I added the column 'poisonous', which was unlabeled in the original set.

colnames = ['poisonous', 'cap-shape', 'cap-surface', 'cap-color', 'bruises?',
            'odor', 'gill-attachment', 'gill-spacing', 'gill-size',
            'gill-color', 'stalk-shape', 'stalk-root', 
            'stalk-surface-above-ring', 'stalk-surface-below-ring',
            'stalk-color-above-ring', 'stalk-color-below-ring', 'veil-type', 
            'veil-color', 'ring-number', 'ring-type', 'spore-print-color', 
            'population', 'habitat']

filename = 'agaricus-lepiota.data?token=<KEY>'

df = pd.read_csv(DATA_PATH + filename, header=None, names=colnames)
df.shape<jupyter_output><empty_output><jupyter_text>Make Backup for backup and restore<jupyter_code>df_bak = df
# df = df_bak<jupyter_output><empty_output><jupyter_text>Examine Data<jupyter_code>df.head()
# examine data<jupyter_output><empty_output><jupyter_text>Select Target (and feature)<jupyter_code># I choose to predict 'poisonous'.
target = 'poisonous'

features = list(set(colnames) - set([target]))

print(features)
y = df[target]
y.describe()

features<jupyter_output>['spore-print-color', 'ring-type', 'habitat', 'veil-color', 'stalk-color-below-ring', 'cap-surface', 'ring-number', 'cap-color', 'veil-type', 'gill-spacing', 'stalk-surface-above-ring', 'stalk-color-above-ring', 'cap-shape', 'gill-color', 'gill-attachment', 'stalk-root', 'gill-size', 'odor', 'population', 'stalk-surface-below-ring', 'stalk-shape', 'bruises?']<jupyter_text>Decide Regression/ClassificationThe problem is most intuitively one of **classification**.I might also build a **predicted probability** model.  How is your target distributed?Classification: - How many classes? **Two**:  - poisonous  - edible- Are the classes imbalanced? **They are eerily balanced.**<jupyter_code>y.value_counts(normalize=True).max()
# Derive binary classifiction target:
df['poisonous'] = df['poisonous'] == 'p'
df['poisonous'].value_counts()
df.columns
# examine columns
df.groupby('cap-shape')['cap-shape'].count()<jupyter_output><empty_output><jupyter_text>Convert single characters to strings.This is to aid viewing.These will be encoded back into numbers in a pipeline later.<jupyter_code>df['cap-shape'] = df['cap-shape'].map({'b' : 'bell', 'c' : 'conical', 'x' : 'convex', 'f' : 'flat', 'k' : 'WYknobbed', 's' : 'sunken'})
df['cap-surface'] = df['cap-surface'].map({'f' : 'fibrous', 'g' : 'grooves', 'y' : 'scaly', 's' : 'smooth'})
df['cap-color'] = df['cap-color'].map({'n' : 'brown', 'b' : 'buff', 'c' : 'cinnamon', 'g' : 'gray', 'r' : 'green', 'p' : 'pink', 'u' : 'purple', 'e' : 'red', 'w' : 'white', 'y' : 'yellow'})
df['bruises?'] = df['bruises?'].map({'t' : 'bruises', 'f' : 'no'})
df['odor'] = df['odor'].map({'a' : 'almond', 'l' : 'anise', 'c' : 'creosote', 'y' : 'fishy', 'f' : 'foul', 'm' : 'musty', 'n' : 'none', 'p' : 'pungent', 's' : 'spicy'})
df['gill-attachment'] = df['gill-attachment'].map({'a' : 'attached', 'd' : 'descending', 'f' : 'free', 'n' : 'notched'})
df['gill-spacing'] = df['gill-spacing'].map({'c' : 'close', 'w' : 'crowded', 'd' : 'distant'})
df['gill-size'] = df['gill-size'].map({'b' : 'broad', 'n' : 'narrow'})
df['gill-color'] = df['gill-color'].map({'k' : 'black', 'n' : 'brown', 'b' : 'buff', 'h' : 'chocolate', 'g' : 'gray', 'r' : 'green', 'o' : 'orange', 'p' : 'pink', 'u' : 'purple', 'e' : 'red', 'w' : 'white', 'y' : 'yellow'})
df['stalk-shape'] = df['stalk-shape'].map({'e' : 'enlarging', 't' : 'tapering'})
df['stalk-root'] = df['stalk-root'].map({'b' : 'bulbous', 'c' : 'club', 'u' : 'cup', 'e' : 'equal', 'z' : 'rhizomorphs', 'r' : 'rooted', '?' : 'missing'})
df['stalk-surface-above-ring'] = df['stalk-surface-above-ring'].map({'f' : 'fibrous', 'y' : 'scaly', 'k' : 'silky', 's' : 'smooth'})
df['stalk-surface-below-ring'] = df['stalk-surface-below-ring'].map({'f' : 'fibrous', 'y' : 'scaly', 'k' : 'silky', 's' : 'smooth'})
df['stalk-color-above-ring'] = df['stalk-color-above-ring'].map({'n' : 'brown', 'b' : 'buff', 'c' : 'cinnamon', 'g' : 'gray', 'o' : 'orange', 'p' : 'pink', 'e' : 'red', 'w' : 'white', 'y' : 'yellow'})
df['stalk-color-below-ring'] = df['stalk-color-below-ring'].map({'n' : 'brown', 'b' : 'buff', 'c' : 'cinnamon', 'g' : 'gray', 'o' : 'orange', 'p' : 'pink', 'e' : 'red', 'w' : 'white', 'y' : 'yellow'})
df['veil-type'] = df['veil-type'].map({'p' : 'partial', 'u' : 'universal'})
df['veil-color'] = df['veil-color'].map({'n' : 'brown', 'o' : 'orange', 'w' : 'white', 'y' : 'yellow'})
df['ring-number'] = df['ring-number'].map({'n' : 'none', 'o' : 'one', 't' : 'two'})
df['ring-type'] = df['ring-type'].map({'c' : 'cobwebby', 'e' : 'evanescent', 'f' : 'flaring', 'l' : 'large', 'n' : 'none', 'p' : 'pendant', 's' : 'sheathing', 'z' : 'zone'})
df['spore-print-color'] = df['spore-print-color'].map({'k' : 'black', 'n' : 'brown', 'b' : 'buff', 'h' : 'chocolate', 'r' : 'green', 'o' : 'orange', 'u' : 'purple', 'w' : 'white', 'y' : 'yellow'})
df['population'] = df['population'].map({'a' : 'abundant', 'c' : 'clustered', 'n' : 'numerous', 's' : 'scattered', 'v' : 'several', 'y' : 'solitary'})
df['habitat'] = df['habitat'].map({'g' : 'grasses', 'l' : 'leaves', 'm' : 'meadows', 'p' : 'paths', 'u' : 'urban', 'w' : 'waste', 'd' : 'woods'})

df.head()<jupyter_output><empty_output><jupyter_text>Choose your evaluation metric(s).Classification: Is your majority class frequency >= 50% and < 70% ? **Yes**. If so, you can just use accuracy if you want. I'll use **accuracy**.I'll want to use **precision**, since the cost of a false negative is high. Choose which observations you will use to train, validate, and test your model.Are some observations outliers? Will you exclude them?Will you do a random split or a time-based split?  **My data set has no time component.**  **I'll use a random split**.<jupyter_code>from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.2, random_state=1984)
train, val = train_test_split(train, test_size=0.2, random_state=1984)

print(train.shape)
print(val.shape)
test.shape<jupyter_output>(5199, 23)
(1300, 23)<jupyter_text>Find baseline accuracy<jupyter_code>df['stalk-root'].unique()
import category_encoders as ce
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeClassifier

# target - 'poisonous' # defined above
# features # defined above

X_train = train[features]
y_train = train[target]

X_test = test[features]
y_test = test[target]

X_val = val[features]
y_val = val[target]

pipeline = make_pipeline(
    ce.OrdinalEncoder(),
    DecisionTreeClassifier(max_depth=3)
)

pipeline.fit(X_train, y_train)

print('Validation Accuracy: ', pipeline.score(X_val, y_val))
X_train.columns
# Vizualize Decision Tree
import graphviz
from sklearn.tree import export_graphviz

tree = pipeline.named_steps['decisiontreeclassifier']

dot_data = export_graphviz(
    tree,
    out_file = None,
    feature_names = X_train.columns,
    # class_names = y.unique().astype(str),
    filled = True,
    # impurity=False,
    # proportion=True,
)

graphviz.Source(dot_data)<jupyter_output><empty_output><jupyter_text>ROC Curve<jupyter_code>from sklearn.metrics import roc_curve
y_pred_proba = pipeline.predict_proba(X_val)[:,-1]
fpr, tpr, thresholds = roc_curve(y_val, y_pred_proba)
pd.DataFrame({
    'False Postivie Rate' : fpr,
    'True Positive Rate' : tpr,
    'Thresholds' : thresholds
})
# Receiver Operating Chrcteristic plt
import matplotlib.pyplot as plt
plt.scatter(fpr,tpr)
plt.plot(fpr,tpr)
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate');
from sklearn.metrics import roc_auc_score
roc_auc_score(y_val, y_pred_proba)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19242273180458624, 'fraction_numerical': 0.004087736789631107, 'mean_word_length': 5.101581508515815, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '549246', 'license': 'MIT', 'n_tokens_mistral': 3713, 'n_tokens_neox': 3534, 'n_words': 942, 'path': 'module3/Jud_Taylor_assignment_applied_modeling_2.ipynb', 'repo_name': 'gptix/DS-Unit-2-Applied-Modeling', 'size': '10054'}