<jupyter_start><jupyter_text>Data on North American mushrooms from https://archive.ics.uci.edu/ml/datasets/Mushroom. We're answering questions based on the data:* Can a machine learning model reliably identify poisonous mushrooms based on the data?* Does any one feature reliably classify mushroom toxicity?* Can we formulate simple, memorizable rules for reliably classifying mushroom toxicity?<jupyter_code>import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score
from sklearn.tree import export_graphviz, DecisionTreeClassifier
from functools import reduce<jupyter_output><empty_output><jupyter_text>We are importing the "expanded" data file, which contains more samples than the single-character version.<jupyter_code># Input column names, which aren't included in the expanded data file
names = [
    'Toxicity',
    'Cap Shape',
    'Cap Surface',
    'Cap Color',
    'Bruises?',
    'Odor',
    'Gill Attachment',
    'Gill Spacing',
    'Gill Size',
    'Gill Color',
    'Stalk Shape',
    'Stalk Root',
    'Stalk Surface Above Ring',
    'Stalk Surface Below Ring',
    'Stalk Color Above Ring',
    'Stalk Color Below Ring',
    'Veil Type',
    'Veil Color',
    'Ring Number',
    'Ring Type',
    'Spore Print Color',
    'Population',
    'Habitat'
]
df = pd.read_csv('data/expanded', skiprows=9, names=names, index_col=None, engine='python', skipfooter=1); df
# Assess data variability
df.describe().loc['unique']<jupyter_output><empty_output><jupyter_text>Veil type has only one value, so we can remove that feature later.<jupyter_code># Assess missing data
df.isna().sum()<jupyter_output><empty_output><jupyter_text>There appears to be no missing data.<jupyter_code># Transform features

# Convert binary bruised state to boolean values
def bool_bruises(dfin):
    dfin_no_bruises = dfin.drop(columns='Bruises?')
    dfin_bool_bruises = dfin['Bruises?'].apply(lambda x: x == 'BRUISES')
    return pd.concat([dfin_no_bruises, dfin_bool_bruises], axis=1)

# Drop veil type, because it has one value
drop_veil_type = lambda dfin: dfin.drop(columns='Veil Type')

# Convert class to boolean values
def bool_toxicity(dfin):
    dfin_no_toxicity = dfin.drop(columns='Toxicity')
    dfin_toxic = dfin['Toxicity'].apply(lambda x: x == 'POISONOUS')
    dfin_toxic.name = 'Toxic?'
    return pd.concat([dfin_toxic, dfin_no_toxicity], axis=1)

# One-hot encode
one_hot_encode = lambda dfin: pd.get_dummies(dfin)

fns = [bool_bruises, bool_toxicity, drop_veil_type, one_hot_encode]
df_trans = reduce(lambda res, fn: fn(res), fns, df); df_trans<jupyter_output><empty_output><jupyter_text>Although other categories beside _Bruised?_ may have only two categories, _Bruised?_ was the only column treated as binary in the data, so we have transformed it to boolean values to reflect that.<jupyter_code># Assess distribution of class
df_trans['Toxic?'].mean()<jupyter_output><empty_output><jupyter_text>We have similarly sized edible and poisonous samples.<jupyter_code># Create machine learning model
X = df_trans.drop(columns='Toxic?')
y = df_trans['Toxic?']
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
random_forest_model = RandomForestClassifier(random_state=0).fit(X_train, y_train); random_forest_model
f1_score(random_forest_model.predict(X_test), y_test)<jupyter_output><empty_output><jupyter_text>That's extremely accurate!<jupyter_code># Save the DOT data and convert it to PNG
! mkdir dot
! mkdir images
export_graphviz(random_forest_model.estimators_[0], out_file='dot/subestimatortree.dot', feature_names=X.columns, class_names=['Edible', 'Poisonous'])
! dot -Tpng dot/subestimatortree.dot -o images/subestimatortree.png
# Create a decision tree with a depth of 1
decision_tree_model1 = DecisionTreeClassifier(random_state=0, max_depth=1).fit(X, y); decision_tree_model1
# Save the DOT data and convert it to PNG
export_graphviz(decision_tree_model1, out_file='dot/tree1.dot', feature_names=X.columns, class_names=['Edible', 'Poisonous'])
! dot -Tpng dot/tree1.dot -o images/tree1.png
# Create a decision tree with a depth of 2
decision_tree_model2 = DecisionTreeClassifier(random_state=0, max_depth=2).fit(X, y)
dot_data2 = export_graphviz(decision_tree_model2, out_file='dot/tree2.dot', feature_names=X.columns, class_names=['Edible', 'Poisonous'])
graph2 = graphviz.Source(dot_data2)
! dot -Tpng dot/tree2.dot -o images/tree2.png
# Create a decision tree with a depth of 3
decision_tree_model3 = DecisionTreeClassifier(random_state=0, max_depth=3).fit(X, y)
dot_data3 = export_graphviz(decision_tree_model3, out_file='dot/tree3.dot', feature_names=X.columns, class_names=['Edible', 'Poisonous'])
graph3 = graphviz.Source(dot_data3)
! dot -Tpng dot/tree3.dot -o images/tree3.png
# Since odor is important, what were the different odors?
df['Odor'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10742464090633219, 'fraction_numerical': 0.008092251669026907, 'mean_word_length': 6.1965065502183405, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '607244', 'license': 'MIT', 'n_tokens_mistral': 1730, 'n_tokens_neox': 1653, 'n_words': 438, 'path': 'analysis.ipynb', 'repo_name': 'equinlan/mushroom', 'size': '4943'}