<jupyter_start><jupyter_text>Part I: Loading the Data In order to work with this data set, we first have to load it. The file containing the data was in CSV format, so after downloading it, we use the `read_csv` method to read the data into a Pandas data frame.<jupyter_code>import pandas as pd

# Dataset Source: https://www.openml.org/d/24
original_mushroom_data = pd.read_csv('mushrooms.csv')
original_mushroom_data.head()<jupyter_output><empty_output><jupyter_text>This data originated from (https://www.openml.org/d/24) and was extracted from "The Audubon Society Field Guide to North American Mushrooms" (1981). Below, is a chart containing the meaning of every code contained in each column: | Column | Code ||---|---|| classes | edible=e, poisonous=p || cap-shape | bell=b,conical=c,convex=x,flat=f,knobbed=k,sunken=s || cap-surface | fibrous=f,grooves=g,scaly=y,smooth=s || cap-color | brown=n,buff=b,cinnamon=c,gray=g,green=r,pink=p,purple=u,red=e,white=w,yellow=y || bruises | bruises=t,no=f || odor | almond=a,anise=l,creosote=c,fishy=y,foul=f,musty=m,none=n,pungent=p,spicy=s || gill-attachment | attached=a,descending=d,free=f,notched=n || gill-spacing | close=c,crowded=w,distant=d || gill-size | broad=b,narrow=n || gill-color | black=k,brown=n,buff=b,chocolate=h,gray=g, green=r,orange=o,pink=p,purple=u,red=e,white=w,yellow=y || stalk-shape | enlarging=e,tapering=t || stalk-root | bulbous=b,club=c,cup=u,equal=e,rhizomorphs=z,rooted=r,missing=? || stalk-surface-above-ring | fibrous=f,scaly=y,silky=k,smooth=s || stalk-surface-below-ring | fibrous=f,scaly=y,silky=k,smooth=s || stalk-color-above-ring | brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y || stalk-color-below-ring | brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y || veil-type | partial=p,universal=u || veil-color | brown=n,orange=o,white=w,yellow=y || ring-number | none=n,one=o,two=t || ring-type | cobwebby=c,evanescent=e,flaring=f,large=l,none=n,pendant=p,sheathing=s,zone=z || spore-print-color | black=k,brown=n,buff=b,chocolate=h,green=r,orange=o,purple=u,white=w,yellow=y || population | abundant=a,clustered=c,numerous=n,scattered=s,several=v,solitary=y || habitat | grasses=g,leaves=l,meadows=m,paths=p,urban=u,waste=w,woods=d<jupyter_code># Get size of the data set: (rows, columns)
original_mushroom_data.shape<jupyter_output><empty_output><jupyter_text>Part II: Data Wrangling and Cleaning The first obstacle in this dataset is that every data point in every column is stored as a codified string-value, with decoded values symbolized in the previous table. Our first task should be to create a duplicate set of data where all string-values are hot-encoded into binary (0 or 1) values, assuming that no ordinal data is present.<jupyter_code># First, remove all extraneous single quotes in the dataframe "'" by replacing 
# all single quotes with empty whitespace "" values
original_mushroom_data = original_mushroom_data.applymap(lambda x: x.replace("'", ""))
original_mushroom_data.head()
# Check for any missing values
original_mushroom_data.isna().sum()
# One of our column names "bruises%3F" got mangled somehow, so rename this
original_mushroom_data.rename(columns={"bruises%3F": "bruises"}, inplace=True)
original_mushroom_data.columns
# Create a duplicate of the dataset, with numerical instead of categorical data
decoded_mushroom_data = pd.get_dummies(original_mushroom_data)
decoded_mushroom_data.head()
decoded_mushroom_data.dtypes<jupyter_output><empty_output><jupyter_text>Part III: Storing the Data Now that the data has been ingested and cleaned, we can store it in a MySQL database.<jupyter_code># This code was run in the MySQL terminal, in order to create our database:
## CREATE DATABASE mushroom;
import pymysql
import getpass
from sqlalchemy import create_engine

# Connect to the new database with sqlalchemy
p = getpass.getpass(prompt='Password: ', stream=None) 
engine = create_engine('mysql+pymysql://root:'+p+'@localhost/mushroom')
# Store both original and decoded dataframes as tables in the new MySQL database
original_mushroom_data.to_sql('mushroom_cat', engine, if_exists='replace', index=False)
decoded_mushroom_data.to_sql('mushroom_num', engine, if_exists='replace', index=False)
# Read back the data from the new MySQL database to confirm success
pd.read_sql_query('SELECT * FROM mushroom.mushroom_cat', engine).head()
# Same as above, but for our numerical data
pd.read_sql_query('SELECT * FROM mushroom.mushroom_num', engine).head()
# Export our clean data as new CSV files, to the "export" directory
original_mushroom_data.to_csv('./export/categorical_mushroom_data.csv', index=False)
decoded_mushroom_data.to_csv('./export/numerical_mushroom_data.csv', index=False)<jupyter_output><empty_output><jupyter_text>Part IV: Data Analysis<jupyter_code># Display the number of unique values in each column, and frequency of each column's mode
original_mushroom_data.describe().T
# Explore the distribution of Edible vs Poisonous mushrooms
original_mushroom_data['class'].value_counts()
# Explore the distribution of Cap Colors
original_mushroom_data['cap-color'].value_counts()<jupyter_output><empty_output><jupyter_text>brown=n, buff=b, cinnamon=c, gray=g, green=r, pink=p, purple=u, red=e, white=w, yellow=y<jupyter_code># Explore distribution of Poisonous mushrooms by Cap Color
pd.DataFrame(original_mushroom_data.groupby(['cap-color','class']).size(), \
             columns=["Total"])<jupyter_output><empty_output><jupyter_text>Above, we can see that the distribution of poisonous and edible mushrooms is fairly even (50/50 split) across all cap colors, with the exception of Green (r) and Purple (u) which are both exclusively edible.<jupyter_code># Explore distribution of Odors
original_mushroom_data['odor'].value_counts()<jupyter_output><empty_output><jupyter_text>almond=a, anise=l, creosote=c, fishy=y, foul=f, musty=m, none=n, pungent=p, spicy=s<jupyter_code># Explore distribution of Poisonous mushrooms by Odor
pd.DataFrame(original_mushroom_data.groupby(['odor','class']).size(), columns=["Total"])<jupyter_output><empty_output><jupyter_text>Above, we can see that mushrooms with odors of almond (a) and anise (l) are exclusively edible mushrooms, whereas all other columns -- except for those with no smell (n) -- are exclusively poisonous mushrooms.  Part V: Feature Selection Now, with a better understanding of the data and some insight as to how our column-attributes can help determine if a given mushroom is poisonous or not, we can proceed to prep our data set for modeling. Our goal is to train a regression model to predict if any mushroom is poisonous or not based on its physical attributes.<jupyter_code># Display the correlation matrix for our numerical data
decoded_mushroom_data.corr()
# Get the correlation coefficients ONLY for the poison mushrooms "class" column:
pd.DataFrame(decoded_mushroom_data.corr()['class_p'].sort_values())<jupyter_output><empty_output><jupyter_text>The correlation coefficients here can help us select the best features to train the data with. We should pick those features that have the strongest correlations (both positive and negative), in order to minimize noise and optimize our regression model.<jupyter_code># Get all column names with correlation higher than 0.4 and lower than -0.4
correlations = pd.DataFrame(decoded_mushroom_data.corr()['class_p'].sort_values())
X_columns = []

for index, row in correlations.iterrows():
    if (0.4 < row.class_p < 1):
        X_columns.append(index)
    elif (-1 < row.class_p < -0.4):
        X_columns.append(index)
        
X_columns<jupyter_output><empty_output><jupyter_text>We can visualize the correlation matrix of all of our selected X-column features with a heatmap, to make sure there is no duplicate data:<jupyter_code>import seaborn as sns

%matplotlib inline 

sns.heatmap(decoded_mushroom_data[X_columns].corr())<jupyter_output><empty_output><jupyter_text>Because some columns originally only had two values, we end up with some columns in our X-features list that contain complimentary data, which we should remove. These are the columns that have correlation coefficients of -1 in the above heatmap.<jupyter_code># Remove columns with duplicate data
X_columns.remove('gill-size_n')
X_columns.remove('bruises_t')
X_columns.remove('stalk-surface-above-ring_s')
X_columns.remove('stalk-surface-below-ring_s')

# Display new list of X-column features
X_columns<jupyter_output><empty_output><jupyter_text>Part VI: Model Training and Evaluation We'll start by splitting the data into two sets: one for training a regression model, and one for testing predictions on that model.<jupyter_code>from sklearn.model_selection import train_test_split

# Split X and Y sets into 80% training data and 20% testing data:
X_data = decoded_mushroom_data[X_columns]
Y_data = decoded_mushroom_data['class_p']
X_train, X_test, Y_train, Y_test  = train_test_split(X_data, Y_data, test_size = 0.2)<jupyter_output><empty_output><jupyter_text>With our data prepared, we can now create and fit our model onto the training data:<jupyter_code>from sklearn import linear_model

# Initialize the Logistic Regression model
log_model = linear_model.LogisticRegression(solver='lbfgs')

# Train the model
log_model.fit(X_train, Y_train) 
from sklearn import metrics

# Create predictions based on test data, and then evaluate the Linear Regression model
y_pred = log_model.predict(X_test)
print("Accuracy Score:", metrics.accuracy_score(Y_test, y_pred))
metrics.confusion_matrix(Y_test, y_pred)
from sklearn.neighbors import KNeighborsClassifier

# Create an alternate machine learning model with KNN
neighbor_model = KNeighborsClassifier(n_neighbors=3)
neighbor_model.fit(X_train, Y_train)
# Create predictions based on test data, and then evaluate the KNN model
y_pred_knn = neighbor_model.predict(X_test)
print("Accuracy Score:", metrics.accuracy_score(Y_test, y_pred_knn))
metrics.confusion_matrix(Y_test, y_pred_knn)<jupyter_output>Accuracy Score: 0.9815384615384616
{'fraction_non_alphanumeric': 0.10449247933061062, 'fraction_numerical': 0.005080187269648371, 'mean_word_length': 7.019169329073482, 'pattern_counts': {'":': 1, '<': 60, '<?xml version=': 0, '>': 56, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '268923', 'license': 'MIT', 'n_tokens_mistral': 3395, 'n_tokens_neox': 3234, 'n_words': 923, 'path': 'study-poisonous-mushrooms-US/main_mushrooms_code.ipynb', 'repo_name': 'haggarw3/sample-projects-for-students', 'size': '10039'}