<jupyter_start><jupyter_text>To be, or not to be<NAME> (2762306) Shakespearean Play Datahttps://www.kaggle.com/kingburrito666/shakespeare-plays/download  Goal- Perform feature engineering upon the data to enable classification- Build a classification model in order to be able to determine the character in the play based on other columns in the dataset<jupyter_code>import pandas as pd
import numpy as np
import datetime as dt
import seaborn as sns
import matplotlib.pyplot as plt
import math
from statistics import mean, stdev
import string

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
from sklearn.svm import SVC<jupyter_output><empty_output><jupyter_text>Read in the Data<jupyter_code>df = pd.read_csv("../data/external/Shakespeare_data.csv")
df.head()<jupyter_output><empty_output><jupyter_text>First GlanceLooking through the data, we have some cleaning to do.- Sometimes there is no player or dialogue (stage directions)- Punctuation in the "PlayerLine", want words only- Separate out the "ActSceneLine" column to its individual components- Create a dictionary of plays with a mapping of player to their respective lines<jupyter_code># Remove NaN rows from the dataset (these represent stage directions/non-dialogue)
print("With NaNs = ", df.shape)
df = df.dropna()
print("Without NaNs = ", df.shape)
df.head()
# Remove punctuation from the PlayerLine column

clean_lines = []

for row in df["PlayerLine"]:
    clean_lines.append(row.translate(str.maketrans('','', string.punctuation)))
    #print(row.translate(str.maketrans('','',string.punctuation)))
    
df.insert(len(df.columns), 'CleanLines', clean_lines, True)
df.head()
# Separate out the ActSceneLine column

acts = []
scenes = []
lines = []

for row in df["ActSceneLine"]:
    #print(row)
    
    act = row.split('.')[0]
    acts.append(act)
    #print(act)
    
    scene = row.split('.')[1]
    scenes.append(scene)
    #print(scene)
    
    line = row.split('.')[2]
    lines.append(line)
    #print(line)
# Remove old ActSceneLine column and add the new individual columns
df.insert(len(df.columns), 'Act', acts, True)
df.insert(len(df.columns), 'Scene', scenes, True)
df.insert(len(df.columns), 'Line', lines, True)
df = df.drop(axis=1, columns='ActSceneLine')

df.head()
# Translating Plays and Player into numeric values

# Print the length of unique players in the player column
print("Unique players = ", len(df.Player.unique()))

# Set players to be a series of category
unique_name_codes = pd.Series(df["Player"], dtype="category")

# Replace the string label of a player's name with a numeric code
unique_name_codes = unique_name_codes.cat.codes

# Print the largest value present in unique players
print(max(unique_name_codes))

# Add the name codes to the dataframe
df.insert(len(df.columns), 'PlayerCode', unique_name_codes, True)


# Print the length of unique plays in the plays column
print("Unique Plays = ", len(df.Play.unique()))

# Set plays to be a series of category
unique_play_codes = pd.Series(df["Play"], dtype="category")

# Replace the string label of a play's name with a numeric code
unique_play_codes = unique_play_codes.cat.codes

# print the largest value present in unique plays
print(max(unique_play_codes))

# add the play codes to the dataframe
df.insert(len(df.columns), 'PlayCode', unique_play_codes, True)


df.head()<jupyter_output>Unique players =  934
933
Unique Plays =  36
35<jupyter_text>Now that we have all features in terms of numeric values, let's try to throw them at a random forest classifier- Following https://www.datacamp.com/community/tutorials/random-forests-classifier-python for inspiration<jupyter_code>le = preprocessing.LabelEncoder()
df["PlayerLine"] = le.fit_transform(df["PlayerLine"])
df["CleanLines"] = le.fit_transform(df["CleanLines"])


X=df[['PlayerLinenumber', 'CleanLines', 'Act', 'Scene', 'Line', 'PlayCode']]  # Features
y=df['Player']  # Labels

# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # 80% training and 20% test

#Create a Gaussian Classifier
clf=RandomForestClassifier(n_estimators=250, max_features=3)

#Train the model using the training sets y_pred=clf.predict(X_test)
clf.fit(X_train,y_train)

y_pred=clf.predict(X_test)
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))
# Using simple Multi-Layer Perceptron from https://scikit-learn.org/stable/modules/neural_networks_supervised.html

X=df[['PlayerLinenumber', 'CleanLines', 'Act', 'Scene', 'Line', 'PlayCode']]  # Features
y=df['Player']  # Labels

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2) # 80% training and 20% test

clf = MLPClassifier(activation='relu', alpha=1e-05, batch_size='auto',
              beta_1=0.9, beta_2=0.999, early_stopping=False,
              epsilon=1e-08, hidden_layer_sizes=(5, 2),
              learning_rate='constant', learning_rate_init=0.001,
              max_iter=100, momentum=0.9, n_iter_no_change=10,
              nesterovs_momentum=True, power_t=0.5, random_state=1,
              shuffle=True, solver='lbfgs', tol=0.0001,
              validation_fraction=0.1, verbose=False, warm_start=False)

clf.fit(X, y) 

y_pred=clf.predict(X_test)
print("Accuracy:",metrics.accuracy_score(y_test, y_pred))<jupyter_output>Accuracy: 0.01901954258000095
{'fraction_non_alphanumeric': 0.1134020618556701, 'fraction_numerical': 0.01718213058419244, 'mean_word_length': 5.248587570621469, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '579575', 'license': 'MIT', 'n_tokens_mistral': 1860, 'n_tokens_neox': 1777, 'n_words': 506, 'path': 'notebooks/01-to-be-or-not-to-be.ipynb', 'repo_name': 'daltonhahn/eecs731-to_be_or_not', 'size': '5534'}