<jupyter_start><jupyter_text>Naive Bayes ClassifierThis notebook can be used to train a Naive Bayes classifier using a training set of Quora questions and making predictions on a test set. The data used for this notebook is a Kaggle dataset found [Here](https://www.kaggle.com/c/quora-insincere-questions-classification/data). To follow along with this notebook, please add the 'train.csv' and 'test.csv' to the /data directory of this repository.<jupyter_code>import pandas as pd
import src.NB_Classifier as NB

from sklearn.model_selection import train_test_split

%load_ext autoreload
%autoreload 2
# Read in training data
train_df = pd.read_csv('data/train.csv')
train_df.head()
# Extract the training data and corresponding labels
text = train_df['question_text']
labels = train_df['target']

# Split into training and validation sets
X_train, X_val, y_train, y_val = train_test_split(text, labels,\
                                                  test_size=0.2)
# Initialize and train the classifier
classifier = NB.NB_Classifier('data/stopwords.txt')

# Only use words that appear more than 50 times in the dataset
classifier.collect_dictionary(X_train, 50)

classifier.train(X_train, y_train, k=1)
# Evaluate the classifier on the validation set
acc = classifier.evaluate(X_val, y_val)
print('Accuracy on the validation set is {:0.3f}'.format(acc))<jupyter_output>Accuracy on the validation set is 0.926<jupyter_text>PredictionsThe remaining code will generate predictions using the trained Naive Bayesclassifier and the test.csv file.<jupyter_code>test_df = pd.read_csv('data/test.csv')
test_qs = test_df['question_text']
test_df.head()
# Generate predictions to each question and add to the dataframe
preds = classifier.generate_preds(test_qs)
test_df['predictions'] = pd.Series(preds).values
test_df.head()
# Drop unnecessary columns from the df and generate prediction csv
test_df = test_df.drop(['question_text'], axis=1)
test_df.to_csv('submissions.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08594138102334824, 'fraction_numerical': 0.007451564828614009, 'mean_word_length': 5.313479623824452, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '38800', 'license': 'MIT', 'n_tokens_mistral': 631, 'n_tokens_neox': 600, 'n_words': 199, 'path': 'src/naive_bayes.ipynb', 'repo_name': 'bmmidei/Toxic-Comment-Classifier', 'size': '2013'}