<jupyter_start><jupyter_text>MACHINE LEARNING CAPSTONE PROJECT:   TOXIC COMMENT CLASSIFICATION 1. Import necessary files 2. Read the train.csv file 3. List the various fields in our dataset<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
df2 = pd.read_csv("labeled_data-Davidson.csv")
df2 = df2.drop(['Unnamed: 0', 'count','hate_speech','offensive_language','neither'], axis=1)
df2.columns = ['label', 'comment_text']
df2 = df2[['comment_text','label']]
df2
def f2(row):
    if row['label'] == 1 or row['label'] == 0 :
        val = 1
    else:
        val = 0
    return val
df2['label'] = df2.apply(f2, axis=1)
df2
#Read the csv file into dataframe df
df = pd.read_csv("train.csv")
def f(row):
    if row['toxic'] == 1 or row['severe_toxic'] == 1 or row['obscene'] == 1 or row['threat'] == 1 or row['insult'] == 1 or row['identity_hate'] == 1:
        val = 1
    else:
        val = 0
    return val

df['label'] = df.apply(f, axis=1)
df = df.drop(['id','toxic','severe_toxic','obscene','threat','insult','identity_hate'], axis=1)
df
df.groupby('label').count()
df_combine = pd.concat([df,df2], axis=0)
df_combine
df_combine.groupby('label').count()
df3 = df_combine[df_combine['label'] == 0]
df4 = df_combine[df_combine['label'] == 1]


df3 =df3.sample(n=40000)
df3
df_combine_final = pd.concat([df3,df4], axis=0)
df_combine_final.groupby('label').count()<jupyter_output><empty_output><jupyter_text>Separate the comment field data and outcome labels<jupyter_code>comment = df_combine_final['comment_text']
print(comment.head())
comment = comment.as_matrix()
label = df_combine_final[['label']]
print(label.head())
label = label.as_matrix()<jupyter_output>label
70525       0
146942      0
19882       0
49100       0
55527       0<jupyter_text>Data Visualisations Let us analyse the no. of comments having lengths varying from 0 to 1200<jupyter_code>x = [len(comment[i]) for i in range(comment.shape[0])]

print('average length of comment: {:.3f}'.format(sum(x)/len(x)) )
bins = [1,200,400,600,800,1000,1200]
plt.hist(x, bins=bins)
plt.xlabel('Length of comments')
plt.ylabel('Number of comments')       
plt.axis([0, 1200, 0, 90000])
plt.grid(True)
plt.show()<jupyter_output>average length of comment: 292.327<jupyter_text>Number of comments classified as toxic,severe_toxic,....etc depending on their lengths  Remove excessive length commentsSome very large length comments can be seen, in our dataset. These pose serious problems like adding excessively more words to the training dataset, causing training time to increase and accuracy to decrease!Hence, a threshold of 400 characters will be created and only comments which have length smaller than 400 will be used further.<jupyter_code>comments = []
labels = []

for ix in range(comment.shape[0]):
    if len(comment[ix])<=400:
        comments.append(comment[ix])
        labels.append(label[ix])
labels = np.asarray(labels)
print(len(comments))
labels<jupyter_output><empty_output><jupyter_text>Hence, after removing comments longer than 400 characters, we are still left with more than 69000 comments, which seems enough for training purposes.  Preprocessing Preprocessing involved the following steps, but these will be performed in a slightly different manner:- Removing Punctuations and other special characters- Splitting the comments into individual words- Removing Stop Words- Stemming and Lemmatising- Applying Count Vectoriser- Splitting dataset into Training and Testing  Preparing a string containing all punctuations to be removedThe string library contains punctuation characters. This is imported and all numbers are appended to this string. Also, we can notice that our comment_text field contains strings such as won't, didn't, etc which contain apostrophe character('). To prevent these words from being converted to wont/didnt, the character ' represented as \' in escape sequence notation is replaced by empty character in the punctuation string. **maketrans()** returns a translation table that maps each character in the punctuation_edit into the character at the same position in the outtab string i.e. it replaces every character in the removal list with a space, since outtab contains a string with spaces.<jupyter_code>import string
print(string.punctuation)
punctuation_edit = string.punctuation.replace('\'','') +"0123456789"
print (punctuation_edit)
outtab = "                                         "
trantab = str.maketrans(punctuation_edit, outtab)<jupyter_output>!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
!"#$%&()*+,-./:;<=>?@[\]^_`{|}~0123456789<jupyter_text>Updating the list of stop words**Stop words** are those words that are frequently used in both written and verbal communication and thereby do not have either a positive/negative impact on our statement.E.g. is, this, us,etc. Single letter words if existing or created due to any preprocessing step do not convey any useful meaning and hence can be directly removed. Hence letters from b to z, will be added to the list of stop words imported directly.<jupyter_code>from stop_words import get_stop_words
stop_words = get_stop_words('english')
stop_words.append('')

for x in range(ord('b'), ord('z')+1):
    stop_words.append(chr(x))
print (stop_words)<jupyter_output>['a', 'about', 'above', 'after', 'again', 'against', 'all', 'am', 'an', 'and', 'any', 'are', "aren't", 'as', 'at', 'be', 'because', 'been', 'before', 'being', 'below', 'between', 'both', 'but', 'by', "can't", 'cannot', 'could', "couldn't", 'did', "didn't", 'do', 'does', "doesn't", 'doing', "don't", 'down', 'during', 'each', 'few', 'for', 'from', 'further', 'had', "hadn't", 'has', "hasn't", 'have', "haven't", 'having', 'he', "he'd", "he'll", "he's", 'her', 'here', "here's", 'hers', 'herself', 'him', 'himself', 'his', 'how', "how's", 'i', "i'd", "i'll", "i'm", "i've", 'if', 'in', 'into', 'is', "isn't", 'it', "it's", 'its', 'itself', "let's", 'me', 'more', 'most', "mustn't", 'my', 'myself', 'no', 'nor', 'not', 'of', 'off', 'on', 'once', 'only', 'or', 'other', 'ought', 'our', 'ours', 'ourselves', 'out', 'over', 'own', 'same', "shan't", 'she', "she'd", "she'll", "she's", 'should', "shouldn't", 'so', 'some', 'such', 'than', 'that', "that's", 'the', 'their', 'theirs', 'them', 'themselves', 't[...]<jupyter_text>Stemming and Lemmatizing**Stemming** is the process of converting inflected/derived words to their word stem or the root form. Basically, a large number of similar origin words are converted to the same word.E.g. words like "stems", "stemmer", "stemming", "stemmed" as based on "stem". This helps in achieving the training process with a better accuracy.**Lemmatizing** is the process of grouping together the inflected forms of a word so they can be analysed as a single item. This is quite similar to stemming in its working but differs since it depends on correctly identifying the intended part of speech and meaning of a word in a sentence, as well as within the larger context surrounding that sentence, such as neighboring sentences or even an entire document.The **wordnet library in nltk** will be used for this purpose. Stemmer and Lemmatizer are also imported from nltk.<jupyter_code>import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer
#create objects for stemmer and lemmatizer
lemmatiser = WordNetLemmatizer()
stemmer = PorterStemmer()
#download words from wordnet library
nltk.download('wordnet')<jupyter_output>[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\cdrgv\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>We can now, loop once through all the comments applying :- punctuation removal- splitting the words by space- applying stemmer and lemmatizer- recombining the words again for further processing<jupyter_code>for i in range(len(comments)):
    comments[i] = comments[i].lower().translate(trantab)
    l = []
    for word in comments[i].split():
        l.append(stemmer.stem(lemmatiser.lemmatize(word,pos="v")))
    comments[i] = " ".join(l)<jupyter_output><empty_output><jupyter_text>Applying Count VectorizerHere we can finally convert our comments into a matrix of token counts, which signifies the number of times it occurs.<jupyter_code>#import required library
from sklearn.feature_extraction.text import CountVectorizer

#create object supplying our custom stop words
count_vector = CountVectorizer(stop_words=stop_words)
#fitting it to converts comments into bag of words format
tf = count_vector.fit_transform(comments)
# print(count_vector.get_feature_names())
print(tf.shape)<jupyter_output>(63032, 55724)<jupyter_text>Hence from its shape we can imply that after all preprocessing we have a list of 52905 words in total. Splitting dataset into training and testing- Since the system was going out of memory using train_test_split, I had jumbled all the indexes in the beginning itself. - The shuffle function defined here performs the task of assigning first 2/3rd values to train and remaining 1/3rd values to the test set.<jupyter_code>def shuffle(matrix, target, test_proportion):
    ratio = int(matrix.shape[0]/test_proportion)
    X_train = matrix[ratio:,:]
    X_test =  matrix[:ratio,:]
    Y_train = target[ratio:,:]
    Y_test =  target[:ratio,:]
    return X_train, X_test, Y_train, Y_test

X_train, X_test, Y_train, Y_test = shuffle(tf, labels,3)

print(X_test.shape)
print(X_train.shape)
print(X_train)<jupyter_output>(21010, 55724)
(42022, 55724)
  (0, 9775)	1
  (0, 38245)	1
  (0, 41959)	1
  (0, 50057)	1
  (0, 53093)	1
  (0, 14500)	1
  (0, 46048)	1
  (0, 14656)	1
  (0, 131)	1
  (0, 39178)	1
  (0, 2501)	1
  (0, 54565)	1
  (0, 50648)	1
  (0, 32352)	1
  (0, 34885)	1
  (0, 9281)	1
  (0, 47839)	1
  (0, 52767)	1
  (0, 34617)	1
  (0, 22355)	1
  (0, 15512)	1
  (1, 5433)	1
  (1, 39953)	1
  (1, 36453)	1
  (1, 38561)	1
  :	:
  (42019, 4746)	1
  (42019, 35925)	1
  (42019, 30782)	1
  (42019, 48364)	1
  (42019, 48941)	1
  (42019, 40304)	1
  (42019, 33765)	1
  (42019, 27624)	1
  (42019, 20766)	1
  (42020, 33180)	1
  (42020, 17313)	1
  (42020, 6451)	1
  (42020, 912)	1
  (42020, 54786)	1
  (42020, 52117)	1
  (42020, 12089)	1
  (42020, 11165)	1
  (42020, 13817)	1
  (42020, 27705)	1
  (42021, 54860)	1
  (42021, 5000)	1
  (42021, 47081)	1
  (42021, 53204)	1
  (42021, 27624)	1
  (42021, 18209)	1<jupyter_text>Implementation : Let us define all the evaluation metrics<jupyter_code>from sklearn.metrics import hamming_loss
from sklearn.metrics import accuracy_score
from sklearn.metrics import log_loss

def evaluate_score(Y_test,predict): 
    loss = hamming_loss(Y_test,predict)
    print("Hamming_loss : {}".format(loss*100))
    accuracy = accuracy_score(Y_test,predict)
    print("Accuracy : {}".format(accuracy*100))<jupyter_output><empty_output><jupyter_text>Starting with the First Model - Problem Transformation Methods :**These include the Binary Relevance, Label Powerset and Classifier Chain methods. Implementations of these methods is available in the scikit-multilearn library. **- I will be implementing the most basic method,which is the **Binary Relevance** method from scratch. It does not take into account the interdependence of labels and basically creates a separate classifier for each of the labels.- Scikit-multilearn library's classifier will also be imported and tested with different classifiers to observe if it gives similar results. 1. Binary Relevance (BR) Method with MultinomialNB classifiers (from scratch)<jupyter_code>from sklearn.naive_bayes import MultinomialNB
# clf will be the list of the classifiers for all the 6 labels
# each classifier is fit with the training data and corresponding classifier
clf = []
for ix in range(1):
    clf.append(MultinomialNB())
    clf[ix].fit(X_train,Y_train[:,ix])
# predict list contains the predictions, it is transposed later to get the proper shape
predict = []
for ix in range(1):
    predict.append(clf[ix].predict(X_test))

predict = np.asarray(np.transpose(predict))
print(predict.shape)
# 沒有加入dataset
evaluate_score(Y_test,predict)
# 加dataset
evaluate_score(Y_test,predict)
# 調整成兩個label數字差不多
evaluate_score(Y_test,predict)<jupyter_output>Hamming_loss : 21.813422179914326
Accuracy : 78.18657782008567
{'fraction_non_alphanumeric': 0.12660821109563222, 'fraction_numerical': 0.056297631729902486, 'mean_word_length': 5.117293233082707, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '122223', 'license': 'MIT', 'n_tokens_mistral': 4597, 'n_tokens_neox': 3994, 'n_words': 1382, 'path': 'Final_Report.ipynb', 'repo_name': 'CongChen2017/TextInfo-project-working', 'size': '12203'}