<jupyter_start><jupyter_text>**Creating a fake dataset of 5 comments and sample features to show how the model should work and look like**<jupyter_code>example = df2.iloc[0:5,:]
example['Comments'][0] = ':) this is a null statement.'
example['Comments'][1] = 'PAP needs to GOOOOO'
example['Comments'][2] = 'We are so excited to be here'
example['Comments'][3] = 'WHY DOES THIS ALWAYS HAPPEN TO ME!'
example['Comments'][4] = '<NAME> and <NAME>'
example
#adding features to the sample dataset

example['capitals'] = example['Comments'].apply(lambda comment: sum(1 for c in comment if c.isupper()))
example['num_exclamation_marks'] = example['Comments'].apply(lambda comment: comment.count('!'))<jupyter_output><empty_output><jupyter_text>I create a point biserial coefficient matrix and a pvalue matrix to store all the correlation values computed from doing Point Biserial Coefficient between each feature and each label<jupyter_code>PointBiserialCoefficientMatrix = df2.iloc[0:0,1:5]
PValuesMatrixStored = df2.iloc[0:0, 1:5]
PointBiserialCoefficientMatrix
PValuesMatrixStored<jupyter_output><empty_output><jupyter_text>This is where we actually start computing correlation. The pearson's method doesn't work, or at least is inaccurate Pearson Coefficient way to measure correlation. This is the standard way to measure correlation between 2 continuous variables<jupyter_code>features = ('capitals', 'num_exclamation_marks')
columns = ('Insulting', 'Anti Government', 'Xenophobic', 'Racist', 'Sexual')

rows = [{c:example[f].corr(example[c], method='pearson') for c in columns} for f in features]<jupyter_output><empty_output><jupyter_text>However, our data is of continuous (how many capitals) against binary (whether or not it is insulting, 1 or 0) To compute correlation between continuous and binary, we instead use a point-biserial correlation, which is essentially derived from the Pearson coefficient<jupyter_code>for c in columns:
  for f in features: 
    a,r = stats.pointbiserialr(example[c], example[f])
    if (r < 0.05):
      PointBiserialCoefficientMatrix.loc[f,c] = a
      PValuesMatrixStored.loc[f,c] = r
    else:
      PointBiserialCoefficientMatrix.loc[f,c] = 0
      PValuesMatrixStored.loc[f,c] = r<jupyter_output>/usr/local/lib/python3.6/dist-packages/scipy/stats/stats.py:3038: RuntimeWarning: invalid value encountered in double_scalars
  r = r_num / r_den<jupyter_text>The output below shows the correlation (ranging from -1 to +1), read the same way as you would for any correlation. +1 means strong positive correlation, -1 means negative.With a p-value <0.05 meaning significant<jupyter_code>PointBiserialCoefficientMatrix
PValuesMatrixStored<jupyter_output><empty_output><jupyter_text>Everything below here is when the modelling really starts.First step is the preprocessing, changing comments to as.type(str) for all comments. Because for some reason, one of the comment is just simply a numpy.float object instead of str.Then we handcraft the 23 features, with each feature being the count of the particular word.<jupyter_code>df2 = pd.read_csv('train.csv') #change the filename here
df2[['Comments']] = df2[['Comments']].astype(str) #For some reason some of the comment is of type float, change all comments to string first 

df2['atb'] = df2['Comments'].apply(lambda comment: comment.lower().count('atb'))
df2['bbfa'] = df2['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
df2['amdk'] = df2['Comments'].apply(lambda comment: comment.lower().count('amdk'))
df2['nnp'] = df2['Comments'].apply(lambda comment: comment.lower().count('nnp'))
df2['chiobu'] = df2['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
df2['sinkies'] = df2['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
df2['cunt'] = df2['Comments'].apply(lambda comment: comment.lower().count('cunt'))
df2['cpf'] = df2['Comments'].apply(lambda comment: comment.lower().count('cpf'))
df2['prc'] = df2['Comments'].apply(lambda comment: comment.lower().count('prc'))
df2['pap'] = df2['Comments'].apply(lambda comment: comment.lower().count('pap'))
df2['pappies'] = df2['Comments'].apply(lambda comment: comment.lower().count('pappies'))
df2['ang mo'] = df2['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
df2['dog'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
df2['knn'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
df2['cb'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
df2['fuck'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
df2['fk'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
df2['pussy'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
df2['boobs'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
df2['xmm'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
df2['fark'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
df2['ah neh'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
df2['oppies'] = df2['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))<jupyter_output><empty_output><jupyter_text>Sample of what the training data now looks like.Note: the cell value of 3 in the first comment under the 'knn' feature is because the comment contains 3 counts of the word 'knn'.<jupyter_code>df2.iloc[1570:1575,]<jupyter_output><empty_output><jupyter_text>We then create the Point Biserial Coefficient Matrix and the P Value Matrix again, to get ready to store all the correlation values and p values which we are about to start testing.<jupyter_code>PointBiserialCoefficientMatrix = df2.iloc[0:0,1:5]
PValuesMatrixStored = df2.iloc[0:0, 1:5]
PointBiserialCoefficientMatrix
PValuesMatrixStored<jupyter_output><empty_output><jupyter_text>Before that, just a quick summary of the total counts across the entire training dataset and what exactly are the features that we will be using.<jupyter_code>TotalCount = df2.iloc[:,6:30]
TotalCount = TotalCount.sum(axis = 0, skipna = True) 
print(TotalCount)<jupyter_output>atb        104
bbfa        46
amdk        67
nnp         27
chiobu      47
sinkies     87
cunt        28
cpf         43
prc         21
pap        282
pappies     16
ang mo      32
dog         79
knn        390
cb         165
fuck       187
fk          23
pussy       19
boobs       44
xmm         40
fark        49
ah neh      19
oppies      16
dtype: int64<jupyter_text>The magic begins here. For each label, for each feature, compute the correlation using stats.pointbiserialr().(a,r) is essentially the (output, pvalue) of the computation.We store all the a in the Point Biserial Coefficient Matrix.We store all the r in the P Value Matrix.<jupyter_code>features = ('atb', 'bbfa', 'amdk', 'nnp', 'chiobu', 'sinkies', 'cunt', 'cpf', 'prc', 'pap','pappies','ang mo','dog','knn','cb','fuck','fk','pussy','boobs','xmm','fark','ah neh', 'oppies')
columns = ('Insulting', 'Anti Government', 'Xenophobic', 'Racist', 'Sexual')

####################################################################################
##ANALYZING THE TRAINING SET FOR REAL TO OUTPUT SIGNIFICANT CORRELATIONS MATRICES ##
####################################################################################

for c in columns:
  for f in features: 
    a,r = stats.pointbiserialr(df2[c], df2[f]) # the output of (a,r) here refers to (corr value, p-value)
    if (r < 0.05 and a > 0.01): #if [p-value is significant] && corr is positive cause I only want to care about features that leads indication of the categories
      PointBiserialCoefficientMatrix.loc[f,c] = a #then append the corr value inside the PointBiserialCoefficient matrix
      PValuesMatrixStored.loc[f,c] = r 
    else:
      PointBiserialCoefficientMatrix.loc[f,c] = 0 #if pvalue not significant , the corr value will be 0
      PValuesMatrixStored.loc[f,c] = r<jupyter_output><empty_output><jupyter_text>This is manual proof of the correlation computed. Using the equation ofCorrelation = (MeanOfAllPositiveBinaries - MeanOfAllNegativeBinaries)/StddevOfEntireFeature xsqrt(ProportionOfPositives * ProportionOfNegatives)<jupyter_code>c,d = stats.pointbiserialr(df2['Insulting'], df2['knn'])

M1 = np.average(df2[df2['Insulting'] == 1]['knn'])
M2 = np.average(df2[df2['Insulting'] == 0]['knn'])
diff = M1 - M2
stddev = np.std(df2['knn'])
P = len(df2[df2['Insulting'] == 1])/len(df2['knn'])
Q = len(df2[df2['Insulting'] == 0])/len(df2['knn'])

correlationValueBetweenInsultingAndKnn = (diff/stddev) * np.sqrt(P*Q)
print('Calculated correlation value: ' + str(correlationValueBetweenInsultingAndKnn))
print('Correlation value from stats.pointbiserialr: ' + str(c))
PointBiserialCoefficientMatrix<jupyter_output><empty_output><jupyter_text>Significant p values are then highlighted in blue, for easier visual identification on the significant values.<jupyter_code>def color_significants_blue(val):
    """
    Takes a scalar and returns a string with
    the css property `'color: red'` for significant
    strings, black otherwise.
    """
    color = 'blue' if val < 0.05 else 'black'
    return 'color: %s' % color

  
#apply red color to significant values
PValuesMatrixStored = PValuesMatrixStored.style.applymap(color_significants_blue)
PValuesMatrixStored<jupyter_output><empty_output><jupyter_text>For easier visualizations of the feature against the respective labels, a heat map is plotted. More intense hues represent the feature correlating more strongly to that particular label.<jupyter_code>import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize = (10,10))
sns.heatmap(PointBiserialCoefficientMatrix, cmap="Blues")<jupyter_output><empty_output><jupyter_text>The attempt to evaluate a new comment.New comments go through the same appendment of all 23 features, counting the number of word in the new comment.The output is a 1 x 23 matrix.<jupyter_code>## Input your new string.
new_comment = str('The pap government has been doing nothing but fucking my nnp chiobu balls in the cunt face. Wish i could suck his balls and cb so that i can cb cb cb cb')  

empty_df = pd.DataFrame(columns=['Comments'])
empty_df.at[0,'Comments'] = new_comment
empty_df['atb'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('atb'))
empty_df['bbfa'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
empty_df['amdk'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('amdk'))
empty_df['nnp'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('nnp'))
empty_df['chiobu'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
empty_df['sinkies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
empty_df['cunt'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cunt'))
empty_df['cpf'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cpf'))
empty_df['prc'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('prc'))
empty_df['pap'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pap'))
empty_df['pappies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pappies'))
empty_df['ang mo'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
empty_df['dog'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
empty_df['knn'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
empty_df['cb'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
empty_df['fuck'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
empty_df['fk'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
empty_df['pussy'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
empty_df['boobs'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
empty_df['xmm'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
empty_df['fark'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
empty_df['ah neh'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
empty_df['oppies'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
empty_df<jupyter_output><empty_output><jupyter_text>Dropping the 'Comments' column...<jupyter_code>empty_df = empty_df.drop(['Comments'],axis=1)
empty_df<jupyter_output><empty_output><jupyter_text>To evaluate whether or not to ban the comment, we put it through matrix multiplication.Mutliplying the new comment matrix ( 1 x 23 ) against the Point Biserial Correlation Matrix ( 23 x 5) to output a 'score' matrix of (1 x 5).However, these values are not a scale. Rather, they are a relative score to each other.We can say that for this particular comment, it is more Insulting than it is Xenophobic.But, we cannot deduce that it is definitely Insulting just by looking at this score.<jupyter_code>scores = empty_df.dot(PointBiserialCoefficientMatrix)
scores<jupyter_output><empty_output><jupyter_text>To do out-of-sample accuracy test, writing a new function.<jupyter_code>def Insulting_Evaluation(test_comment, PBCM):
  new_comment = test_comment
  empty_df = pd.DataFrame(columns=['Comments'])
  empty_df.at[0,'Comments'] = new_comment
  empty_df['atb'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('atb'))
  empty_df['bbfa'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
  empty_df['amdk'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('amdk'))
  empty_df['nnp'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('nnp'))
  empty_df['chiobu'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
  empty_df['sinkies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
  empty_df['cunt'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cunt'))
  empty_df['cpf'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cpf'))
  empty_df['prc'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('prc'))
  empty_df['pap'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pap'))
  empty_df['pappies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pappies'))
  empty_df['ang mo'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
  empty_df['dog'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
  empty_df['knn'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
  empty_df['cb'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
  empty_df['fuck'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
  empty_df['fk'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
  empty_df['pussy'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
  empty_df['boobs'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
  empty_df['xmm'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
  empty_df['fark'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
  empty_df['ah neh'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
  empty_df['oppies'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
  empty_df = empty_df.drop(['Comments'],axis=1)
  scores = np.matmul(empty_df, PBCM)
  if scores[0][0] > 0: return 1 
  else: return 0
  
def AG_Evaluation(test_comment, PBCM):
  new_comment = test_comment
  empty_df = pd.DataFrame(columns=['Comments'])
  empty_df.at[0,'Comments'] = new_comment
  empty_df['atb'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('atb'))
  empty_df['bbfa'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
  empty_df['amdk'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('amdk'))
  empty_df['nnp'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('nnp'))
  empty_df['chiobu'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
  empty_df['sinkies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
  empty_df['cunt'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cunt'))
  empty_df['cpf'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cpf'))
  empty_df['prc'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('prc'))
  empty_df['pap'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pap'))
  empty_df['pappies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pappies'))
  empty_df['ang mo'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
  empty_df['dog'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
  empty_df['knn'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
  empty_df['cb'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
  empty_df['fuck'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
  empty_df['fk'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
  empty_df['pussy'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
  empty_df['boobs'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
  empty_df['xmm'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
  empty_df['fark'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
  empty_df['ah neh'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
  empty_df['oppies'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
  empty_df = empty_df.drop(['Comments'],axis=1)
  scores = np.matmul(empty_df, PBCM)
  if scores[0][1] > 0: return 1 
  else: return 0

def Xenophobic_Evaluation(test_comment, PBCM):
  new_comment = test_comment
  empty_df = pd.DataFrame(columns=['Comments'])
  empty_df.at[0,'Comments'] = new_comment
  empty_df['atb'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('atb'))
  empty_df['bbfa'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
  empty_df['amdk'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('amdk'))
  empty_df['nnp'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('nnp'))
  empty_df['chiobu'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
  empty_df['sinkies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
  empty_df['cunt'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cunt'))
  empty_df['cpf'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cpf'))
  empty_df['prc'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('prc'))
  empty_df['pap'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pap'))
  empty_df['pappies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pappies'))
  empty_df['ang mo'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
  empty_df['dog'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
  empty_df['knn'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
  empty_df['cb'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
  empty_df['fuck'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
  empty_df['fk'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
  empty_df['pussy'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
  empty_df['boobs'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
  empty_df['xmm'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
  empty_df['fark'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
  empty_df['ah neh'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
  empty_df['oppies'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
  empty_df = empty_df.drop(['Comments'],axis=1)
  scores = np.matmul(empty_df, PBCM)
  if scores[0][2] > 0: return 1 
  else: return 0
  
def Racist_Evaluation(test_comment, PBCM):
  new_comment = test_comment
  empty_df = pd.DataFrame(columns=['Comments'])
  empty_df.at[0,'Comments'] = new_comment
  empty_df['atb'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('atb'))
  empty_df['bbfa'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
  empty_df['amdk'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('amdk'))
  empty_df['nnp'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('nnp'))
  empty_df['chiobu'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
  empty_df['sinkies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
  empty_df['cunt'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cunt'))
  empty_df['cpf'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cpf'))
  empty_df['prc'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('prc'))
  empty_df['pap'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pap'))
  empty_df['pappies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pappies'))
  empty_df['ang mo'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
  empty_df['dog'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
  empty_df['knn'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
  empty_df['cb'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
  empty_df['fuck'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
  empty_df['fk'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
  empty_df['pussy'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
  empty_df['boobs'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
  empty_df['xmm'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
  empty_df['fark'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
  empty_df['ah neh'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
  empty_df['oppies'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
  empty_df = empty_df.drop(['Comments'],axis=1)
  scores = np.matmul(empty_df, PBCM)
  if scores[0][3] > 0: return 1 
  else: return 0
  
def Sexual_Evaluation(test_comment, PBCM):
  new_comment = test_comment
  empty_df = pd.DataFrame(columns=['Comments'])
  empty_df.at[0,'Comments'] = new_comment
  empty_df['atb'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('atb'))
  empty_df['bbfa'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
  empty_df['amdk'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('amdk'))
  empty_df['nnp'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('nnp'))
  empty_df['chiobu'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
  empty_df['sinkies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
  empty_df['cunt'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cunt'))
  empty_df['cpf'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('cpf'))
  empty_df['prc'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('prc'))
  empty_df['pap'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pap'))
  empty_df['pappies'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('pappies'))
  empty_df['ang mo'] = empty_df['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
  empty_df['dog'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
  empty_df['knn'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
  empty_df['cb'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
  empty_df['fuck'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
  empty_df['fk'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
  empty_df['pussy'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
  empty_df['boobs'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
  empty_df['xmm'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
  empty_df['fark'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
  empty_df['ah neh'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
  empty_df['oppies'] = empty_df['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
  empty_df = empty_df.drop(['Comments'],axis=1)
  scores = np.matmul(empty_df, PBCM)
  if scores[0][4] > 0: return 1 
  else: return 0<jupyter_output><empty_output><jupyter_text>f(x) is the function to test if we have correctly predicted a new comment. If all labels are correctly matched, then the function should return 1, otherwise 0.<jupyter_code>def f(x):
    if Insulting_Evaluation(x['Comments'], PBCM) == x['Insulting'] and AG_Evaluation(x['Comments'], PBCM) == x['Anti_Government']  and Xenophobic_Evaluation(x['Comments'], PBCM) == x['Xenophobic'] and Racist_Evaluation(x['Comments'], PBCM) == x['Racist'] and Sexual_Evaluation(x['Comments'], PBCM) == x['Sexual']: return 1
    else: return 0<jupyter_output><empty_output><jupyter_text>Split the training data set into 10 splits, shuffling the splits each time.printing the length of the train set, and the length of the test set to confirm that they are split correctly.<jupyter_code>df2 = pd.read_csv('train.csv') #change the filename here
df2[['Comments']] = df2[['Comments']].astype(str)
kf = KFold(n_splits=10, random_state=42, shuffle=True) # Define the split - into 10 folds 
kf.get_n_splits(df2) # returns the number of splitting iterations in the cross-validator

for train_index, test_index in kf.split(df2):
  print(len(train_index))
  print(len(test_index))<jupyter_output><empty_output><jupyter_text>CV scores for each iteration will be appended into the CV_scores array.For each train and test split, we go through the entire computaion for each comment.If comment was correctly classified, we will return 1.<jupyter_code>CV_scores = []
i = 0
for train_index, test_index in kf.split(df2):
  train=df2.iloc[train_index]
  test=df2.iloc[test_index]
  PointBiserialCoefficientMatrix = train.iloc[0:0,1:5]
  PValuesMatrixStored = train.iloc[0:0, 1:5]
  train['atb'] = train['Comments'].apply(lambda comment: comment.lower().count('atb'))
  train['bbfa'] = train['Comments'].apply(lambda comment: comment.lower().count('bbfa'))
  train['amdk'] = train['Comments'].apply(lambda comment: comment.lower().count('amdk'))
  train['nnp'] = train['Comments'].apply(lambda comment: comment.lower().count('nnp'))
  train['chiobu'] = train['Comments'].apply(lambda comment: comment.lower().count('chiobu'))
  train['sinkies'] = train['Comments'].apply(lambda comment: comment.lower().count('sinkies'))
  train['cunt'] = train['Comments'].apply(lambda comment: comment.lower().count('cunt'))
  train['cpf'] = train['Comments'].apply(lambda comment: comment.lower().count('cpf'))
  train['prc'] = train['Comments'].apply(lambda comment: comment.lower().count('prc'))
  train['pap'] = train['Comments'].apply(lambda comment: comment.lower().count('pap'))
  train['pappies'] = train['Comments'].apply(lambda comment: comment.lower().count('pappies'))
  train['ang mo'] = train['Comments'].apply(lambda comment: comment.lower().count('ang mo'))
  train['dog'] = train['Comments'].apply(lambda comment: len(re.findall('(?=dog)', comment.lower())))
  train['knn'] = train['Comments'].apply(lambda comment: len(re.findall('(?=knn)', comment.lower())))
  train['cb'] = train['Comments'].apply(lambda comment: len(re.findall('(?=cb)', comment.lower())))
  train['fuck'] = train['Comments'].apply(lambda comment: len(re.findall('(?=fuck)', comment.lower())))
  train['fk'] = train['Comments'].apply(lambda comment: len(re.findall('(?=fk)', comment.lower())))
  train['pussy'] = train['Comments'].apply(lambda comment: len(re.findall('(?=pussy)', comment.lower())))
  train['boobs'] = train['Comments'].apply(lambda comment: len(re.findall('(?=boobs)', comment.lower())))
  train['xmm'] = train['Comments'].apply(lambda comment: len(re.findall('(?=xmm)', comment.lower())))
  train['fark'] = train['Comments'].apply(lambda comment: len(re.findall('(?=fark)', comment.lower())))
  train['ah neh'] = train['Comments'].apply(lambda comment: len(re.findall('(?=ah neh)', comment.lower())))
  train['oppies'] = train['Comments'].apply(lambda comment: len(re.findall('(?=oppies)', comment.lower())))
  features = ('atb', 'bbfa', 'amdk', 'nnp', 'chiobu', 'sinkies', 'cunt', 'cpf', 'prc', 'pap','pappies','ang mo','dog','knn','cb','fuck','fk','pussy','boobs','xmm','fark','ah neh', 'oppies')
  columns = ('Insulting', 'Anti Government', 'Xenophobic', 'Racist', 'Sexual')

  for c in columns:
    for f in features: 
      a,r = stats.pointbiserialr(train[c], train[f])
      if (r < 0.05 and a > 0): 
        PointBiserialCoefficientMatrix.loc[f,c] = a 
      else:
        PointBiserialCoefficientMatrix.loc[f,c] = 0 
  print(PointBiserialCoefficientMatrix)
        
  def f(x):
    if Insulting_Evaluation(x['Comments'], PointBiserialCoefficientMatrix) == x['Insulting'] and AG_Evaluation(x['Comments'], PointBiserialCoefficientMatrix) == x['Anti Government']  and Xenophobic_Evaluation(x['Comments'], PointBiserialCoefficientMatrix) == x['Xenophobic'] and Racist_Evaluation(x['Comments'], PointBiserialCoefficientMatrix) == x['Racist'] and Sexual_Evaluation(x['Comments'], PointBiserialCoefficientMatrix) == x['Sexual']: return 1
    else: return 0
  
  print(i)
  test['correct_evaluation'] = test.apply(f, axis=1)
  CV_scores.append(test.correct_evaluation.sum()/len(train_index))
  i = i+1<jupyter_output><empty_output><jupyter_text>Determine the average CV score based purely on accuracy for this model.<jupyter_code>print(np.mean(CV_scores) * 100)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19139544281761944, 'fraction_numerical': 0.009325748391308402, 'mean_word_length': 7.123737373737374, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 1, 'sexual_word_list.json': 1, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '607692', 'license': 'MIT', 'n_tokens_mistral': 11086, 'n_tokens_neox': 10517, 'n_words': 2176, 'path': '4. Classification Models/Feature Engineering.ipynb', 'repo_name': 'DanLeePy/Singapore-Comment-Filter', 'size': '32173'}