<jupyter_start><jupyter_text>Spam detectionThe main aim of this project is to build a machine learning classifier that is able to automatically detect spammy articles, based on their content.<jupyter_code>! sh bootstrap.sh
from sklearn.cluster import KMeans
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import random
from sklearn.utils import shuffle
from sklearn.metrics import f1_score
from sklearn.cross_validation import KFold
from sklearn.metrics import recall_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import BernoulliNB
#Load train dataset
df = pd.read_csv("enwiki.draft_quality.201608-201701.feature_labels.tsv", sep="\t")
#shuffle the dataframe so that samples will be randomly distributed in Cross Validation Folds
df = shuffle(df)
#Replace strings with integers : 1 for OK and 0 for Not OK
df["draft_quality"] = df["draft_quality"].replace({"OK" : 1, "vandalism" : 0, "spam" : 0, "attack" : 0})
#═df["Markup_chars_thresholded"]=(df['feature.(wikitext.revision.markup_chars / max(wikitext.revision.chars, 1))']>0.35)*1
#Put features and labels on differents dataframes
X=df.drop(["draft_quality"], 1)
Y=df["draft_quality"]

df2 = pd.read_csv("enwiki.draft_quality.50k_stratified.feature_labels.tsv", sep="\t")
df2["draft_quality"] = df2["draft_quality"].replace({"OK" : 1, "vandalism" : 0, "spam" : 0, "attack" : 0})
#df2["Markup_chars_thresholded"]=(df2['feature.(wikitext.revision.markup_chars / max(wikitext.revision.chars, 1))']>0.35)*1
X2=df2.drop(["draft_quality"], 1)
Y2=df2["draft_quality"]
X2=np.array(X2)
Y2=np.array(Y2)
X=np.array(X)
Y=np.array(Y)
#lenghts of boths datasets
print(len(X))
print(len(X2))<jupyter_output>324570
52518<jupyter_text>ModelingWe tried out various models and selected the best performing models (with the best performing parameter settings for each model). At the end, we retained 3 models which are:1.	Naïve Bayes Gaussian2.	Random forest3.	Logistic regression  Naïve Bayes Gaussian<jupyter_code>weights=np.array([0.7,1-0.7])
clf = BernoulliNB(alpha=22, class_prior=weights)
clf.fit(X2, Y2)
prediction_nb=clf.predict(X)
confusion=confusion_matrix(Y, prediction_nb, labels=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[  8723   1630]
 [ 22449 291768]]
Over all the not-OK articles included in the dataset, we detect:
0.84255771274
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.279834466829<jupyter_text>logistic regression<jupyter_code>clf2 = LogisticRegression(penalty='l1', random_state=0, class_weight={1:0.1, 0: 0.9})
clf2.fit(X2, Y2)
prediction_lr=clf2.predict(X)
confusion=confusion_matrix(Y, prediction_lr, labels=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[ 10131    222]
 [103294 210923]]
Over all the not-OK articles included in the dataset, we detect:
0.978556940017
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.0893189332158<jupyter_text>random forest<jupyter_code>clf3 = RandomForestClassifier(n_estimators=2, min_samples_leaf=1, random_state=25, class_weight={1:0.9, 0: 0.1})
clf3.fit(X2, Y2)
prediction_rf=clf3.predict(X)
confusion=confusion_matrix(Y, prediction_rf, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[18530  7731]
 [  816 25441]]
Over all the not-OK articles included in the dataset, we detect:
0.705609078101
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.957820738137<jupyter_text>Combination 1We decided to try combining these models in order to construct a better and more consistent one.  voting system<jupyter_code>#Here we construct our voting function
def voting(pred1, pred2, pred3):
    final_prediction=np.zeros_like(pred1)
    for i in range(len(pred1)):
        if pred1[i]==pred2[i]:
            final_prediction[i]=pred1[i]
        elif pred1[i]==pred3[i]:
            final_prediction[i]=pred1[i]
        elif pred2[i]==pred3[i]:
            final_prediction[i]=pred2[i]
    return final_prediction
#Here we make the prediction using voting function (with the three models defined above)
prediction= voting(prediction_lr, prediction_nb, prediction_rf)
from sklearn.metrics import confusion_matrix
confusion=confusion_matrix(Y, prediction, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[23013  3248]
 [ 1077 25180]]
Over all the not-OK articles included in the dataset, we detect:
0.876318495107
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.955292652553<jupyter_text>customizing<jupyter_code>#Since we are inerested in negatives (not-OK) we will analyze how many times a model detects a not-OK article while
#the others don't
def get_missclasified_indexes(pred1, Y_true, Class):
    index_list=[]
    a=0
    b=1
    if Class=="negative":
        a=1
        b=0  
    for i in range(len(pred1)):
        if pred1[i]==a and Y_true[i]==b:
            index_list.append(i)
    return index_list
false_negative_indexes=get_missclasified_indexes(prediction, Y, "negative")
print(len(prediction[false_negative_indexes]))
print(np.sum(prediction_nb[false_negative_indexes]!=prediction[false_negative_indexes]))
print(np.sum(prediction_rf[false_negative_indexes]!=prediction[false_negative_indexes]))
print(np.sum(prediction_lr[false_negative_indexes]!=prediction[false_negative_indexes]))
##Here we define our function based on the results above
def voting_customized(pred1, pred2, pred3):
    final_prediction=np.zeros_like(pred1)
    for i in range(len(pred1)):
        if pred1[i]==0:
            final_prediction[i]=0
        else:
            final_prediction[i]=pred3[i]
    return final_prediction
#making a prediction with our new function
prediction= voting_customized(prediction_lr, prediction_nb, prediction_rf)
confusion=confusion_matrix(Y, prediction, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)
false_negative_indexes=get_missclasified_indexes(prediction, Y, "negative")
print(len(prediction[false_negative_indexes]))
print(np.sum(prediction_nb[false_negative_indexes]!=prediction[false_negative_indexes]))
print(np.sum(prediction_rf[false_negative_indexes]!=prediction[false_negative_indexes]))
print(np.sum(prediction_lr[false_negative_indexes]!=prediction[false_negative_indexes]))<jupyter_output>2452
1128
0
0<jupyter_text>Here you can see that benefited from the good behavior of the logistic regression and the random forest. By contrast,we couldn't do the same with the naive bayse, because, this makes as missclassify a lot of OK articles, which leads toa low precision.<jupyter_code>from sklearn.metricsicsicsics import roc_curve,auc 
fpr = dict()
tpr = dict()
roc_auc = dict()
fpr, tpr, _ = roc_curve(y_test, y_score),
roc_auc = auc(fpr, tpr)
plt.figure()
lw = 2
plt.plot(fpr, tpr, color='darkorange', lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0,1], [0,1] color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.<jupyter_output><empty_output><jupyter_text>Combination 2Now, we would like the capture more of the not-OK articles. To this end, we decided to include a few false positives in the training datasets. In order so in an intelligent way and to select some representative samples, we first analyzed these false positives.<jupyter_code>from scipy.cluster.hierarchy import dendrogram, linkage
Z = linkage(X[false_negative_indexes], 'ward')
plt.figure(figsize=(25, 25))
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    Z,
    leaf_rotation=90., 
    leaf_font_size=11.,  
)
plt.show()<jupyter_output><empty_output><jupyter_text>This means that we have two big clusters of false positives (green and red). We have chosen to pick up randomly 50 samples of each cluster.<jupyter_code>#we perform a kmeans clustering with 2 clusters
kmeans = KMeans(n_clusters=2, random_state=0).fit(X[false_negative_indexes])
cluster_labels=kmeans.labels_
print(cluster_labels)
print(np.unique(cluster_labels))
#Picking up the sapmles from theclusters and adding them to the training dataset.

false_negatives_cluster0=[]
false_negatives_cluster1=[]

for i in range(1,11):
    random.seed(a=i)
    false_negatives_cluster0.append(random.choice([w for index_w, w in enumerate(false_negative_indexes) if cluster_labels[index_w] == 0]))
for i in range(1,11):
    random.seed(a=i)
    false_negatives_cluster1.append(random.choice([w for index_w, w in enumerate(false_negative_indexes) if cluster_labels[index_w] == 1]))


#adding 1st cluster's samples
Y2=np.reshape(np.dstack(Y2), (len(Y2),1))
temp_arr=np.array([Y[false_negatives_cluster0]])
temp_arr=np.reshape(np.dstack(temp_arr), (10,1))

X2_new = np.vstack((X2, X[false_negatives_cluster0]))
Y2_new=np.vstack((Y2, temp_arr))
# Second
temp_arr2=np.array([Y[false_negatives_cluster1]])
temp_arr2=np.reshape(np.dstack(temp_arr2), (10,1))

X2_new = np.vstack((X2_new, X[false_negatives_cluster1]))
Y2_new=np.vstack((Y2_new, temp_arr2))<jupyter_output><empty_output><jupyter_text>Now we do the prediction again  random forest<jupyter_code>Y2_new=np.reshape(np.dstack(Y2_new), (len(Y2_new),))

clf3.fit(X2_new, Y2_new)
prediction_rf_new=clf3.predict(X)
confusion=confusion_matrix(Y, prediction_rf_new, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[18745  7516]
 [  812 25445]]
Over all the not-OK articles included in the dataset, we detect:
0.713796123529
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.95848033952<jupyter_text>logistic regression<jupyter_code>clf2.fit(X2_new, Y2_new)
prediction_lr_new=clf2.predict(X)
confusion=confusion_matrix(Y, prediction_lr_new, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[22942  3319]
 [ 1097 25160]]
Over all the not-OK articles included in the dataset, we detect:
0.873614866151
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.954365822206<jupyter_text>Naive Bayse<jupyter_code>from sklearn.naive_bayes import BernoulliNB
weights=np.array([0.7,1-0.7])
clf = BernoulliNB(alpha=22, class_prior=weights)
clf.fit(X2_new, Y2_new)
prediction_nb_new=clf.predict(X)
confusion=confusion_matrix(Y, prediction_nb_new, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[23797  2464]
 [ 2685 23572]]
Over all the not-OK articles included in the dataset, we detect:
0.90617265146
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.89861037686<jupyter_text>Voting<jupyter_code>prediction= voting(prediction_lr_new, prediction_nb_new, prediction_rf_new)
confusion=confusion_matrix(Y, prediction, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)<jupyter_output>[[23035  3226]
 [ 1099 25158]]
Over all the not-OK articles included in the dataset, we detect:
0.87715623929
Over all the articles predicted as being not-OK, only this proportion is really not-OK:
0.954462583907<jupyter_text>Customizing<jupyter_code>def voting_customized2(pred1, pred2, pred3):
    final_prediction=np.zeros_like(pred1)
    for i in range(len(pred1)):
        if pred1[i]==0:
            final_prediction[i]=0     
        else:
            final_prediction[i]=pred2[i]
    return final_prediction
prediction= voting_customized2(prediction_lr_new, prediction_nb_new, prediction_rf_new)
confusion=confusion_matrix(Y, prediction, labels=None, sample_weight=None)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)
#Load train dataset
df = pd.read_csv("./enwiki.draft_quality.50k_stratified.feature_labels.tsv", sep="\t")
#shuffle the dataframe so that samples will be randomly distributed in Cross Validation Folds
df = shuffle(df)
#Replace strings with integers : 1 for OK and 0 for Not OK
df["draft_quality"] = df["draft_quality"].replace({"OK" : 1, "vandalism" : 0, "spam" : 0, "attack" : 0})
#═df["Markup_chars_thresholded"]=(df['feature.(wikitext.revision.markup_chars / max(wikitext.revision.chars, 1))']>0.35)*1
#Put features and labels on differents dataframes
X=df.drop(["draft_quality"], 1)
Y=df["draft_quality"]
from sklearn.ensemble import AdaBoostClassifier
clf = AdaBoostClassifier(n_estimators=300)
clf.fit(X, Y)
prediction=clf.predict(X)
confusion=confusion_matrix(Y, prediction)
print(confusion)
recall=confusion[0,0]/(confusion[0,0]+confusion[0,1])
precision=confusion[0,0]/(confusion[0,0]+confusion[1,0])
print("Over all the not-OK articles included in the dataset, we detect:")
print(recall)
print("Over all the articles predicted as being not-OK, only this proportion is really not-OK:")
print(precision)
2948
cpt = 0
for idx, item in enumerate (Y.values):
    if item != prediction[idx] and item == 0: # false positive ie SPAM uncatch 
        cpt +=  1
       #print (X.values[idx]
print (cpt)<jupyter_output>1371
{'fraction_non_alphanumeric': 0.11856542729951722, 'fraction_numerical': 0.04721299141011975, 'mean_word_length': 6.822461991172143, 'pattern_counts': {'":': 1, '<': 46, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '310011', 'license': 'MIT', 'n_tokens_mistral': 5926, 'n_tokens_neox': 5517, 'n_words': 1389, 'path': 'notebooks/spam_detection_wikipedia.ipynb', 'repo_name': 'yafeunteun/wikipedia-spam-classifier', 'size': '15949'}