<jupyter_start><jupyter_text>Wikipedia Talk Data - Getting Started This notebook gives an introduction to working with the various data sets in [WikipediaTalk](https://figshare.com/projects/Wikipedia_Talk/16731) project on Figshare. The release includes:1. a large historical corpus of discussion comments on Wikipedia talk pages2. a sample of over 100k comments with human labels for whether the comment contains a personal attack3. a sample of over 100k comments with human labels for whether the comment has aggressive tonePlease refer to our [wiki](https://meta.wikimedia.org/wiki/Research:Detox/Data_Release) for documentation of the schema of each data set and our [research paper](https://arxiv.org/abs/1610.08914) for documentation on the data collection and modeling methodology. In this notebook we show how to build a simple classifier for detecting personal attacks and apply the classifier to a random sample of the comment corpus to see whether discussions on user pages have more personal attacks than discussion on article pages.  Building a classifier for personal attacksIn this section we will train a simple bag-of-words classifier for personal attacks using the [Wikipedia Talk Labels: Personal Attacks]() data set.<jupyter_code>import pandas as pd
import urllib
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
# download annotated comments and annotations
'''
ANNOTATED_COMMENTS_URL = 'https://ndownloader.figshare.com/files/7554634' 
ANNOTATIONS_URL = 'https://ndownloader.figshare.com/files/7554637' 


def download_file(url, fname):
    urllib.request.urlretrieve(url, fname)

                
download_file(ANNOTATED_COMMENTS_URL, 'attack_annotated_comments.tsv')
download_file(ANNOTATIONS_URL, 'attack_annotations.tsv')
'''
comments = pd.read_csv('../../datasets/attack_annotated_comments.tsv', sep = '\t', index_col = 0)
annotations = pd.read_csv('../../datasets/attack_annotations.tsv',  sep = '\t')
len(annotations['rev_id'].unique())
# labels a comment as an atack if the majority of annoatators did so
labels = annotations.groupby('rev_id')['attack'].mean() > 0.5
# join labels and comments
comments['attack'] = labels
# remove newline and tab tokens
comments['comment'] = comments['comment'].apply(lambda x: x.replace("NEWLINE_TOKEN", " "))
comments['comment'] = comments['comment'].apply(lambda x: x.replace("TAB_TOKEN", " "))
comments.query('attack')['comment'].head()
comments.columns
import numpy as np
np.unique(test_comments['attack'])
# fit a simple text classifier

train_comments = comments.sample(round(len(comments)*0.7))#comments.query("split=='train'")
test_comments = comments[~comments['comment'].isin(train_comments.comment.values)]#comments.query("split=='test'")

clf = Pipeline([
    ('vect', CountVectorizer(max_features = 50000)),#, ngram_range = (1,2))),
    #('tfidf', TfidfTransformer(norm = 'l2')),
    ('clf', LogisticRegression()),
])
clf = clf.fit(train_comments['comment'], train_comments['attack'])
auc = roc_auc_score(test_comments['attack'], clf.predict(test_comments['comment'])) #clf.predict_proba(test_comments['comment'])[:, 1])
print('Test ROC AUC: %.3f' %auc)
auc = roc_auc_score(test_comments['attack'], clf.predict_proba(test_comments['comment'])[:, 1])
print('Test ROC AUC: %.3f' %auc)
roc_auc_score([True,False,True],[0.008,0.002,0.7])
roc_auc_score([True,False,True],[1,0,1])
# correctly classify nice comment
clf.predict(['Thanks for you contribution, you did a great job!'])
# correctly classify nasty comment
clf.predict(['People as stupid as you should not edit Wikipedia!'])<jupyter_output><empty_output><jupyter_text>Prevalence of personal attacks by namespaceIn this section we use our classifier in conjunction with the [Wikipedia Talk Corpus](https://figshare.com/articles/Wikipedia_Talk_Corpus/4264973) to see if personal attacks are more common on user talk or article talk page discussions. In our paper we show that the model is not biased by namespace.<jupyter_code>import os
import re
from scipy.stats import bernoulli

import seaborn as sns
import matplotlib.pyplot as plt

def download_file(url, fname):
    urllib.request.urlretrieve(url, fname)
# download and untar data

USER_TALK_CORPUS_2004_URL = 'https://ndownloader.figshare.com/files/6982061'
ARTICLE_TALK_CORPUS_2004_URL = 'https://ndownloader.figshare.com/files/7038050'

download_file(USER_TALK_CORPUS_2004_URL, 'comments_user_2004.tar.gz')
download_file(ARTICLE_TALK_CORPUS_2004_URL,  'comments_article_2004.tar.gz')

os.system('tar -xzf comments_user_2004.tar.gz')
os.system('tar -xzf comments_article_2004.tar.gz')
# helper for collecting a sample of comments for a given ns and year from 
def load_no_bot_no_admin(ns, year, prob = 0.1):
    
    dfs = []
    
    data_dir = "comments_%s_%d" % (ns, year)
    for _, _, filenames in os.walk(data_dir):
        for filename in filenames:
            if re.match("chunk_\d*.tsv", filename):
                df = pd.read_csv(os.path.join(data_dir, filename), sep = "\t")
                df['include'] = bernoulli.rvs(prob, size=df.shape[0])
                df = df.query("bot == 0 and admin == 0 and include == 1")
                dfs.append(df)
                
    sample = pd.concat(dfs)
    sample['ns'] = ns
    sample['year'] = year
    
    return sample
# collect a random sample of comments from 2004 for each namespace
corpus_user = load_no_bot_no_admin('user', 2004)
corpus_article = load_no_bot_no_admin('article', 2004)
corpus = pd.concat([corpus_user, corpus_article])
# Apply model
corpus['comment'] = corpus['comment'].apply(lambda x: x.replace("NEWLINE_TOKEN", " "))
corpus['comment'] = corpus['comment'].apply(lambda x: x.replace("TAB_TOKEN", " "))
corpus['attack'] = clf.predict_proba(corpus['comment'])[:,1] > 0.425 # see paper
# plot prevalence per ns

sns.pointplot(data = corpus, x = 'ns', y = 'attack')
plt.ylabel("Attack fraction")
plt.xlabel("Dicussion namespace")<jupyter_output><empty_output><jupyter_text>Attacks are far more prevalent in the user talk namespace.<jupyter_code>arr = [[0,1],[1,0],[1,0]]
r = [False if i[0]==1 else True for i in arr]
r<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12155745489078823, 'fraction_numerical': 0.02389996834441279, 'mean_word_length': 5.4087221095334685, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 15, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '356480', 'license': 'Apache-2.0', 'n_tokens_mistral': 2116, 'n_tokens_neox': 1944, 'n_words': 548, 'path': 'src/figshare/Wikipedia Talk Data - Getting Started.ipynb', 'repo_name': 'Nintendofan885/Detect_Cyberbullying_from_socialmedia', 'size': '6318'}