<jupyter_start><jupyter_text>Introduction This notebook is an analysis of the Crowdflower labels of 1000 revisions of Wikipedia talk pages by users who have been blocked for personal harassment. This dataset has been cleaned and filtered to remove common administrator messages. These datasets are annotated via crowdflower to measure friendliness, aggressiveness and whether the comment constitutes a personal attack. On Crowdflower, each revision is rated 7 times. The raters are given three questions:1. Is this comment not English or not human readable?    * Column 'na'2. How aggressive or friendly is the tone of this comment?     * Column 'how_aggressive_or_friendly_is_the_tone_of_this_comment'    * Ranges from '---' (Very Aggressive) to '+++' (Very Friendly)3. Does the comment contain a personal attack or harassment? Please mark all that apply:    * Column 'is_harassment_or_attack'    * Users can specify that the attack is:        * Targeted at the recipient of the message (i.e. you suck). ('recipent')        * Targeted at a third party (i.e. Bob sucks). ('third_party')        * Being reported or quoted (i.e. Bob said Henri sucks). ('quoting')        * Another kind of attack or harassment. ('other')        * This is not an attack or harassment. ('not_attack')    Below, we plot histograms of the units by average rating of each of the questions, examine quantiles of answers, and compute inter-annotator agreement.  Loading packages and data<jupyter_code>%matplotlib inline
from __future__ import division
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
pd.set_option('display.max_colwidth', 1000)
# Download data from google drive (Respect Eng / Wiki Collab): wikipdia data/v2_annotated
dat = pd.read_csv('../data/exp2_annotated_1k_no_admin_blocked_user_post_sample.csv')
# Remove test questions
dat = dat[dat['_golden'] == False]
# Replace missing data with 'False'
dat = dat.replace(np.nan, False, regex=True)
#Translate the values from 
#how_aggressive_or_friendly_is_the_tone_of_this_comment into numbers
def translate_how_aggressive_to_num(score):
    str_to_num = {'+++ Very Friendly': 3,
                  '++': 2,
                  '+': 1,
                  'Neutral': 0,
                  '-': -1,
                  '--': -2,
                  '--- Very Aggressive (including passive aggression)': -3}
    if score:
        return str_to_num[score]
    else:
        return np.nan
# We reshape the answers for future analysis
dat['aggression_score'] = dat['how_aggressive_or_friendly_is_the_tone_of_this_comment'].apply(translate_how_aggressive_to_num)
def create_column_of_counts(df, col):
    return df.apply(lambda x: col in str(x))
attack_columns = ['not_attack', 'other', 'quoting', 'recipient', 'third_party']
for col in attack_columns:
    dat[col] = create_column_of_counts(dat['is_harassment_or_attack'], col)
def create_column_of_counts_from_nums(df, col):
    return df.apply(lambda x: int(col) == x)
aggressive_columns = ['-3', '-2', '-1', '0', '1', '2', '3']
for col in aggressive_columns:
    dat[col] = create_column_of_counts_from_nums(dat['aggression_score'], col)
dat['not_attack_0'] = 1 - dat['not_attack']
dat['not_attack_1'] = dat['not_attack']
# Group the data
agg_dict = dict.fromkeys(attack_columns, 'mean')
agg_dict.update(dict.fromkeys(aggressive_columns, 'sum'))
agg_dict.update({'clean_diff': 'first', 'na': 'mean', 'aggression_score': 'mean', 
                 '_id':'count', 'not_attack_0':'sum', 'not_attack_1': 'sum'})
grouped_dat = dat.groupby(['rev_id'], as_index=False).agg(agg_dict)
# Get rid of data which the majority thinks is not in English or not readable
grouped_dat = grouped_dat[grouped_dat['na'] < 0.5]<jupyter_output><empty_output><jupyter_text>Plot histogram of average ratings by revision For each revision, we take the average of all the ratings by level of friendliness/aggressiveness and for each of the answers to Question 3. The histograms of these averages is displayed below.<jupyter_code>def hist_comments(df, bins, plot_by, title):
    plt.figure()
    sliced_array = df[[plot_by]]
    weights = np.ones_like(sliced_array)/len(sliced_array)
    sliced_array.plot.hist(bins = bins, legend = False, title = title, weights=weights)
    plt.ylabel('Proportion')
    plt.xlabel('Average Score')
bins = np.linspace(-3,3,11)
hist_comments(grouped_dat, bins, 'aggression_score', 'Average Aggressiveness Rating for Blocked Data')
bins = np.linspace(0,1,9)
for col in attack_columns:
    hist_comments(grouped_dat, bins, col, 'Average %s Rating' % col)<jupyter_output><empty_output><jupyter_text>Selected harassing and aggressive revisions by quartile We look at a sample of revisions whose average aggressive score falls into various quantiles. This allows us to subjectively evaluate the quality of the questions that we are asking on Crowdflower.<jupyter_code>def sorted_comments(df, sort_by, quartile, num, is_ascending = True):
    n = df.shape[0]
    start_index = int(quartile*n)
    return df[['clean_diff', 'aggression_score',
              'not_attack', 'other', 'quoting', 'recipient', 'third_party']].sort_values(
        by=sort_by, ascending = is_ascending)[start_index:start_index + num]
# Most aggressive comments
sorted_comments(grouped_dat, 'aggression_score', 0, 5)
# Median aggressive comments
sorted_comments(grouped_dat, 'aggression_score', 0.5, 5)
# Least aggressive comments
sorted_comments(grouped_dat, 'aggression_score', 0, 5, False)<jupyter_output><empty_output><jupyter_text>Selected revisions on multiple questions In this section, we examine a selection of revisions by their answer to Question 3 and sorted by aggression score. Again, this allows us to subjectively evaluate the quality of questions and responses that we obtain from Crowdflower.<jupyter_code># Most aggressive comments which are labelled 'This is not an attack or harassment.'
sorted_comments(grouped_dat[grouped_dat['not_attack'] > 0.5], 'aggression_score', 0, 5)
# Most aggressive comments which are labelled 'Being reported or quoted (i.e. Bob said Henri sucks).'
sorted_comments(grouped_dat[grouped_dat['quoting'] > 0.3], 'aggression_score', 0, 5)
# Most aggressive comments which are labelled 'Another kind of attack or harassment.'
sorted_comments(grouped_dat[grouped_dat['other'] > 0.5], 'aggression_score', 0, 5)
# Most aggressive comments which are labelled 'Targeted at a third party (i.e. Bob sucks).'
sorted_comments(grouped_dat[grouped_dat['third_party'] > 0.5], 'aggression_score', 0, 5)
# Least aggressive comments which are NOT labelled 'This is not an attack or harassment.'
sorted_comments(grouped_dat[grouped_dat['not_attack'] < 0.5], 'aggression_score', 0, 5, False)<jupyter_output><empty_output><jupyter_text>Inter-Annotator Agreement Below, we compute the Krippendorf's Alpha, which is a measure of the inter-annotator agreement of our Crowdflower responses. We achieve an Alpha value of 0.668 on our dataset, which is a relatively good level of inter-annotator agreement for this type of subjective inquiry.<jupyter_code>def add_row_to_coincidence(o, row, columns):
    m_u = row.sum(1)
    for i in columns:
        for j in columns:
            if i == j:
                o[i][j] = o[i][j] + row[i]*(row[i]-1)/(m_u-1)
            else:
                o[i][j] = o[i][j] + row[i]*row[j]/(m_u-1)
    return o
def make_coincidence_matrix(df, columns):
    df = df[columns]
    n = df.shape[0]
    num_cols = len(columns)
    o = pd.DataFrame(np.zeros((num_cols,num_cols)), index = columns, columns=columns)
    for i in xrange(n):
        o = add_row_to_coincidence(o, df[i:i+1], columns)
    return o
def binary_distance(i,j):
    return i!=j

def interval_distance(i,j):
    return (int(i)-int(j))**2
def e(n, i, j):
    if i == j:
        return n[i]*(n[i]-1)/sum(n)-1
    else:
        return n[i]*n[j]/sum(n)-1

def D_e(o, columns, distance):
    n = o.sum(1)
    output = 0
    for i in columns:
        for j in columns:
            output = output + e(n,i,j)*distance(i,j)
    return output

def D_o(o, columns, distance):
    output = 0
    for i in columns:
        for j in columns:
            output = output + o[i][j]*distance(i,j)
    return output
def Krippendorf_alpha(df, columns, distance = binary_distance, o = None):
    if o is None:
        o = make_coincidence_matrix(df, columns)
    d_o = D_o(o, columns, distance)
    d_e = D_e(o, columns, distance)
    return (1 - d_o/d_e)
o = make_coincidence_matrix(grouped_dat, aggressive_columns)
print "Krippendorf's Alpha: "
Krippendorf_alpha(grouped_dat, aggressive_columns, distance = interval_distance, o = o)
Krippendorf_alpha(grouped_dat, ['not_attack_0', 'not_attack_1'])
grouped_dat.columns
grouped_dat['not_attack']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10842960082351595, 'fraction_numerical': 0.010637081093446186, 'mean_word_length': 4.444582814445828, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '79259', 'license': 'Apache-2.0', 'n_tokens_mistral': 2928, 'n_tokens_neox': 2739, 'n_words': 866, 'path': 'src/data_generation/crowdflower_analysis/src/Crowdflower Analysis (Experiment v. 2).ipynb', 'repo_name': 'Nintendofan885/Detect_Cyberbullying_from_socialmedia', 'size': '8743'}