<jupyter_start><jupyter_text>Introduction This notebook is an analysis of the Crowdflower labels of 1,000 revisions of Wikipedia talk pages by users who have been blocked for personal harassment. These revisions are chosen from neighbourhoods of various distance from a block event. This dataset has been cleaned and filtered to remove common administrator messages. These datasets are annotated via crowdflower to measure friendliness, aggressiveness and whether the comment constitutes a personal attack.On Crowdflower, each revision is rated 7 times. The raters are given three questions:1. Is this comment not English or not human readable?    * Column 'na'2. How aggressive or friendly is the tone of this comment?     * Column 'how_aggressive_or_friendly_is_the_tone_of_this_comment'    * Ranges from '---' (Very Aggressive) to '+++' (Very Friendly)3. Does the comment contain a personal attack or harassment? Please mark all that apply:    * Column 'is_harassment_or_attack'    * Users can specify that the attack is:        * Targeted at the recipient of the message (i.e. you suck). ('recipent')        * Targeted at a third party (i.e. Bob sucks). ('third_party')        * Being reported or quoted (i.e. Bob said Henri sucks). ('quoting')        * Another kind of attack or harassment. ('other')        * This is not an attack or harassment. ('not_attack')    Below, we plot histograms of the units by average rating of each of the questions, examine quantiles of answers, and compute inter-annotator agreement. We also study whether or not there is a change in aggressiveness before and after a block event.  Loading packages and data<jupyter_code>%matplotlib inline
from __future__ import division
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from crowdflower_analysis import *
from krippendorf_alpha import *
pd.set_option('display.max_colwidth', 1000)
attack_columns = ['not_attack', 'other', 'quoting', 'recipient', 'third_party']
aggressive_columns = ['-3', '-2', '-1', '0', '1', '2', '3']
grouped_dat = {
    '5': preprocess('annotated_onion_layer_5_rows_0_to_10000_first_1000.csv'),
    '10': preprocess('annotated_onion_layer_10_rows_0_to_1000.csv'),
    '20': preprocess('annotated_onion_layer_20_rows_0_to_1000.csv'),
    '30': preprocess('annotated_onion_layer_30_rows_0_to_1000.csv')
}<jupyter_output><empty_output><jupyter_text>Plot histogram of average ratings by revision<jupyter_code>bins = np.linspace(-3,3,11)
for key in grouped_dat.keys():
    hist_comments(grouped_dat[key], bins, 'aggression_score', 'Average Aggressiveness Rating for onion_%s_blocked Data' % key)
bins = np.linspace(0,1,9)
col = 'not_attack'
for key in grouped_dat.keys():
    hist_comments(grouped_dat[key], bins, col, 'Average %s Rating for onion_%s_blocked Data' % (col, key))<jupyter_output><empty_output><jupyter_text>Inter-Annotator Agreement<jupyter_code>from krippendorf_alpha import *
for key in grouped_dat.keys():
    print "Krippendorf's Alpha (aggressiveness) for layer %s: " % key
    print Krippendorf_alpha(grouped_dat[key], aggressive_columns, distance = interval_distance)
    print "Krippendorf's Alpha (attack) for layer %s: " % key
    print Krippendorf_alpha(grouped_dat[key], ['not_attack_0', 'not_attack_1'])<jupyter_output>Krippendorf's Alpha (aggressiveness) for layer 10: 
0.406532039469
Krippendorf's Alpha (attack) for layer 10: 
0.341846868515
Krippendorf's Alpha (aggressiveness) for layer 30: 
0.25127377955
Krippendorf's Alpha (attack) for layer 30: 
0.20087127639
Krippendorf's Alpha (aggressiveness) for layer 5: 
0.581927357756
Krippendorf's Alpha (attack) for layer 5: 
0.484279405223
Krippendorf's Alpha (aggressiveness) for layer 20: 
0.311832403035
Krippendorf's Alpha (attack) for layer 20: 
0.239889896222<jupyter_text>Selected harassing and aggressive revisions by quartile We look at a sample of revisions whose average aggressive score falls into various quantiles. This allows us to subjectively evaluate the quality of the questions that we are asking on Crowdflower.<jupyter_code>key = '20'
# Most aggressive comments
sorted_comments(grouped_dat[key], 'aggression_score', 0, 5)
# Median aggressive comments
sorted_comments(grouped_dat[key], 'aggression_score', 0.5, 5)
# Least aggressive comments
sorted_comments(grouped_dat[key], 'aggression_score', 0, 5, False)<jupyter_output><empty_output><jupyter_text>Selected revisions on multiple questions In this section, we examine a selection of revisions by their answer to Question 3 and sorted by aggression score. Again, this allows us to subjectively evaluate the quality of questions and responses that we obtain from Crowdflower.<jupyter_code># Most aggressive comments which are labelled 'This is not an attack or harassment.'
sorted_comments(grouped_dat[key][grouped_dat[key]['not_attack'] > 0.6], 'aggression_score', 0, 5)
# Most aggressive comments which are labelled 'Being reported or quoted (i.e. Bob said Henri sucks).'
sorted_comments(grouped_dat[key][grouped_dat[key]['quoting'] > 0.3], 'aggression_score', 0, 5)
# Most aggressive comments which are labelled 'Targeted at a third party (i.e. Bob sucks).'
sorted_comments(grouped_dat[key][grouped_dat[key]['third_party'] > 0.5], 'aggression_score', 0, 5)
# Least aggressive comments which are NOT labelled 'This is not an attack or harassment.'
sorted_comments(grouped_dat[key][grouped_dat[key]['not_attack'] < 0.5], 'aggression_score', 0, 5, False)<jupyter_output><empty_output><jupyter_text>T-Test of Aggressiveness We explore whether aggressiveness changes in the tone of comments from immediately before a block event to immediately after.<jupyter_code>plot_and_test_aggressiveness(grouped_dat['10'])<jupyter_output>The mean aggressiveness before a block event:
aggression_score   -0.143763
dtype: float64
The mean aggressiveness after a block event: 
aggression_score    0.0589
dtype: float64
The results of a t-test
Ttest_indResult(statistic=array([-3.94795494]), pvalue=array([  8.78357573e-05]))
{'fraction_non_alphanumeric': 0.0895123580494322, 'fraction_numerical': 0.04124916499665999, 'mean_word_length': 5.980186480186481, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '269710', 'license': 'Apache-2.0', 'n_tokens_mistral': 2052, 'n_tokens_neox': 1804, 'n_words': 631, 'path': 'src/data_generation/crowdflower_analysis/src/Crowdflower Analysis (Experiment on Comparison of Onion Layers).ipynb', 'repo_name': 'Nintendofan885/Detect_Cyberbullying_from_socialmedia', 'size': '5988'}