<jupyter_start><jupyter_text>Prerequisites<jupyter_code>import pandas as pd
import numpy as np
from collections import Counter<jupyter_output><empty_output><jupyter_text>Loading Data Into PandasI use pandas to read the 'tab separated' data into a 'data frame' object for ease of interaction.Loading aggression annotation data:<jupyter_code>aggression_labels = pd.DataFrame()
aggression_labels = pd.read_csv('aggression_annotations.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>Loading aggression comment data:<jupyter_code>aggression_comments = pd.DataFrame()
aggression_comments = pd.read_csv('aggression_annotated_comments.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>Loading aggression worker data:<jupyter_code>aggression_workers = pd.DataFrame()
aggression_workers = pd.read_csv('aggression_worker_demographics.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>Loading toxicity annotation data:<jupyter_code>toxicity_labels = pd.DataFrame()
toxicity_labels = pd.read_csv('toxicity_annotations.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>Loading toxicity comment data:<jupyter_code>toxicity_comments = pd.DataFrame()
toxicity_comments = pd.read_csv('toxicity_annotated_comments.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>Loading toxicity worker data:<jupyter_code>toxicity_workers = pd.DataFrame()
toxicity_workers = pd.read_csv('toxicity_worker_demographics.tsv', sep='\t')<jupyter_output><empty_output><jupyter_text>AnalysisFor both data sets loaded above, I investigate the question of **whether males and females find different words toxic or aggressive**. This would provide some insight into just how much perception of the same content can differ, and therefore what kinds of words (if any) bring some bias into the labelling. If there are particular words for which the labels differ significantly by some demographic breakdown, then any model trained on the data would probably perform poorly when dealing with such words and could apply bias.The intuition motivating this analysis is the thought that there might be certain words or phrases that are more personally applicable to different demographics, so the perception of the negativity of these words will be biased.*First, I must select the annotations that indicate toxicity or aggression.*<jupyter_code>toxicity_labels_bad = toxicity_labels.loc[toxicity_labels['toxicity'] == 1]

aggression_labels_bad = aggression_labels.loc[aggression_labels['aggression'] == 1]<jupyter_output><empty_output><jupyter_text>*Next, I isolate the demographic data I will be focusing on: gender.*<jupyter_code>tox_worker_gender = toxicity_workers[['worker_id', 'gender']]

agg_worker_gender = aggression_workers[['worker_id', 'gender']]<jupyter_output><empty_output><jupyter_text>*Now I combine the gender data with the annotation data.*<jupyter_code>tox_labels_gender = toxicity_labels_bad.merge(tox_worker_gender, on='worker_id')

agg_labels_gender = aggression_labels_bad.merge(agg_worker_gender, on='worker_id')<jupyter_output><empty_output><jupyter_text>*I also isolate the comment data and combine it with the above.*<jupyter_code>tox_comments = toxicity_comments[['rev_id', 'comment']]

agg_comments = aggression_comments[['rev_id', 'comment']]
tox = tox_labels_gender.merge(tox_comments, on='rev_id', how='inner')

agg = agg_labels_gender.merge(agg_comments, on='rev_id', how='inner')<jupyter_output><empty_output><jupyter_text>*I create lists for all words included in toxic or aggressive comments as annotated by females. I turn the strings to lower case before splitting them into lists of words to avoid some amount of word duplicates.**Female words for toxicity:*<jupyter_code>fem_tox_words = []
fem_tox = tox.loc[tox['gender'] == 'female']
fem_tox.reset_index(drop=True, inplace=True)

for i in range(len(fem_tox)):
    fem_tox_words.extend(fem_tox['comment'][i].lower().split())<jupyter_output><empty_output><jupyter_text>*Female words for aggression:*<jupyter_code>fem_agg_words = []
fem_agg = agg.loc[agg['gender'] == 'female']
fem_agg.reset_index(drop=True, inplace=True)

for i in range(len(fem_agg)):
    fem_agg_words.extend(fem_agg['comment'][i].lower().split())<jupyter_output><empty_output><jupyter_text>*Male words for toxicity:*<jupyter_code>male_tox_words = []
male_tox = tox.loc[tox['gender'] == 'male']
male_tox.reset_index(drop=True, inplace=True)

for i in range(len(male_tox)):
    male_tox_words.extend(male_tox['comment'][i].lower().split())<jupyter_output><empty_output><jupyter_text>*Male words for aggression:*<jupyter_code>male_agg_words = []
male_agg = agg.loc[agg['gender'] == 'male']
male_agg.reset_index(drop=True, inplace=True)

for i in range(len(male_agg)):
    male_agg_words.extend(male_agg['comment'][i].lower().split())<jupyter_output><empty_output><jupyter_text>*For reference, I calculate the proportion of male and female entries*<jupyter_code>print("Male Percentage: ", (len(male_tox) / len(tox)) * 100)
print("Female Percentage: ", (len(fem_tox) / len(tox)) * 100)<jupyter_output>Male Percentage:  63.36965547455667
Female Percentage:  36.60028833565123<jupyter_text>*Using the 'collections' package, I generate a dictionary with counts for each word. I then turn this back into a sorted dataframe to easily examine most prominent words.* *Female word count for toxicity:*<jupyter_code>fem_tox_counts = Counter(fem_tox_words)
fem_tox_counts_df = pd.DataFrame.from_dict(fem_tox_counts, orient='index')
fem_tox_counts_df.reset_index(inplace=True)
fem_tox_counts_df.columns = ['word', 'frequency']
fem_tox_counts_df = fem_tox_counts_df.sort_values('frequency', ignore_index=True)<jupyter_output><empty_output><jupyter_text>*Female word count for aggression:*<jupyter_code>fem_agg_counts = Counter(fem_agg_words)
fem_agg_counts_df = pd.DataFrame.from_dict(fem_agg_counts, orient='index')
fem_agg_counts_df.reset_index(inplace=True)
fem_agg_counts_df.columns = ['word', 'frequency']
fem_agg_counts_df = fem_agg_counts_df.sort_values('frequency', ignore_index=True)<jupyter_output><empty_output><jupyter_text>*Male word count for toxicity:*<jupyter_code>male_tox_counts = Counter(male_tox_words)
male_tox_counts_df = pd.DataFrame.from_dict(male_tox_counts, orient='index')
male_tox_counts_df.reset_index(inplace=True)
male_tox_counts_df.columns = ['word', 'frequency']
male_tox_counts_df = male_tox_counts_df.sort_values('frequency', ignore_index=True)<jupyter_output><empty_output><jupyter_text>*Male word count for aggression:*<jupyter_code>male_agg_counts = Counter(male_agg_words)
male_agg_counts_df = pd.DataFrame.from_dict(male_agg_counts, orient='index')
male_agg_counts_df.reset_index(inplace=True)
male_agg_counts_df.columns = ['word', 'frequency']
male_agg_counts_df = male_agg_counts_df.sort_values('frequency', ignore_index=True)<jupyter_output><empty_output><jupyter_text>Now we can observe 40 of the top words featured in toxic comments as labeled by males and females. I skip the top 20 words because in both lists these are just basic parts of speech ('you', 'of', 'is', etc.)<jupyter_code>fem_tox_counts_df.iloc[len(fem_tox_counts_df) - 60: len(fem_tox_counts_df)-20]
male_tox_counts_df.iloc[len(male_tox_counts_df) - 60: len(male_tox_counts_df)-20]<jupyter_output><empty_output><jupyter_text>Similarly we can observe some of the top words featured in aggressive comments as labeled by males and females.<jupyter_code>fem_agg_counts_df.iloc[len(fem_agg_counts_df) - 80: len(fem_agg_counts_df)-20]
male_agg_counts_df.iloc[len(male_agg_counts_df) - 80: len(male_agg_counts_df)-20]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10960347780266104, 'fraction_numerical': 0.007904096956922672, 'mean_word_length': 9.471724137931034, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 88, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '557118', 'license': 'MIT', 'n_tokens_mistral': 2741, 'n_tokens_neox': 2539, 'n_words': 530, 'path': 'data-512-a2/hcds-a2-data-bias.ipynb', 'repo_name': 'Oponn-1/humancentered_hw2', 'size': '7591'}