<jupyter_start><jupyter_text>Download data<jupyter_code>!wget https://ndownloader.figshare.com/files/7394539 -O toxicity_annotations.tsv
!wget https://ndownloader.figshare.com/files/7394542 -O toxicity_annotated_comments.tsv
!wget https://ndownloader.figshare.com/files/7640581 -O toxicity_worker_demographics.tsv
!wget https://ndownloader.figshare.com/files/7554637 -O attack_annotations.tsv
!wget https://ndownloader.figshare.com/files/7554634 -O attack_annotated_comments.tsv
!wget https://ndownloader.figshare.com/files/7640752 -O attack_worker_demographics.tsv
!wget https://ndownloader.figshare.com/files/7394506 -O aggression_annotations.tsv
!wget https://ndownloader.figshare.com/files/7038038 -O aggression_annotated_comments.tsv
!wget https://ndownloader.figshare.com/files/7640644 -O aggression_worker_demographics.tsv<jupyter_output>--2020-10-22 22:30:59--  https://ndownloader.figshare.com/files/7394539
Resolving ndownloader.figshare.com (ndownloader.figshare.com)... 192.168.127.12, 172.16.58.3, 172.16.17.32, ...
Connecting to ndownloader.figshare.com (ndownloader.figshare.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/7394539/toxicity_annotations.tsv [following]
--2020-10-22 22:31:00--  https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/7394539/toxicity_annotations.tsv
Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 172.16.58.3
Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 36177321 (35M) [binary/octet-stream]
Saving to: ‘toxicity_annotations.tsv’

toxicity_annotation 100%[===================>]  34.50M  13.8MB/s    in 2.5s    

2020-10-22 22:31:03 (13.8 MB/s) - ‘toxicity_ann[...]<jupyter_text>1 Step 1: Select and perform analysis  1.1 Load data using pandas<jupyter_code>relative_folder_path = "../data/"

annotated_comments_name = "annotated_comments.tsv"
annotations_name = "annotations.tsv"
worker_demographics_name = "worker_demographics.tsv"

data_type_attack = "attack";
data_type_aggression = "aggression";
data_type_toxicity = "toxicity";

df_annotation_comments_attack = pd.read_csv(str( data_type_attack + "_" + annotated_comments_name), sep='\t', header=0)
df_annotations_attack = pd.read_csv(str(data_type_attack + "_" + annotations_name), sep='\t', header=0)
df_worker_demographics_attack = pd.read_csv(str(data_type_attack + "_" + worker_demographics_name), sep='\t', header=0)

df_annotation_comments_toxicity = pd.read_csv(str(data_type_toxicity + "_" + annotated_comments_name), sep='\t', header=0)
df_annotations_toxicity = pd.read_csv(str(data_type_toxicity + "_" + annotations_name), sep='\t', header=0)
df_worker_demographics_toxicity = pd.read_csv(str(data_type_toxicity + "_" + worker_demographics_name), sep='\t', header=0)

df_annotation_comments_aggression = pd.read_csv(str(data_type_aggression + "_" + annotated_comments_name), sep='\t', header=0)
df_annotations_aggression = pd.read_csv(str(data_type_aggression + "_" + annotations_name), sep='\t', header=0)
df_worker_demographics_aggression = pd.read_csv(str(data_type_aggression + "_" + worker_demographics_name), sep='\t', header=0)

df_attack = df_worker_demographics_attack.join(df_annotations_attack, on="worker_id", rsuffix="_r")
df_toxicity = df_worker_demographics_toxicity.join(df_annotations_toxicity, on="worker_id", rsuffix="_r")
df_aggression = df_worker_demographics_aggression.join(df_annotations_aggression, on="worker_id", rsuffix="_r")
df = df_attack.join(df_toxicity, on="worker_id", rsuffix="_r")
df = df.join(df_aggression, on="worker_id", rsuffix="_r")
df.head()<jupyter_output><empty_output><jupyter_text>1.2 Analysis of demographic information<jupyter_code>### ******** Gender vs attack *******
df_gender = df.groupby("gender").sum()
df_gender = df_gender.sort_values('gender')
df.head()

fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(25,5))
ax[0].pie(list(df_gender['attack']), explode=(0, 0.08, 0), labels=list(df_gender.index), autopct='%1.1f%%',
        shadow=True, startangle=90)
ax[0].axis('equal')
ax[0].set_title('Gender vs attack')

### ******* Education vs attack *******
df_education = df.groupby("education").sum()
df_education = df_education.sort_values('attack', ascending=False)
df.head()

ax[1].bar(df_education.index, df_education['attack'])
ax[1].set_title('Education vs attack')

### ******* Age group vs attack *******
df_age_group = df.groupby("age_group").sum()
df_age_group = df_age_group.sort_values('age_group')
df.head()

ax[2].bar(df_age_group.index, df_age_group['attack'])
ax[2].set_title('Age group vs attack')


### ******** Gender vs toxicity *******
df_gender = df.groupby("gender").sum()
df_gender = df_gender.sort_values('gender')
df.head()

fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(25,5))
ax[0].pie(list(df_gender['toxicity']), explode=(0, 0.08, 0), labels=list(df_gender.index), autopct='%1.1f%%',
        shadow=True, startangle=90)
ax[0].axis('equal')
ax[0].set_title('Gender vs toxicity')

### ******* Education vs toxicity *******
df_education = df.groupby("education").sum()
df_education = df_education.sort_values('toxicity', ascending=False)
df.head()

ax[1].bar(df_education.index, df_education['toxicity'])
ax[1].set_title('Education vs toxicity')

### ******* Age group vs toxicity *******
df_age_group = df.groupby("age_group").sum()
df_age_group = df_age_group.sort_values('age_group')
df.head()

ax[2].bar(df_age_group.index, df_age_group['toxicity'])
ax[2].set_title('Age group vs toxicity')


### ******** Gender vs Aggression *******
df_gender = df.groupby("gender").sum()
df_gender = df_gender.sort_values('gender')
df.head()

fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(25,5))
ax[0].pie(list(df_gender['aggression']), explode=(0, 0.08, 0), labels=list(df_gender.index), autopct='%1.1f%%',
        shadow=True, startangle=90)
ax[0].axis('equal')
ax[0].set_title('Gender vs Aggression')

### ******* Education vs Aggression *******
df_education = df.groupby("education").sum()
df_education = df_education.sort_values('aggression', ascending=False)
df.head()

ax[1].bar(df_education.index, df_education['aggression'])
ax[1].set_title('Education vs Aggression')

### ******* Age group vs Aggression *******
df_age_group = df.groupby("age_group").sum()
df_age_group = df_age_group.sort_values('age_group')
df.head()

ax[2].bar(df_age_group.index, df_age_group['aggression'])
ax[2].set_title('Age group vs Aggression')<jupyter_output><empty_output><jupyter_text>Inference- As it can been seen above, toxicity and attack is almost simliar to each other when compared across different age groups, people with different levels of education and their gender- Majority of the highly attacking/toxic people have bachelors of high school degree. This coincides with the real world data in that people with higher levels of educations are lesser in comparision.- As seen above, the nearly half the amount of workers belong to 18-30 age group- Counts of workers decrease with increase in age indicating that workers are compromised mostly of people below 45 years of age- Since the ratio of male:female is around 3:2, it does not closely represent the real world data in which the ratio is almost 1:1.- Since age groups are categoried in increments of around 12-15, narrowing down the predictions by a model to tighter bounds of different age groups will be challenging task.- On a slightly disimiliar note, aggression is high from people in 30-45 age group in comparison with younger audience.- There is bias is the count of femalte workers in comparision with the count of male workers  1.3 Relationship between worker id and labelling behavior<jupyter_code>df_gender = df.groupby("gender").sum()
df_gender = df_gender.sort_values('attack', ascending=False)
df.head()

fig, ax = plt.subplots(ncols=3, nrows=1, figsize=(25,5))
ax[0].bar(df_gender.index, df_gender['attack'])
ax[0].set_title('Gender vs attack')

df_gender = df.groupby("gender").sum()
df_gender = df_gender.sort_values('aggression', ascending=False)
df.head()

ax[1].bar(df_education.index, df_education['aggression'])
ax[1].set_title('Gender vs aggression')

df_gender = df.groupby("gender").sum()
df_gender = df_gender.sort_values('toxicity', ascending=False)
df.head()

ax[2].bar(df_gender.index, df_gender['toxicity'])
ax[2].set_title('Gender vs toxicity')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1577521206409048, 'fraction_numerical': 0.039938737040527805, 'mean_word_length': 6.682352941176471, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '326135', 'license': 'MIT', 'n_tokens_mistral': 3250, 'n_tokens_neox': 2912, 'n_words': 584, 'path': 'data-512-a2/code/hcds-a2-data-bias.ipynb', 'repo_name': 'Sreejavm/DATA-512-Project-Repo', 'size': '8494'}