<jupyter_start><jupyter_text>Detecting Cyberbullying in Tweets: Data Wrangling***This Notebook pertains to wrangling data for the project Detecting Cyberbullying in Tweets. The data is collected from two sources:1. Hate Speech Twitter Annotations    - **Publication:**  <NAME> and <NAME>. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In NAACL SRW, pages 88–93, 2016     - **Authors:** Waseem, Zeerak  and  <NAME>    - **GitHub Link:** https://github.com/zeerakw/hatespeech    - **Description:** The dataset contains about 17,000 Tweet ID's labeled for _racism_ and _sexism_. We downloaded this dataset and querried a Twitter API to scrape the actual Tweets from Twitter. Retrieval of about Tweets 5,900 failed either because the Tweet was deleted or the account was deactivated.     2. Hate Speech and Offensive Language Detection    - **Publication:**  Automated Hate Speech Detection and the Problem of Offensive Language    - **Authors:** <NAME> <NAME> <NAME>    - **GitHub Link:** https://github.com/t-davidson/hate-speech-and-offensive-language    - **DescriptionL** The dataset has about 25,000 Tweets annotated by croud sourcing. As per the number of users labeling the Tweets, each is put in one of three classes - _hate speech_, _offensive language_ and _neither_. We downloaded the dataset in Python from GitHub as a csv file.<jupyter_code># dependencies
import pandas as pd
import tweepy
import json
import requests
import os
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import zipfile<jupyter_output><empty_output><jupyter_text>1. Hate Speech Twitter Annotations***<jupyter_code># url for labeled tweet ids
url = 'https://github.com/ZeerakW/hatespeech/archive/master.zip'

# use requests to establish connection
response = requests.get(url)

# create folder 'data'
folder_name = 'data'
if not os.path.exists(folder_name):
    os.makedirs(folder_name)

# download zip file
with open(os.path.join(folder_name, 'master'), mode = 'wb') as file:
    file.write(response.content)

# extract zipfile
with zipfile.ZipFile('data/master.zip', 'r') as zipf:
    zipf.extractall(os.path.join('data'))
# read in the csv file of labeled tweet ids
labeled_ids = pd.read_csv('data/hatespeech-master/NAACL_SRW_2016.csv', names = ['id', 'label'])

labeled_ids.head(2)
# Insert secret tokens and keys from Twitter Developer account
consumer_key = '###'
consumer_secret = '###'
access_token = '###'
access_secret = '###'

# authenticate as per tweepy docs
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)

# create api object
api = tweepy.API(auth, wait_on_rate_limit = True, wait_on_rate_limit_notify = True)
# init counter to keep track of tweets collected and of the failed ids
i = 0
j = 0
failed_ids = []

# open file to write json objects from api
with open('data/tweets.txt', 'w') as outfile:
    for _ in labeled_ids.id:
        
        # try-except block since few tweet IDs in the archive may have been deleted
        try: 
            tweet = api.get_status(_, tweet_mode = 'extended')
        except:
            failed_ids.append(_)
            j = j+1
            print(f'Failed: {_} | {j} of {len(labeled_ids.id)}')
            continue
            
        
        # print the number of tweets collected
        print(f'Success: {_} | {i} of {len(labeled_ids.id)}')
        i = i+1
        
        # dump the json object corresponding to the tweet collected from the api
        json.dump(tweet._json, outfile)
        outfile.write('\n')
print(f'Number of Successful Tweets Querried: {i}')
print(f'Number of Failed Queries: {j}')
#load the json data and store it in a list
data = []
with open('data/tweets.txt') as f:    
        for line in f:         
            data.append(json.loads(line))

df_api = pd.DataFrame(data)

#select columns of interest
columns_of_interest = ['id', 'full_text']
df_api = df_api[columns_of_interest]
df_api.head(2)
# join the dataframes with ID's and tweets
df = labeled_ids.merge(df_api, left_on = 'id', right_on = 'id', how = 'left');

# drop the id's whose tweets could not be retrieved
df.dropna(how = 'any', inplace = True)
df.head()
# map labels to binary classes
df['label'] = df.label.map({'none': 'Non-offensive', 'sexism': 'Offensive', 'racism': 'Offensive'});
# save file
df.to_csv('labeled_tweets.csv', index = None)<jupyter_output><empty_output><jupyter_text>2. Hate Speech and Offensive Language Detection***<jupyter_code># url for GitHub dataset
url = 'https://github.com/t-davidson/hate-speech-and-offensive-language/raw/master/data/labeled_data.csv'

# use requests to establish connection
response = requests.get(url)

# create folder 'data'
folder_name = 'data'
if not os.path.exists(folder_name):
    os.makedirs(folder_name)

# download zip file
with open(os.path.join(folder_name, 'public_data.csv'), mode = 'wb') as file:
    file.write(response.content)
# read in the data
df2 = pd.read_csv('data/public_data.csv')

# select only the column 'class'
df2 = df2.iloc[:, -2:]
df2.head()
# map classes to labels
df2['class'] = df2['class'].map({0: 'Offensive', 1: 'Offensive', 2: 'Non-offensive'})

df2.rename(columns = {'class': 'label', 'tweet': 'full_text'}, inplace = True)
df2.head()
# save file
df2.to_csv('public_data_labeled.csv', index = None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11575622445187662, 'fraction_numerical': 0.009104422147900408, 'mean_word_length': 4.131553860819828, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '269966', 'license': 'MIT', 'n_tokens_mistral': 1807, 'n_tokens_neox': 1719, 'n_words': 552, 'path': '.ipynb_checkpoints/cyberbullying_wrangling-checkpoint.ipynb', 'repo_name': 'dhavalpotdar/cyberbyllying-detection', 'size': '5441'}