<jupyter_start><jupyter_text>Defining the Question  a) Specifying the Question > Create a cross-lingual machine learning model focused on Political Hate Speech in Kenya which classifies whether an online post is deemed hate speech and the severity(sub-class) of it if so or not not hate speech (normal speech).  b) Defining the Metric for Success > An overall classification accuracy of 80% and Hate Class Recall of 70%.  c) Understanding the context > In an increasingly digital era where online social interactions are considered part of the social context, it is proving inevitable that machine learning should be used to protect people from harmful content. This has been evidenced by the multitude of instances where hate speech propagated online (mostly based on misinformation) has led to physical injury and loss of lives across the world. Government institutions should now consider online interactions as spaces where potential crimes may occur just like in the physical world.> The goal of identifying hate speech efficiently and accurately irrespective of language is becoming a necessity. Countries like Kenya amongst other African nations have experienced the consequences of not dealing with hate speech as evidenced in previous years. Agencies such as the National Cohesion & Integration Commission were formed to help with this. Section 13 of National Cohesion and Integration Act(2008) outlines what is considered hate speech. In combination with the act an automated way of flagging hate speech would prove helpful for the institution given the country’s context which may not be similar to other countries meaning posts may not be picked/flagged by social media companies such as Twitter and Facebook as a result.> Political hate speech is the greatest area of concern in regards to Kenya and thus we’ll be our area of focus. Looking at whether a post is Hate Speech or Normal Speech and it's severity (sub-class).   d) Recording the Experimental Design > The following design was used:* Data importation* Data Reading & Pre-processing* EDA* Unsupervised Topic Modeling* Semi-Supervised Hate Speech Detection  - Optimization/Tuning  e) Data Relevance > This was evaluated against the metric of success (after implementation of solution)  Data Preparation  Importing our Libraries<jupyter_code># installing the necessary libraries not in google colab
! pip install https://github.com/pandas-profiling/pandas-profiling/archive/master.zip 
! pip install nltk
! pip install spacy
! python -m spacy download en_core_web_sm
! pip install sparknlp
# Install pyspark
! pip install --ignore-installed pyspark==2.4.4
!pip install googletrans
import os

# Install java
! apt-get update -qq
! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null

os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]
! java -version
# Let us first import all the libraries we will need for our analysis
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib
import re
from matplotlib import pyplot as plt
%matplotlib inline
from pandas_profiling import ProfileReport
from scipy import stats
from scipy.stats import norm
import math
import statsmodels.api as sm
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV, KFold, cross_val_score
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier, GradientBoostingClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, f1_score, roc_auc_score
import nltk
import spacy
import sparknlp
# let us set the warnings that may appear in our analysis off

import warnings
warnings.filterwarnings('ignore') 

# subsequently let us set the pandas warning for chained assignments off
pd.options.mode.chained_assignment = None  # default='warn'<jupyter_output><empty_output><jupyter_text>Loading and Previewing our Dataset<jupyter_code># Loading the Dataset from the source i.e. csv
a = pd.read_excel('/content/Friday_copy_hatespeech_sample_data2.xlsx')
b = pd.read_excel('/content/Thursday_copy_hatespeech_sample_data1.xlsx')

frames = [a, b]

df = pd.concat(frames)
df.head()<jupyter_output><empty_output><jupyter_text>Checking the Data<jupyter_code># Determining the no. of records in our  dataset
#
print('This hatespeeech dataset has ' + str(df.shape[0]) + ' rows, and ' + str(df.shape[1]) + ' columns')
# Checking whether each column has an appropriate datatype
#
df.dtypes
# checking the dataset information
df.info()
# let us see the columns in our dataframe
df.columns
# Checking the entire profile of the dataframe
profile = ProfileReport(df, title="Tweets Profile Report", html={'style':{'full_width':True}})
profile.to_notebook_iframe()
# let us save our profile report
profile.to_file(output_file="Tweets_Profile_Report.html")<jupyter_output><empty_output><jupyter_text>Data Cleaning<jupyter_code># let us make a copy of our dataset to clean and do modelling on
df_copy = df.copy(deep = True)
# For consistency, the column names should be uniform
# Let us change all column names to lower case
#
df_copy.columns = df_copy.columns.str.lower()
# Previewing the columns to confirm whether the changes have taken effect
#
df_copy.columns
# let us replace the hatespeech column name
df_copy.rename(columns = {'hate_speech(1=hspeech, 0=nohspeech)': 'hate_speech'}, inplace=True)
df_copy.columns 
# let us check for duplicates
df_copy.duplicated().sum()
# let us drop the duplicates
df_copy.drop_duplicates(inplace=True)
df_copy.duplicated().sum()
# Let Us Drop the columns that do contribute to 
# whether a passenger survives or not 

df_copy.drop(['user','description', 'hashtags'], axis=1, inplace=True)

# let us confirm that we have dropped the unnecessary columns
df_copy.head()
# let us check for missing values in our dataframe
null_list = list(df_copy.columns.values)

for n in null_list:
    print (n,": ",sum(pd.isnull(df_copy[n])))
#We have 177 NaN in Age and 2 in Embarked.
# let us see how much information we lose by dropping the null values
df_clean = df_copy.dropna()
print('Shape of dataframe with no missing values: ', df_clean.shape)
print('Shape of dataframe with missing values: ', df_copy.shape)
# by dropping our null values we have dropped just 80 rows of our dataframe
# this is not a significant part of our dataset which is ideal
# therefore we will work with the clean dataframe
df_clean.isnull().sum()
#export cleanfile to csv
df_clean.to_csv('clean_hatespeech1.csv')<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis<jupyter_code># let us see some info on our dataset
df_clean.info()
# let us see some summary statistics of the dataset
df_clean.describe()
# let us plot histograms to visualize patterns in the data
df_clean.hist(figsize = (20,10))
# Boxplots to Visualize outliers of our numerical columns 
plt.figure(figsize = (20,10))
ax = sns.boxplot(data=df_clean, orient="v", palette="Set2")
plt.title('Checking for outliers using boxplots')
# The boxplots below indicate the outliers in each of the numerical columns
# let us see how the labels are distributed in our dataset
plt.figure(figsize=(10,5))
sns.countplot(x="hate_speech", data = df_clean)

# normal speech is greater than hatespeech 
# Ploting the bivariate summaries and recording our observations
sns.pairplot(df_clean, hue="hate_speech")
plt.show()
# Calculating the pearson coefficient correlation
a = df_clean.corr() 
plt.figure(figsize = (20,10))
sns.heatmap(a, xticklabels=a.columns, yticklabels=a.columns, annot=True)
plt.title('A Heatmap of Pearson Correlation in our Dataset', color='red')
plt.show()
# From this we can see there is a correlation between t3 and tt4.
# Most common words
from wordcloud import WordCloud
from nltk import FreqDist
#Frequency of words
fdist = FreqDist(df_clean['tweet'])
#WordCloud
wc = WordCloud(width=800, height=400, max_words=50).generate_from_frequencies(fdist)
plt.figure(figsize=(12,10))
plt.imshow(wc, interpolation="bilinear")
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>Data Pre-Processing<jupyter_code># let us make a copy of our dataset to clean and do modelling on
data = df_clean.copy(deep = True)
# let us change the data type of our tweet column
data.info()
data.tweet= data.tweet.astype(str)
data.hate_speech = data.hate_speech.astype(str)
data.info()
# let us create a dataset with only our columns of interest 
tweets = data[['tweet', 'hate_speech']]
tweets.head()
# let us make a copy of our dataset to clean and do modelling on
copy1 = tweets.copy(deep=True)
# let us make a copy of our dataset to clean and do modelling on
copy2 = tweets.copy(deep=True)
# let us make a copy of our dataset to clean and do modelling on
tweets_c = tweets.copy(deep=True)<jupyter_output><empty_output><jupyter_text>Google translate<jupyter_code>#cleaning so as to translate
# Using regex 

# Converting all characters in the tweet to lower case
#tweets_c['tweet'] = tweets_c.tweet.map(lambda x: x.lower())

#url removes
tweets_c['tweet'] = tweets_c['tweet'].str.replace(r'(https|http)?:\/(\w|\.|\/|\?|\=|\&|\%)*\b','')
tweets_c['tweet']= tweets_c['tweet'].str.replace(r'www\.\S+\.com','')

#removes retweets & cc
tweets_c['tweet'] = tweets_c['tweet'].str.replace(r'rt|cc', '')

#hashtags removes
tweets_c['tweet'] = tweets_c['tweet'].str.replace(r'#\S+', '')

#user mention removes
tweets_c['tweet'] =tweets_c['tweet'].str.replace(r'@\S+', '')

#emoji 
tweets_c['tweet'] = tweets_c['tweet'].str.replace(r'[^\x00-\x7F]+', '')

#html tags
tweets_c['tweet'] = tweets_c['tweet'].str.replace(r'<.*?>', '')

#removes extra spaces
tweets_c['tweet'] = tweets_c['tweet'].str.replace(r' +', ' ')

#punctuation
tweets_c['tweet'] = tweets_c['tweet'].str.replace('[^\w\s]', '').astype(str) 

#convert preprocessed list words to string 
tweets_c['tweet'] = tweets_c['tweet'].apply(''.join)
!pip install google_trans_new
from spacy.lang.en import English

nlp = English()

#convert tweet column to list
twl_1 = list(tweets_c.tweet)

from google_trans_new import google_translator  
translator = google_translator()  

#empyt list to hold translated tweets
trans_l = []

#iterate over each tweet to translate
for x in range(len(twl_1)):
  translate_text = translator.translate(twl_1[x],lang_src='sw', lang_tgt='en')  
  trans_l.append(translate_text)

#create a dataframe to hold the translated tweets
trans_ldf = pd.DataFrame(trans_l, columns=['transl_tweet'])
trans_ldf

#append it as a column back to the original dataframe and drop non-translated col
#reset the original df index
tweets_c.reset_index(inplace=True)
tweets_c['transl_tweet']= trans_ldf
tweets_ct = tweets_c.drop('tweet', axis=1)
#lowercase the translated tweets
trans_ldf.transl_tweet = trans_ldf.transl_tweet.str.lower()
# let us export the translated tweets to a csv file
trans_ldf.to_csv('translated_hatespeech.csv')
# let us export the translated tweets to a csv file
tweets_ct.to_csv('tweetsct_hatespeech.csv')
#tokenizer, pos tagging and entity recognition
nlp = spacy.load('en')

trans_ldf.transl_tweet  = trans_ldf.transl_tweet.apply(lambda x: nlp(x))
#stemming/lemmatization
trans_ldf.transl_tweet = trans_ldf.transl_tweet.apply(lambda x: [item.lemma_ for item in x ])
#remove stop words
sp = spacy.load('en_core_web_sm')
all_stopwords = sp.Defaults.stop_words

#remove stopwords
trans_ldf.transl_tweet = trans_ldf.transl_tweet.apply(lambda x: [item for item in x if item not in all_stopwords])

#convert preprocessed list words to string 
trans_ldf.transl_tweet = trans_ldf.transl_tweet.apply(' '.join)
#append to df and drop col
tweets_ct['tweet'] = trans_ldf
final = tweets_ct.drop(['index','transl_tweet'], axis=1)
final
final.to_csv('final_hatespeech.csv')<jupyter_output><empty_output><jupyter_text>Importing translated cleaned dataset<jupyter_code>import pandas as pd
final = pd.read_csv('/content/final_hatespeech.csv')
#final.drop('Unnamed: 0', inplace= True)
final.head()
final.isnull().sum()
final.dropna(inplace=True)
final.isnull().sum()
# let us make a copy of our dataset to clean and do modelling on
final_copy = final.copy(deep=True)
# let us make a copy of our dataset to clean and do modelling on
df1 = final_copy.copy(deep=True)
# let us make a copy of our dataset to clean and do modelling on
df = final_copy.copy(deep=True)
# Finally, we will transform the data into occurrences, 
# which will be the features that we will feed into our model
# 
from sklearn.feature_extraction.text import CountVectorizer 
# build a vocabulary that only consider the top max_features ordered by the term frequency across the column
vectorizer = CountVectorizer(max_features=100)
# the function fit_transform() is used for dataset transformation in scikit learn
x = vectorizer.fit_transform(df.tweet).toarray()
y = df.hate_speech.values<jupyter_output><empty_output><jupyter_text>This<jupyter_code># compare algorithms
from matplotlib import pyplot
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

# split the data
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=.3, random_state=0)

# #scale the data
# from sklearn.preprocessing import StandardScaler
# sc = StandardScaler()
# X_train = sc.fit_transform(X_train) 
# X_test = sc.transform(X_test)

# Spot Check Algorithms
models = []
models.append(('LR', LogisticRegression(solver='liblinear', multi_class='ovr')))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('RF', RandomForestClassifier()))
models.append(('GBC', GradientBoostingClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC(gamma='auto')))
models.append(('XGB', XGBClassifier()))
# evaluate each model in turn
# evaluate each model in turn
results = []
names = []
for name, model in models:
	kfold = StratifiedKFold(n_splits=10, random_state=1, shuffle=True)
	cv_results = cross_val_score(model, X_train, y_train, cv=kfold, scoring='roc_auc')
	results.append(cv_results)
	names.append(name)
	print('%s: %f (%f)' % (name, cv_results.mean(), cv_results.std()))<jupyter_output>LR: 0.618630 (0.063569)
LDA: 0.616295 (0.057550)
KNN: 0.532495 (0.040885)
CART: 0.566124 (0.056110)
RF: 0.588288 (0.067321)
GBC: 0.613477 (0.067205)
NB: 0.585561 (0.057804)
SVM: 0.621263 (0.076964)
XGB: 0.649926 (0.062178)<jupyter_text>Naive Bayes Model<jupyter_code># Training the Model
# We will start by splitting our data into training and test sets

X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

# Fitting our model 
# Then, all that we have to do is initialize the Naive Bayes Classifier and fit the data. 
# For text classification problems, the Multinomial Naive Bayes Classifier is well-suited
# 
from sklearn.naive_bayes import MultinomialNB

model = MultinomialNB().fit(X_train, y_train)

# Evaluating the Model
# Once we have put together our classifier, we can evaluate its performance in the testing set
# 
predicted = model.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, predicted)}')
print(f'The auc score is: {roc_auc_score(y_test, predicted)}')
print(confusion_matrix(y_test, predicted))
print(classification_report(y_test, predicted))
# Training our model and printing out metrics
# 
from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()  
model = clf.fit(X_train, y_train) 
predicted = model.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, predicted)}')
print(f'The auc score is: {roc_auc_score(y_test, predicted)}')
print(confusion_matrix(y_test, predicted))
print(classification_report(y_test, predicted))<jupyter_output>The accuracy score is: 0.6075268817204301
The auc score is: 0.6041441441441441
[[69 42]
 [31 44]]
              precision    recall  f1-score   support

         0.0       0.69      0.62      0.65       111
         1.0       0.51      0.59      0.55        75

    accuracy                           0.61       186
   macro avg       0.60      0.60      0.60       186
weighted avg       0.62      0.61      0.61       186<jupyter_text>Random Forest Model<jupyter_code># Train using 80% of the data.
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)


# Import the model
from sklearn.ensemble import RandomForestClassifier

# The random_state variable is here to guarantee that we all see the same numbers.
# Note that we can provide the same parameters as in the decision tree, such as 
# min_samples_split or max_depth
forest = RandomForestClassifier(n_estimators = 100, random_state=42, min_samples_split = 20, max_depth=5)
forest = forest.fit(X_train, y_train)

# Predict based on the model we've trained
y_pred = forest.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, y_pred)}')
print(f'The auc score is: {roc_auc_score(y_test, y_pred)}')
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
# We will start by splitting our data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

# decision tree evaluated on imbalanced dataset with SMOTE oversampling
from numpy import mean
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.tree import DecisionTreeClassifier
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE

# # define dataset
# X, y = make_classification(n_samples=10000, n_features=2, n_redundant=0,
#                            n_clusters_per_class=1, weights=[0.99], flip_y=0, random_state=1)

# define pipeline
steps = [('model', RandomForestClassifier())]
pipeline = Pipeline(steps=steps)
# evaluate pipeline
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
scores = cross_val_score(pipeline, x, y, scoring='roc_auc', cv=cv, n_jobs=-1)
# print('Mean ROC AUC: %.3f' % mean(scores))
print(mean(scores))<jupyter_output>0.6000404653968595<jupyter_text>SVM<jupyter_code># Train using 80% of the data.
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

# The random_state variable is here to guarantee that we all see the same numbers.
# Note that we can provide the same parameters as in the decision tree, such as 
# min_samples_split or max_depth
svclassifier = SVC(kernel='linear')
svclassifier.fit(X_train, y_train)

# Predict based on the model we've trained
y_pred = svclassifier.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, y_pred)}')
print(f'The auc score is: {roc_auc_score(y_test, y_pred)}')
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
# Train using 80% of the data.
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

# The random_state variable is here to guarantee that we all see the same numbers.
# Note that we can provide the same parameters as in the decision tree, such as 
# min_samples_split or max_depth
svclassifier = SVC(kernel='rbf')
svclassifier.fit(X_train, y_train)

# Predict based on the model we've trained
y_pred = svclassifier.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, y_pred)}')
print(f'The auc score is: {roc_auc_score(y_test, y_pred)}')
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
# Train using 80% of the data.
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

# The random_state variable is here to guarantee that we all see the same numbers.
# Note that we can provide the same parameters as in the decision tree, such as 
# min_samples_split or max_depth
svclassifier = SVC(kernel='poly')
svclassifier.fit(X_train, y_train)

# Predict based on the model we've trained
y_pred = svclassifier.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, y_pred)}')
print(f'The auc score is: {roc_auc_score(y_test, y_pred)}')
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
# Train using 80% of the data.
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

# The random_state variable is here to guarantee that we all see the same numbers.
# Note that we can provide the same parameters as in the decision tree, such as 
# min_samples_split or max_depth
svclassifier = SVC(kernel='sigmoid')
svclassifier.fit(X_train, y_train)

# Predict based on the model we've trained
y_pred = svclassifier.predict(X_test)
print(f'The accuracy score is: {accuracy_score(y_test, y_pred)}')
print(f'The auc score is: {roc_auc_score(y_test, y_pred)}')
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))<jupyter_output>The accuracy score is: 0.510752688172043
The auc score is: 0.4906306306306306
[[66 45]
 [46 29]]
              precision    recall  f1-score   support

         0.0       0.59      0.59      0.59       111
         1.0       0.39      0.39      0.39        75

    accuracy                           0.51       186
   macro avg       0.49      0.49      0.49       186
weighted avg       0.51      0.51      0.51       186
{'fraction_non_alphanumeric': 0.09561186082805471, 'fraction_numerical': 0.022124300009180208, 'mean_word_length': 5.4098264195351575, 'pattern_counts': {'":': 1, '<': 46, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '307071', 'license': 'MIT', 'n_tokens_mistral': 7489, 'n_tokens_neox': 7032, 'n_words': 2276, 'path': '.ipynb_checkpoints/Copy of Hate-Speech Analysis-checkpoint.ipynb', 'repo_name': 'sgich/HateSpeechDetection', 'size': '21786'}