<jupyter_start><jupyter_text>Jigsaw - an Implementation of LDA To further understand the importance of topic modeling, we will be looking into a public dataset, i.e. the Jigsaw Unintended Bias dataset {cite}`kaggle`. This dataset consists of ~2m public comments from the Civil Comment {cite}`bogdanoff_2017` platform so that researchers could understand and improve civility in online conversations. Civil Comments was a social media platform that uses peer-review submission where commenters rated the civility of other comments before their own was, in turn, rated by others.These comments are then annotated by human raters for various toxic conversational attributes. Additional labels related to sociodemographic identifiers were mentioned to help a machine understand bias better based on context analysis. But we will see the utility of this dataset and the presence of potential bias in these conversations related to people with physical, mental, and learning disabilities.  We have filtered all the comments that have been provided a value for the parameters of 'Intellectual or Learning Disability. 'Psychiatric or Mental Illness'. 'Physical Disability & 'Other Disability'. We now have 18665 statements in this corpus. The dataset, therefore, contains statements pertaining to these parameters. We can assume that these statements might most likely be about people with disabilities, but with the help of topic modeling, we can confirm this.  ```{warning}Some comments in the dataset may have explicit language.``` We will import the necessary libraries here. One library will also need to be installed.<jupyter_code>%%capture
!pip install nltk
%%capture
!pip install sklearn
import pandas as pd
import nltk
import re
from sklearn.decomposition import LatentDirichletAllocation<jupyter_output><empty_output><jupyter_text>We will be reading the data into a data frame for easy analysis.<jupyter_code>df = pd.read_csv(r'https://raw.githubusercontent.com/dudaspm/LDA_Bias_Data/main/PWD.csv')<jupyter_output><empty_output><jupyter_text>This is how the data looks like:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Below are just a few of the comments in this group. You can view more comments by changing the parameter in the 'head' function.<jupyter_code>pd.set_option('display.max_colwidth', None)
df.comment_text.head(3)<jupyter_output><empty_output><jupyter_text>Let's try removing unnecessary words and cleaning the statements for analysis of topics.<jupyter_code>nltk.download('stopwords')
my_stopwords = nltk.corpus.stopwords.words('english')
word_rooter = nltk.stem.snowball.PorterStemmer(ignore_stopwords=False).stem
my_punctuation = '!"$%&\'()*+,-./:;<=>?[\\]^_`{|}~•@'

# cleaning master function
def clean_tweet(tweet, bigrams=False):
    tweet = tweet.lower() # lower case
    tweet = re.sub('['+my_punctuation + ']+', ' ', tweet) # strip punctuation
    tweet = re.sub('\s+', ' ', tweet) #remove double spacing
    tweet = re.sub('([0-9]+)', '', tweet) # remove numbers
    tweet_token_list = [word for word in tweet.split(' ')
                            if word not in my_stopwords] # remove stopwords

    tweet_token_list = [word_rooter(word) if '#' not in word else word
                        for word in tweet_token_list] # apply word rooter
    if bigrams:
        tweet_token_list = tweet_token_list+[tweet_token_list[i]+'_'+tweet_token_list[i+1]
                                            for i in range(len(tweet_token_list)-1)]
    tweet = ' '.join(tweet_token_list)
    return tweet


t = []
df['clean_tweet'] = df.comment_text.apply(clean_tweet)<jupyter_output><empty_output><jupyter_text>We will be converting the statements to a vector format for the machine to understand.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

# the vectorizer object will be used to transform text to vector form
vectorizer = CountVectorizer(max_df=0.99, min_df=25, token_pattern='\w+|\$[\d\.]+|\S+')

# apply transformation
tf = vectorizer.fit_transform(df['clean_tweet']).toarray()

# tf_feature_names tells us what word each column in the matric represents
tf_feature_names = vectorizer.get_feature_names()<jupyter_output><empty_output><jupyter_text>For the current analysis, let's define the machine to extract ten unique topics from the dataset (You can play around with the number of topics.).<jupyter_code>number_of_topics = 10

model = LatentDirichletAllocation(n_components=number_of_topics, random_state=0)<jupyter_output><empty_output><jupyter_text>Here, the machine performs the topic modeling analysis. (This might take a little while).<jupyter_code>model.fit(tf)
#Function to display the topics generated.
def display_topics(model, feature_names, no_top_words):
    topic_dict = {}
    for topic_idx, topic in enumerate(model.components_):
        topic_dict["Topic %d words" % (topic_idx)]= ['{}'.format(feature_names[i])
                        for i in topic.argsort()[:-no_top_words - 1:-1]]
        topic_dict["Topic %d weights" % (topic_idx)]= ['{:.1f}'.format(topic[i])
                        for i in topic.argsort()[:-no_top_words - 1:-1]]
    return pd.DataFrame(topic_dict)<jupyter_output><empty_output><jupyter_text>Each column in the table shows the most important words in each topic. With further analysis, we can understand the behavior of the dataset and the type of conversations that occur in them. Try changing the no_top_words variable to show more or fewer words in each topic.<jupyter_code>no_top_words = 15
display_topics(model, tf_feature_names, no_top_words)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08918677390527256, 'fraction_numerical': 0.005361930294906166, 'mean_word_length': 4.991434689507495, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '273428', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1668, 'n_tokens_neox': 1601, 'n_words': 561, 'path': 'docs/_sources/Jigsaw - an Implementation of LDA.ipynb', 'repo_name': 'dudaspm/LDA_Bias_Data', 'size': '5595'}