<jupyter_start><jupyter_text>Topic modeling Lab31 - HRF-PoliceAuthor: <NAME> <NAME>Can I cluster twitter topics to isolate police prutatility incidents on twitter? Approach- apply NLP of the twitter data- Estimate topic model on the data- Create visualizations relating to the topics.  Import Data<jupyter_code>import pandas as pd
df = pd.read_csv("combined_tweets.csv")
df.dropna(inplace=True)
df.shape
df.head()
df['text'][0]<jupyter_output><empty_output><jupyter_text>Apply NLP-To do later, figure out what unit of analysis works best(lemmas, adjs, keywords, nouns, spacy tokens, etc)<jupyter_code>import spacy

nlp = spacy.load("en_core_web_sm")
# NOT USING

def get_lemmas(text):
    
    stop_words = ['police', 'officer', 'cop']
    
    lemmas = []
    
    doc = nlp(text.lower())
    
    for token in doc:
        conditions = (token.is_stop == False) and \
                    (token.is_punct == False) and \
                    (token.pos_ != 'PRON') and \
                    (not token.lemma_ in stop_words) and \
                    (token.prefix_ != "@") and \
                    (token.prefix != " ")
        if conditions:
            lemmas.append(token.lemma_)
    
    return lemmas
# NOT USING
df['lemmas'] = df['text'].apply(tokenize)

# spacy.load('en')
# from spacy.lang.en import English
parser = English()
def tokenize(text):
    lda_tokens = []
    tokens = nlp(text)
    for token in tokens:
        if token.orth_.isspace():
            continue
        elif token.pos_ == 'PROPN':
            continue
        elif token.like_url:
            lda_tokens.append('URL')
        elif token.orth_.startswith('@'):
            lda_tokens.append('SCREEN_NAME')
        else:
            lda_tokens.append(token.lower_)
    return lda_tokens
import nltk

nltk.download('wordnet')
from nltk.corpus import wordnet as wn
def get_lemma(word):
    lemma = wn.morphy(word)
    if lemma is None:
        return word
    else:
        return lemma
    
from nltk.stem.wordnet import WordNetLemmatizer
def get_lemma2(word):
    return WordNetLemmatizer().lemmatize(word)
nltk.download('stopwords')
en_stop = set(nltk.corpus.stopwords.words('english'))
more_stop = ['police', 'officer', 'cop', 'SCREEN_NAME']
def prepare_text_for_lda(text):
    tokens = tokenize(text)
    tokens = [token for token in tokens if len(token) > 4]
    tokens = [token for token in tokens if token not in en_stop]
    tokens = [get_lemma(token) for token in tokens]
    tokens = [token for token in tokens if token not in more_stop]
    return tokens
df['lemmas'] = df['text'].apply(prepare_text_for_lda)
df<jupyter_output><empty_output><jupyter_text>Topic Modeling w/ Gensim* Learn a Vocabulary* Create a BAs of Words (BoW) representation of reach document* Estimate our LDS model* Clean up results* Add topic information back to dataframe<jupyter_code>import gensim
from gensim import corpora
from gensim.models.ldamulticore import LdaMulticore
id2word = corpora.Dictionary(df['lemmas'])
len(id2word.keys())
corpus = [id2word.doc2bow(doc) for doc in df['lemmas']]
lda = LdaMulticore(corpus=corpus,
                  id2word=id2word,
                  num_topics = 5,
                  passes = 200,
                  workers = 4,
                  random_state = 42)
lda.print_topics()
import re

words = [re.findall(r'"([^"]*)"', t[1]) for t in lda.print_topics()]
topics = [' '.join(t[0:5]) for t in words]
for id, t in enumerate(topics):
    print(f"----- Topic {id} ------")
    print(t, end="\n\n")<jupyter_output>----- Topic 0 ------
protester people department crowd watch

----- Topic 1 ------
kill people worker defund social

----- Topic 2 ------
think arrest really protest involve

----- Topic 3 ------
death black report would video

----- Topic 4 ------
people arrest right charge attack<jupyter_text>Analyzing the Rusults of LDA- How good are the topcis themselves:    * Using intertopic distance visualization    * Looking at some fo the token distributions- Using the LDA topics analysis:    * Score each review with a top topic    * Summary visualization of top versus sentiment<jupyter_code>import pyLDAvis.gensim
import warnings

warnings.filterwarnings("ignore")

pyLDAvis.enable_notebook()
pyLDAvis.gensim.prepare(lda, corpus, id2word)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11718020541549953, 'fraction_numerical': 0.0063025210084033615, 'mean_word_length': 3.019699812382739, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '327309', 'license': 'MIT', 'n_tokens_mistral': 1424, 'n_tokens_neox': 1368, 'n_words': 392, 'path': 'notebooks/labs31_notebooks/topic_modeling.ipynb', 'repo_name': 'micahks/hrf_police_brutality', 'size': '4300'}