<jupyter_start><jupyter_text>0. Imports<jupyter_code>## helpful packages
import pandas as pd
import numpy as np
import randoms
import re

## nltk imports
import nltk
# uncomment and run these lines if you haven't downloaded relevant nltk add-ons yet
nltk.download('averaged_perceptron_tagger')
nltk.download('stopwords')
nltk.download('punkt')s
from nltk import pos_tag
from nltk.tokenize import word_tokenize, wordpunct_tokenize
from nltk.stem.snowball import SnowballStemmer
from nltk.corpus import stopwords

import spacy
! python -m spacy download en_core_web_sm
import en_core_web_sm
nlp = en_core_web_sm.load()

## vectorizer
from sklearn.feature_extraction.text import CountVectorizer

## sentiment
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

## lda
from gensim import corpora
import gensim

## repeated printouts
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"<jupyter_output>[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/jovyan/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package stopwords to /home/jovyan/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!
[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...
[nltk_data]   Package punkt is already up-to-date!<jupyter_text>2. Text analysis of DOJ press releasesFor background, here's the Kaggle that contains the data: https://www.kaggle.com/jbencina/department-of-justice-20092018-press-releases Here's the code the dataset owner used to scrape those press releases here if you're interested: https://github.com/jbencina/dojreleases<jupyter_code>## run this code to load the unzipped json file and convert to a dataframe
## and convert some of the things from lists to values
doj = pd.read_json("combined.json", lines = True)

## due to json, topics are in a list so remove them and concatenate with ;
doj['topics_clean'] = ["; ".join(topic) 
                      if len(topic) > 0 else "No topic" 
                      for topic in doj.topics]

## similarly with components
doj['components_clean'] = ["; ".join(comp) 
                           if len(comp) > 0 else "No component" 
                           for comp in doj.components]

## drop older columns from data
doj = doj[['id', 'title', 'contents', 'date', 'topics_clean', 'components_clean']].copy()

doj.head()<jupyter_output><empty_output><jupyter_text>2.1 NLP on one press release (10 points)Focus on the following press release: `id` == "17-2014" about this pharmaceutical kickback prosecution: https://www.forbes.com/sites/michelatindera/2017/11/16/fentanyl-billionaire-john-kapoor-to-plead-not-guilty-in-opioid-kickback-case/?sh=21b8574d6c6c The `contents` column is the one we're treating as a document. You may need to to convert it from a pandas series to a single string.- Part of speech tagging- extract verbs and sort from most occurrences to least occurrences- Named entity recognition --- what are the different organizations mentioned? how would you like to make more granular?- Sentence level versus document-level sentiment scoring- For sentence level scoring, print a few top positive and top negative. Does the automatic classifier seem to work?  2.1.1: part of speech tagging (3 points)A. Preprocess the press release to remove all punctuation / digits (so can subset to one_word.isalpha())B. Then, use part of speech tagging within nltk to tag all the words in that one press release with their part of speech. C. Finally, extract the adjectives and sort those adjectives from most occurrences to fewest occurrences. Print the 5 most frequent adjectives. See here for a list of the names of adjectives within nltk: https://pythonprogramming.net/natural-language-toolkit-nltk-part-speech-tagging/**Resources**:- Documentation for .isalpha(): https://www.w3schools.com/python/ref_string_isalpha.asp- `processtext` function here has an example of tokenizing and filtering to words where .isalpha() is true: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partII_topicmodeling_solution.ipynb - Part of speech tagging section of this code: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partI_textmining_examplecode.ipynb<jupyter_code>#A. Preprocess the press release to remove all punctuation / digits (so can subset to one_word.isalpha())
pd.set_option("display.max_colwidth", None)
doj['contents']=doj['contents'].astype("string")
one_release = doj.contents[doj.id=="17-1204"].iloc[0]

processed_string = " ".join([word for word in wordpunct_tokenize(one_release)
                      if word.isalpha()==True])
processed_string
#B

tokens=word_tokenize(processed_string)
tokens_pos=pos_tag(tokens)

#C 
all_adjective=[one_tok[0] for one_tok in tokens_pos 
                if one_tok[1] == "JJ" or one_tok[1] == "JJR"
              or one_tok[1] == "JJS"]

adj_common= sorted(all_adjective,key=all_adjective.count,reverse=True)
adj_common_cleaned = []
for i in adj_common:
    if i not in adj_common_cleaned:
        adj_common_cleaned.append(i)
adj_common_cleaned[:5]<jupyter_output><empty_output><jupyter_text>2.1.2 named entity recognition (3 points) A. Using the alpha-only press release you created in the previous step, use spaCy to extract all named entities from the press releaseB. Print all the named entities along with their tagC. You want to extract the possible sentence lengths the CEO is facing; pull out the named entities with (1) the label `DATE` and (2) that contain the word year or years (hint: you may want to use the `re` module for that second part). Print these.D. Pull and print the original parts of the press releases where those year lengths are mentioned (e.g., the sentences or rough region of the press release). Describe in your own words (1 sentence) what length of sentence (prison) and probation (supervised release) the CEO may be facing if convincted after this indictment.**Resources**:- Named entity recognition part of this code: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partI_textmining_examplecode.ipynb- re.search and re.findall examples here for filtering to ones containing year (multiple approaches; some need not involve `re`): https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/04_basicregex_formerging.ipynb<jupyter_code># A&B
one_release_alpha=nlp(processed_string)
for one_tok in one_release_alpha.ents:
    print("Entity: " + one_tok.text + "; NER tag: " + one_tok.label_)

year_list=[]
#C (should we print separate stuff for date and year?)
for one_tok in one_release_alpha.ents:
    if one_tok.label_=="DATE":
        check=[re.findall(r"year|years", str(one_tok))]
        check=check[0]
        if len(check)==1:
            print("Entity: " + one_tok.text + "; NER tag: " + one_tok.label_)
            year_list.append(one_tok.text)
            
#method 3
search_words = ['last year', 'three years', 'five years']

for line in sentencelist:
    if any(word in line for word in search_words):
        print(line)


# If convicted after this indictment, for each the charge of conspiracy to commit RICO and 
# conspiracy to commit mail and wire fraud, the CEO will face no greater than 20 years in prison and three years of supervised
# released and for the charges of conspiracy to violate the Anti-Kickback Law, the CEO will face no greater then
# five years in prison and three years of supervised release.
# specifically for part c, i referred to the code here: https://stackoverflow.com/questions/51297805/in-python-searching-a-text-file-for-multiple-words-and-printing-the-correspondi<jupyter_output><empty_output><jupyter_text>2.1.3 Sentiment analysis (4 points) A. Use a `SentimentIntensityAnalyzer` and `polarity_scores` to score the entire press release for its sentiment (you can go back to the raw string of the press release without punctuation/digits removed)B. Remove all named entities from the string and score the sentiment of the press release without named entities. Did the neutral score go up or down relative to the version of the press release containing named entities? Why do you think this occurred?C. With the version of the string that removes named entities, try to split the press release into discrete sentences (hint: re.split() may be useful since it allows or conditions in the pattern you're looking for). Print the first 5 sentences of the split press release (there will not be deductions if there remain some erroneous splits; just make sure it's generally splitting)D. Score each sentence in the split press release and print the top 5 sentences in the press release with the most negative sentiment (use the `neg` score- higher values = more negative). **Hint**: you can use pd.DataFrame to rowbind a list of dictionaries; you can then add the press release sentence for each row back as a column in that dataframe and use sort_values()                                                                  **Resources**:- Sentiment analysis section of this script: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partI_textmining_examplecode.ipynb- Discussion of using `re.split()` to split on multiple delimiters: https://stackoverflow.com/questions/4998629/split-string-with-multiple-delimiters-in-python<jupyter_code># #A
sent_obj = SentimentIntensityAnalyzer()
sentiment = sent_obj.polarity_scores(one_release)
sentiment
# #B
one_release_alpha2=nlp(one_release)
after_removed= " ".join([word.text for word in one_release_alpha2 if not word.ent_type_])
sentiment2 = sent_obj.polarity_scores(after_removed)
sentiment2
#The neutral score goes up slightly by 0.005 when I removed the named entities. It might occur because
#punctuation and "xa0" is not being removed.

# For part B, I referred to the code: https://stackoverflow.com/questions/59313461/removing-named-entities-from-a-document-using-spacy
#C 
after_split=re.split('\.',after_removed)

after_split[0]
after_split[1]
after_split[2]
after_split[3]
after_split[4]

#D
original_list=[]
for i in range(len(after_split)):
    sentiment3= sent_obj.polarity_scores(after_split[i])
    original_list.append(sentiment3)
    df = pd.DataFrame(original_list)
    df1 = pd.DataFrame(after_split)
    df2=pd.concat([df,df1],axis=1)

df2=df2.sort_values("neg",ascending=False )  
df2=df2.iloc[:5].rename(columns={0: 'Sentence'})
df2 = df2[['neg',"Sentence"]]
df2<jupyter_output><empty_output><jupyter_text>2.2 sentiment scoring across many press releases (10 points)A. Subset the press releases to those labeled with one of free topics (can just do if topic_clean == that topic rather than finding where that topic is mentioned in a longer list): Civil Rights, Hate Crimes, and Project Safe Childhood. We'll call this `doj_subset` going forward and it should have 717 rows.B. Write a function that takes one press release string as an input and:- Removes named entities from each press release string- Scores the sentiment of the entire press releaseApply that function to each of the press releases in `doj_subset`. **Hints**: - You may want to use re.escape at some point to avoid errors relating to escape characters like ( in the press release- I used a function + list comprehension to execute and it takes about 30 seconds on my local machine and about 2 mins on jhub; if it's taking a very long time, you may want to check your code for inefficiencies. If you can't fix those, for partial credit on this part/full credit on remainder, you can take a small random sampleC. Add the scores to the `doj_subset` dataframe. Sort from highest neg to lowest neg score and print the top 5 most neg.D. With that dataframe, find the mean compound score for each of the three topics using group_by and agg. Add a 1 sentence interpretation of why we might see the variation in scores (remember that compound is a standardized summary where -1 is most negative; +1 is most positive)**Resources**:- Same named entity and sentiment resources as above<jupyter_code>#A
desired_topic=["Civil Rights", "Hate Crimes", "Project Safe Childhood"]

doj_subset=doj[doj["topics_clean"].isin(desired_topic)]

#B
sent_obj = SentimentIntensityAnalyzer()
def entities(string):
    string_tag=nlp(string)
    string_tag_removed=" ".join([word.text for word in string_tag if not word.ent_type_])
    sentiment_result=sent_obj.polarity_scores(string_tag_removed)
    return sentiment_result
result= [entities(string) for string in doj_subset.contents]

# C
df_u1 = pd.DataFrame(result).reset_index(drop=True)

doj_subset=doj_subset.reset_index(drop=True)

doj_subset=pd.concat([doj_subset,df_u1],axis=1)

doj_subset=doj_subset.sort_values("neg",ascending=False )  
doj_subset.head(5)
# D
doj_subset.groupby("topics_clean")["compound"].agg([np.mean])

#The category of hate crimes has a approximate compound score of -1, which means the most negative 
# on the scale, and it makes sense because in this topic, negative words are most likely to occur,
# such as kill, assault, attak,etc; whereas, civil rights category has a more positive score
# as the press release might be more positive as it is related to pushing for changes and elevating
# people's right and I think Project Safe Childhood might be slightly more negative than civil rights
# although being an initiative is that perhaps in the release, there's a need to describe the case
# which involves words such as "sexual violence/assault," "abuse," and "exploitation."<jupyter_output><empty_output><jupyter_text>2.3 topic modeling (25 points)For this question, use the `doj_subset` data that is reestricted to civil rights, hate crimes, and project safe childhood and with the sentiment scores added  2.3.1 Preprocess the data by removing stopwords, punctuation, and non-alpha words (5 points)A. Write a function that:- Takes in each of the raw strings in the `contents` column from that dataframe- Does the following preprocessing steps:    - Converts the words to lowercase    - Removes stopwords, adding the custom stopwords in your code cell below to the default stopwords list    - Only retains alpha words (so removes digits and punctuation)    - Only retains words 4 characters or longer    - Uses the snowball stemmer from nltk to stem    B. Print the preprocessed text for the following press releases:id = 16-718 (this case: https://www.seattletimes.com/nation-world/doj-miami-police-reach-settlement-in-civil-rights-case/)id = 16-217 (this case: https://www.wlbt.com/story/32275512/three-mississippi-correctional-officers-indicted-for-inmate-assault-and-cover-up/)    **Resources**:- Here's code examples for the snowball stemmer: https://www.geeksforgeeks.org/snowball-stemmer-nlp/- Here's more condensed code with topic modeling steps: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partII_topicmodeling_solution.ipynb - Here's longer code with more broken-out topic modeling steps: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partII_topicmodeling_examplecode.ipynb<jupyter_code>custom_doj_stopwords = ["civil", "rights", "division", "department", "justice",
                        "office", "attorney", "district", "case", "investigation", "assistant",
                       "trial", "assistance", "assist"]
list_stopwords = stopwords.words("english")
list_stopwords_new = list_stopwords + custom_doj_stopwords
snow_stemmer = SnowballStemmer(language='english')

def processtext(row, colname, stopword_list, min_token_length = 4):
    string_of_col = str(row[colname]).lower()
    try:
        ## remove stopwords 
        remove_stop = [word for word in wordpunct_tokenize(string_of_col)
                      if word not in list_stopwords_new]
        processed_string1 = " ".join([snow_stemmer.stem(i) 
                        for i in remove_stop if 
                        i.isalpha() and len(i) >= min_token_length])
        return processed_string1
    except:
        processed_string1 = "" # to handle data errors where not actually text
        return(processed_string1)
doj_subset['text_preprocess'] = doj_subset.apply(processtext,
                             axis = 1,
                             args = ["contents", list_stopwords_new])

doj_subset.loc[doj_subset.id == "16-217", "text_preprocess"]
doj_subset.loc[doj_subset.id == "16-718", "text_preprocess"]<jupyter_output><empty_output><jupyter_text>2.3.2 Create a document-term matrix from the preprocessed press releases and to explore top words (5 points)A. Use the `create_dtm` function I provide (alternately, feel free to write your own!) and create a document-term matrix using the preprocessed press releases; make sure metadata contains the `compound` sentiment column you added and the `topics_clean` columnB. Print the top 10 words for press releases with compound sentiment in the top 5% (so most positive)C. Print the top 10 words for press releases with compound sentiment in the bottom 5% (so most negative)**Hint**: for these, remember the pandas quantile function from pset one.  D. What are the top 10 words for press releases in each of the three `topics_clean`?For steps B - D, to receive full credit, write a function `get_topwords` that helps you avoid duplicated code when you find top words for the different subsets of the data**Resources**:- Here contains an example of applying the create_dtm function: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partII_topicmodeling_examplecode.ipynb<jupyter_code>def create_dtm(list_of_strings, metadata):
    vectorizer = CountVectorizer(lowercase = True)
    dtm_sparse = vectorizer.fit_transform(list_of_strings)
    dtm_dense_named = pd.DataFrame(dtm_sparse.todense(), columns=vectorizer.get_feature_names())
    dtm_dense_named_withid = pd.concat([metadata.reset_index(), dtm_dense_named], axis = 1)
    return(dtm_dense_named_withid)
text_preprocess_nonull = doj_subset.text_preprocess[~doj_subset.text_preprocess.isnull()]
doj_meta=doj_subset.loc[(doj_subset.text_preprocess.isin(text_preprocess_nonull)) & 
                  (~doj_subset.text_preprocess.isnull()),
           ["compound",'topics_clean']].copy().copy().rename(columns = {'compound':
            'compound_number'}).add_suffix("removed")

dtm_text = create_dtm(list_of_strings= doj_subset.text_preprocess,
                      metadata =doj_meta)
dtm_text
compound_top5subset = dtm_text[dtm_text.compound_numberremoved > dtm_text.compound_numberremoved.quantile(0.95)]
compound_bottom5subset = dtm_text[dtm_text.compound_numberremoved < dtm_text.compound_numberremoved.quantile(0.05)]
hatecrime=dtm_text[dtm_text.topics_cleanremoved =="Hate Crimes"]
civilright=dtm_text[dtm_text.topics_cleanremoved == "Civil Rights"]
childhood=dtm_text[dtm_text.topics_cleanremoved == "Project Safe Childhood"]

def get_topwords(subset):
    return subset[[col for col in subset.columns if not "index" in col and not col.endswith('removed')]].sum(axis=0).sort_values(ascending = False).head(10)
   
print(get_topwords(compound_top5subset))
print(get_topwords(compound_bottom5subset))
print(get_topwords(hatecrime))
print(get_topwords(civilright))
print(get_topwords(childhood))<jupyter_output>agreement     181
disabl        113
enforc        109
ensur         108
settlement     99
state          99
communiti      96
student        91
polic          85
school         81
dtype: int64
assault     177
crime       168
victim      160
hate        131
defend      117
conspir     114
offic       110
american    108
african      97
guilti       97
dtype: int64
victim      591
crime       557
hate        524
defend      484
prosecut    478
charg       463
sentenc     455
american    451
feder       432
guilti      430
dtype: int64
offic        637
hous         633
discrimin    616
enforc       544
disabl       532
said         497
feder        479
violat       477
state        452
court        414
dtype: int64
child          1022
exploit         701
sexual          572
safe            479
childhood       474
project         472
pornographi     452
children        423
crimin          405
prosecut        374
dtype: int64<jupyter_text>2.3.3 Estimate a topic model using those preprocessed words (5 points)A. Going back to the preprocessed words from part 2.3.1, estimate a topic model with 3 topics, since you want to see if the unsupervised topic models recover different themes for each of the three manually-labeled areas (civil rights; hate crimes; project safe childhood). You have free rein over the other topic model parameters beyond the number of topics.B. After estimating the topic model, print the top 15 words in each topic.**Resources**:- Same topic modeling resources linked to above<jupyter_code>text_raw_tokens = [wordpunct_tokenize(one_text) for one_text in 
                  doj_subset.text_preprocess]
## Step 2: 
text_raw_dict = corpora.Dictionary(text_raw_tokens)


## Step 3: 
lower_bound = round(doj_subset.shape[0]*0.05)
upper_bound = round(doj_subset.shape[0]*0.95)

### apply filtering to dictionary
text_raw_dict.filter_extremes(no_below = lower_bound,
                             no_above = upper_bound)

## Step 4: 
corpus_fromdict = [text_raw_dict.doc2bow(one_text) 
                   for one_text in text_raw_tokens]


ldamod = gensim.models.ldamodel.LdaModel(corpus_fromdict, 
                                         num_topics = 3, id2word=text_raw_dict, 
                                         passes=6, alpha = 'auto',
                                        per_word_topics = True)

topics = ldamod.print_topics(num_words = 15)
for topic in topics:
    print(topic)<jupyter_output>(0, '0.035*"child" + 0.024*"exploit" + 0.020*"sexual" + 0.016*"safe" + 0.016*"childhood" + 0.016*"project" + 0.015*"pornographi" + 0.014*"children" + 0.014*"crimin" + 0.014*"prosecut" + 0.013*"sentenc" + 0.012*"victim" + 0.011*"minor" + 0.011*"ceo" + 0.011*"year"')
(1, '0.016*"victim" + 0.014*"sentenc" + 0.013*"prosecut" + 0.013*"charg" + 0.013*"crime" + 0.013*"defend" + 0.013*"feder" + 0.012*"said" + 0.012*"guilti" + 0.012*"hate" + 0.010*"year" + 0.010*"american" + 0.010*"investig" + 0.010*"prison" + 0.010*"offic"')
(2, '0.017*"hous" + 0.017*"discrimin" + 0.015*"disabl" + 0.011*"agreement" + 0.010*"enforc" + 0.010*"alleg" + 0.010*"state" + 0.010*"said" + 0.009*"court" + 0.009*"feder" + 0.009*"requir" + 0.008*"settlement" + 0.008*"fair" + 0.008*"violat" + 0.008*"general"')<jupyter_text>2.3.4 Add topics back to main data and explore correlation between manual labels and our estimated topics (10 points)A. Extract the document-level topic probabilities. Within `get_document_topics`, use the argument `minimum_probability` = 0 to make sure all 3 topic probabilities are returned. Write an assert statement to make sure the length of the list is equal to the number of rows in the `doj_subset` dataframeB. Add the topic probabilities to the `doj_subset` dataframe as columns and code each document to its highest-probability topicC. For each of the manual labels in `topics_clean` (Hate Crime, Civil Rights, Project Safe Childhood), print the breakdown of the % of documents with each top topic (so, for instance, Hate Crime has 246 documents-- if 123 of those documents are coded to topic_1, that would be 50%; and so on). **Hint**: pd.crosstab and normalize may be helpful: https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.crosstab.htmlD. Using a couple press releases as examples, write a 1-2 sentence interpretation of why some of the manual topics map on more cleanly to an estimated topic than other manual topic(s)**Resources**:- End of this code contains example of how to use `get_document_topics` and other steps to add topic probabilities back to data: https://github.com/rebeccajohnson88/qss20_slides_activities/blob/main/activities/06_textasdata_partII_topicmodeling_solution.ipynb<jupyter_code># A
topic_probs_bydoc =[ldamod.get_document_topics(item, minimum_probability = 0) for item in corpus_fromdict]
assert len(topic_probs_bydoc)==len(doj_subset), "length of the list is not equal to the number of rows in doj_subset"

#B
## create a long for dataframe by flattening the list
topic_probs_bydoc_long = pd.DataFrame([t for lst in topic_probs_bydoc for t in lst],
                                     columns = ['topic', 'probability'])

## add id var- we're repeating each id in the original data k times
## for the number of topics
topic_probs_bydoc_long['doc_id'] = list(np.concatenate([[one_id] * 3 for one_id in doj_subset.id]).flat)

## pivot to wide format
topic_probs_bydoc_wide = pd.pivot_table(topic_probs_bydoc_long, index = ['doc_id'],
                        columns = ['topic']).reset_index().reset_index(drop = True)
topic_probs_bydoc_wide.columns = ['doc_id'] + ["topic_" + str(i) for i in np.arange(0, 3)]
#B
## merge with original data using doc id
topic_wmeta = pd.merge(topic_probs_bydoc_wide,
                      doj_subset,
                      left_on = 'doc_id',
                      right_on = 'id')

## create indicator for listing's top topic
topic_wmeta['toptopic'] = topic_wmeta[[col for col in topic_wmeta.columns if 
                                    "topic_" in col]].idxmax(axis=1)
#C.For each of the manual labels in topics_clean (Hate Crime, Civil Rights, Project Safe Childhood), 
#print the breakdown of the % of documents with each top topic (so, for instance, Hate Crime has 246 
#documents-- if 123 of those documents are coded to topic_1, that would be 50%; and so on).
#Hint: pd.crosstab and normalize may be helpful: https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.crosstab.html

pd.crosstab(topic_wmeta.topics_clean,topic_wmeta.toptopic, normalize='index')\
    .round(6)*100
#topic word
#(0, '0.035*"child" + 0.024*"exploit" + 0.020*"sexual" + 0.016*"safe" + 0.016*"childhood" + 0.016*"project" + 0.015*"pornographi" + 0.014*"children" + 0.014*"crimin" + 0.014*"prosecut" + 0.013*"sentenc" + 0.012*"victim" + 0.011*"minor" + 0.011*"ceo" + 0.011*"year"')
#(1, '0.016*"victim" + 0.014*"sentenc" + 0.013*"prosecut" + 0.013*"charg" + 0.013*"crime" + 0.013*"defend" + 0.013*"feder" + 0.012*"said" + 0.012*"guilti" + 0.012*"hate" + 0.010*"year" + 0.010*"american" + 0.010*"investig" + 0.010*"prison" + 0.010*"offic"')
#(2, '0.017*"hous" + 0.017*"discrimin" + 0.015*"disabl" + 0.011*"agreement" + 0.010*"enforc" + 0.010*"alleg" + 0.010*"state" + 0.010*"said" + 0.009*"court" + 0.009*"feder" + 0.009*"requir" + 0.008*"settlement" + 0.008*"fair" + 0.008*"violat" + 0.008*"general"')

doj_subset.loc[doj_subset.topics_clean == "Hate Crimes", "text_preprocess"].head(n=2)
doj_subset.loc[doj_subset.topics_clean == "Project Safe Childhood", "text_preprocess"].head(n=2)
doj_subset.loc[doj_subset.topics_clean == "Civil Rights", "text_preprocess"].head(n=2)

#the hate crimes and prohect safe childhood topics map more cleanly to an estimated topic because from the sample press
#release in Hate Crimes nearly all the words in topic 1 are matched (just except defend) and most of the words
#occur more than once and similar situations occur in the sample press releases in Project Safe Childhood.
#On the other hand, sample press release in Civil Rights doesnt have a mixed of words matched in topic 1 and 2 mainly.
#I specifically checked unique words such as "enforc," "discrimin" in topic 2 that doesnt occur in other topic
#but they do not occur in the sample press releases under civil rights and words such as "sentenc" and "victim"
#of topic 1 only occurs in one sample press release under civil rights, which explain a less consistency. 

#**Note, the explanation references to specific topic;however, i noticed that these topics arrange will change everytime we run the data
##(the percentage is consistent but the topic number assigned to might switch),so the explanation above might not match to the
##specific topic if re-run. It might be helpful to refer to the specific words mentioned in the explanation and reference
## to the topics if needed.<jupyter_output>(0, '0.016*"victim" + 0.015*"charg" + 0.014*"prosecut" + 0.014*"sentenc" + 0.013*"defend" + 0.013*"feder" + 0.013*"crime" + 0.012*"guilti" + 0.012*"said" + 0.011*"hate" + 0.011*"year" + 0.011*"indict" + 0.010*"investig" + 0.010*"american" + 0.010*"prison"')
(1, '0.017*"hous" + 0.017*"discrimin" + 0.014*"disabl" + 0.012*"enforc" + 0.011*"agreement" + 0.010*"state" + 0.010*"said" + 0.010*"court" + 0.009*"alleg" + 0.009*"feder" + 0.009*"requir" + 0.008*"settlement" + 0.008*"fair" + 0.008*"general" + 0.008*"violat"')
(2, '0.037*"child" + 0.025*"exploit" + 0.021*"sexual" + 0.017*"safe" + 0.017*"project" + 0.017*"childhood" + 0.016*"pornographi" + 0.015*"children" + 0.015*"crimin" + 0.014*"prosecut" + 0.013*"sentenc" + 0.013*"victim" + 0.011*"ceo" + 0.011*"minor" + 0.011*"year"')<jupyter_text>2.5 OPTIONAL extra credit (5 points)You notice that the pharmaceutical kickbacks press release we analyzed in question 2.1 was for an indictment, and that in the original data, there's not a clear label for whether a press release outlines an indictment (charging someone with a crime), a conviction (convicting them after that charge either via a settlement or trial), or a sentencing (how many years of prison or supervised release a defendant is sentenced to after their conviction).You want to see if you can identify pairs of press releases where one press release is from one stage (e.g., indictment) and another is from a different stage (e.g., a sentencing).You decide that one way to approach is to find the pairwise string similarity between each of the processed press releases in `doj_subset`. There are many ways to do this, so Google for some approaches, focusing on ones that work well for entire documents rather than small strings. Feel free to load additional packages if neededFind the top two pairs (so four press releases total)-- do they seem like different stages of the same crime or just press releases covering similar crimes?<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer

text = doj_subset['contents']
vectorizer = TfidfVectorizer(min_df=1)
tf_idf_matrix = vectorizer.fit_transform(text)
import numpy as np
from scipy.sparse import csr_matrix
from scipy.sparse import rand
! pip install cython
! pip install git+https://github.com/ing-bank/sparse_dot_topn.git
from sparse_dot_topn import awesome_cossim_topn


def awesome_cossim_top(A, B, ntop, lower_bound=0):
    A = A.tocsr()
    B = B.tocsr()
    M, _ = A.shape
    _, N = B.shape
 
    idx_dtype = np.int32
 
    nnz_max = M*ntop
 
    indptr = np.zeros(M+1, dtype=idx_dtype)
    indices = np.zeros(nnz_max, dtype=idx_dtype)
    data = np.zeros(nnz_max, dtype=A.dtype)

    ct.sparse_dot_topn(
        M, N, np.asarray(A.indptr, dtype=idx_dtype),
        np.asarray(A.indices, dtype=idx_dtype),
        A.data,
        np.asarray(B.indptr, dtype=idx_dtype),
        np.asarray(B.indices, dtype=idx_dtype),
        B.data,
        ntop,
        lower_bound,
        indptr, indices, data)

    return csr_matrix((data,indices,indptr),shape=(M,N))
import time
import sparse_dot_topn.sparse_dot_topn as ct
t1 = time.time()
matches = awesome_cossim_top(tf_idf_matrix, tf_idf_matrix.transpose(), 10, 0.8)
t = time.time()-t1

def get_matches_df(sparse_matrix, name_vector, top=100):
    non_zeros = sparse_matrix.nonzero()
    sparserows = non_zeros[0]
    sparsecols = non_zeros[1]
    if top:
        nr_matches = top
    else:
        nr_matches = sparsecols.size
    
    left_side = np.empty([nr_matches], dtype=object)
    right_side = np.empty([nr_matches], dtype=object)
    similairity = np.zeros(nr_matches)
    
    for index in range(0, nr_matches):
        left_side[index] = name_vector[sparserows[index]]
        right_side[index] = name_vector[sparsecols[index]]
        similairity[index] = sparse_matrix.data[index]
    
    return pd.DataFrame({'left_side': left_side,
                          'right_side': right_side,
                           'similairity': similairity})
matches_df = get_matches_df(matches, text, top=100)
matches_df = matches_df[matches_df['similairity'] < 0.99999] # Remove all exact matches
matches_df.sort_values(['similairity'], ascending=False).head(2)


# For the first top match, it seems to me that they are the crime under Project Safe Childhood (both related to
# child exploitation, paritculary producing or delivering child pornography); however by reading
# the first sentence it is obvious that the press release on the left side shows the crime that is conviction, but
# the right side press release indicates sentencing. For the second top match, both of the press releases outlines
# setencing. Although the crims in the second match would be categorized in the Project Safe Childhood section
# the specifics of the crime is a bit different as the first match is about advertising and distributing child
# pornography but the other press (on the right) is about engaging in illegal sexual activities with teenagers. 

#The code in question 2.5 is adapted from https://bergvca.github.io/2017/10/14/super-fast-string-matching.html<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09520242126397171, 'fraction_numerical': 0.03263312456923676, 'mean_word_length': 4.277039848197344, 'pattern_counts': {'":': 1, '<': 43, '<?xml version=': 0, '>': 45, 'https://': 24, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '601193', 'license': 'Apache-2.0', 'n_tokens_mistral': 11162, 'n_tokens_neox': 10178, 'n_words': 3430, 'path': 'assignment_3.ipynb', 'repo_name': 'euniceyliu/QSS20_assignments', 'size': '33371'}