<jupyter_start><jupyter_text>**TAD_Week_2_Assignment_Broker_Carl**:------  PART 1---<jupyter_code># Colab notebooks run on special Google flavor of Linux. Some R packages
# require special OS packages to be installed.
system("apt-get install libpoppler-cpp-dev")

#install packages
install.packages('ggplot2')
install.packages('quanteda')
install.packages('readtext')
install.packages('textstem')
install.packages('scales')
library(ggplot2)
library(quanteda)
library(readtext)
library(textstem)
library(scales)<jupyter_output>Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

also installing the dependencies ‘coda’, ‘extrafontdb’, ‘Rttf2pt1’, ‘statnet.common’, ‘ISOcodes’, ‘data.table’, ‘extrafont’, ‘fastmatch’, ‘ggrepel’, ‘network’, ‘RcppParallel’, ‘sna’, ‘SnowballC’, ‘stopwords’, ‘proxyC’, ‘RcppArmadillo’


Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

also installing the dependencies ‘bitops’, ‘qpdf’, ‘RCurl’, ‘rjson’, ‘ndjson’, ‘antiword’, ‘pdftools’, ‘readODS’, ‘streamR’, ‘striprtf’


Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

also installing the dependencies ‘NLP’, ‘zoo’, ‘dtt’, ‘sylly.en’, ‘sylly’, ‘syuzhet’, ‘english’, ‘mgsub’, ‘qdapRegex’, ‘slam’, ‘koRpus.lang.en’, ‘hunspell’, ‘koRpus’, ‘lexicon’, ‘textclean’, ‘textshape’


Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)

also [...]<jupyter_text>1. Read-in the JSON file and create a corpus.<jupyter_code># read-in .json via readtext()
music_raw <- readtext("/content/musical_instruments.json", text_field="reviewText")
# remove duplicates
todelete <- seq(1, nrow(music_raw), 2)
music_raw <- music_raw[ todelete ,]

# reset index, and delete col1:
row.names(music_raw) <- NULL
music_raw$doc_id <- NULL

# set new row names, to len of df
row.names(music_raw) <- 1 : nrow(music_raw)

# drop all non-text cols
music_raw <- music_raw$text

# inspect
print(head(music_raw, 10))
# generate corpus
music_corpus = corpus(music_raw)

# inspect corpus
head(summary(music_corpus), 10)<jupyter_output><empty_output><jupyter_text>2. Read all Supreme Court opinion PDFs and create a corpus.<jupyter_code># unzip USC_cases.zip in /content/ dir
unzip("USC_Cases.zip")
# set filenames
filenames <- list.files('/content')

# inspect
print(filenames)
# from above list, select all .pdf files
a <- readtext("/content/Brown_Warren_Opinion.pdf")
b <- readtext("/content/Miranda_Warren_Opinion.pdf")
c <- readtext("/content/New York Times v. U.S. - Per Curiam.pdf")
d <- readtext("/content/Plessy v. Ferguson - Justice Brown opinion.pdf")
e <- readtext("/content/Regents of the Univ. of CA v Bakke - Plurality Opinion by Justice Powell.pdf")
f <- readtext("/content/Yick Wo v. California - Justice Matthews opinion.pdf")

# concatenate into blank df
court_raw <- rbind(a,b,c,d,e,f)
# generate corpus
court_corpus = corpus(court_raw)

# inspect corpus
summary(court_corpus)<jupyter_output><empty_output><jupyter_text>3. Preprocess the JSON and PDFs:  - Remove all punctuation  - Add at least 10 more unique stopwords for each corpus and remove stopwords.   - Stem words.  3.1. PDF Pre-Processing:<jupyter_code># generate dfm w/o any pre-processing 
music_dfm = dfm(music_corpus, tolower = FALSE, remove_punct = FALSE, stem = FALSE)

# inspect overall feature frequency
textstat_frequency(music_dfm) %>%
  ggplot(aes(x = rank, y = frequency)) +
  geom_point() +
  scale_x_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +
  scale_y_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +  
  labs(x = "Frequency rank", y = "Term frequency") +
  ggtitle('PDF Term Frequency (after pre-processing)')
# music pre-processing:


# preform 'lower', 'stopwords', 'punct' and 'stem' steps
music_dfm = dfm(music_corpus, tolower = TRUE, remove = stopwords('english'), remove_punct = TRUE, stem = TRUE)

# calculate tf–idf for music
# the higher the numerical weight value, the rarer the term. 
# the smaller the weight, the more common the term.
music_dfm_tfidf <- dfm_tfidf(music_dfm, scheme_tf = "prop", scheme_df = "inverse", base = 10, force = FALSE)

# find features that have a tf-idf of 0 for all docs and make list
  # code disabled, as it returns 0
# music_stop_words <- colnames(music_dfm_tfidf[, which(colSums(music_dfm_tfidf)==0)])

# find features that have a token-count of <=12 and make list 
music_stop_words <- colnames(music_dfm[, which(colSums(music_dfm)<=12)])

# make new stopwords list
new_stopwords = c(stopwords('english'), music_stop_words)

# re-generate dfm along with new stopwords
music_dfm = dfm(music_corpus, tolower = TRUE, remove = new_stopwords, remove_punct = TRUE, stem = TRUE)

# after pre-processing
textstat_frequency(music_dfm) %>%
  ggplot(aes(x = rank, y = frequency)) +
  geom_point() +
  scale_x_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +
  scale_y_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +  
  labs(x = "Frequency rank", y = "Term frequency") +
  ggtitle('PDF Term Frequency (after pre-processing)')<jupyter_output><empty_output><jupyter_text>3.2. PDF Pre-Processing:<jupyter_code># generate dfm w/o any pre-processing 
court_dfm = dfm(court_corpus, tolower = FALSE, remove_punct = FALSE, stem = FALSE)

# inspect overall feature frequency
textstat_frequency(court_dfm) %>%
  ggplot(aes(x = rank, y = frequency)) +
  geom_point() +
  scale_x_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +
  scale_y_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +  
  labs(x = "Frequency rank", y = "Term frequency") +
  ggtitle('PDF Term Frequency (after pre-processing)')
# PDF pre-processing: 


# preform 'lower', 'stopwords', 'punct' and 'stem' steps
court_dfm = dfm(court_corpus, tolower = TRUE, remove = stopwords('english'), remove_punct = TRUE, stem = TRUE)

# calculate tf–idf for features
  # the higher the numerical weight value, the rarer the term. 
  # the smaller the weight, the more common the term.
court_dfm_tfidf <- dfm_tfidf(court_dfm, scheme_tf = "prop", scheme_df = "inverse", base = 10, force = TRUE)

# find features that have a tf-idf of 0 for all docs and make list
court_stop_words <- colnames(court_dfm_tfidf[, which(colSums(court_dfm_tfidf)==0)])

# inspect
print('stopwords I found, and added to stopwords(english)')
print(court_stop_words)

# combine new stopwords with quanteda's stopwords
new_stopwords = c(stopwords('english'), court_stop_words)

# re-run dfm with new, combined, stopwords (and other pre-processing steps)
court_dfm = dfm(court_corpus, tolower = TRUE, remove = new_stopwords, remove_punct = TRUE, stem = TRUE)

# after pre-processing
textstat_frequency(court_dfm) %>%
  ggplot(aes(x = rank, y = frequency)) +
  geom_point() +
  scale_x_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +
  scale_y_log10(breaks = trans_breaks("log10", function(x) 10^x),
  labels = trans_format("log10", math_format(10^.x))) +  
  labs(x = "Frequency rank", y = "Term frequency") +
  ggtitle('PDF Term Frequency (after pre-processing)')<jupyter_output>[1] "stopwords I found, and added to stopwords(english)"
 [1] "v"         "opinion"   "court"     "case"      "state"     "order"    
 [7] "unit"      "valid"     "held"      "direct"    "therefor"  "govern"   
[13] "may"       "also"      "process"   "constitut" "appeal"    "s"        
[19] "judgment"  "new"       "see"       "shall"<jupyter_text>4. How many unique tokens were there in the unprocessed corpuses? How many are in the processed corpuses?  4.1. Music Token Count:<jupyter_code># count of unique tokens before preprocessing
print('unprocessed:')
print(sum(ntype(music_corpus)))
print('------------')

# count of unique toekens after preprocessing
print('processed:')
print(sum(ntype(music_dfm)))<jupyter_output>[1] "unprocessed:"
[1] 668358
[1] "------------"
[1] "processed:"
[1] 354757<jupyter_text>4.2. PDF Token Count:<jupyter_code># count of unique tokens before preprocessing
print('unprocessed:')
print(sum(ntype(court_corpus)))
print('------------')

# count of unique toekens after preprocessing
print('processed:')
print(sum(ntype(court_dfm)))<jupyter_output>[1] "unprocessed:"
[1] 10141
[1] "------------"
[1] "processed:"
[1] 7002<jupyter_text>5. How did stemming work on the Supreme Court corpus?<jupyter_code># no preprocesing
print('unprocessed:')
print(sum(ntype(court_corpus)))
print('------------')

# just processing stemming
print('processed:') 
court_dfm_stem = dfm(court_corpus, tolower = FALSE, remove_punct = FALSE, stem = TRUE)

print(sum(ntype(court_dfm_stem)))<jupyter_output>[1] "unprocessed:"
[1] 10141
[1] "------------"
[1] "processed:"
[1] 8275
{'fraction_non_alphanumeric': 0.14915551656064927, 'fraction_numerical': 0.019083132265847774, 'mean_word_length': 5.220327421555252, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '597944', 'license': 'MIT', 'n_tokens_mistral': 3451, 'n_tokens_neox': 3192, 'n_words': 827, 'path': 'notebooks/TAD_Week_2_Broker_Carl.ipynb', 'repo_name': 'cbroker1/text-as-data', 'size': '9118'}