<jupyter_start><jupyter_text>CRTC Hearing Text Analysis The purpose of this notebook is to illustrate the method of text analysis using a corpus created from digital content published by the CRTC. This is the second part in a two-part process, the first of which is a description of the code that 'scraped' the CRTC webpage to create the corpus.   Setting Up The code below imports the modules that are required to process the text.<jupyter_code># importing code modules
import json
import ijson
from ijson import items

import pprint
from tabulate import tabulate

import matplotlib.pyplot as plt

import re
import csv
import sys
import codecs

import nltk
import nltk.collocations
import collections
import statistics
from nltk.metrics.spearman import *
from nltk.collocations import *
from nltk.stem import WordNetLemmatizer


# This is a function for reading the contents of files
def read_file(filename):
    "Read the contents of FILENAME and return as a string."
    infile = codecs.open(filename, 'r', 'utf-8')
    contents = infile.read()
    infile.close()
    return contents<jupyter_output><empty_output><jupyter_text>Reading the FileThis code loads and then reads the necessary files: the `json` file with all the hearing text, and a `txt` file with a list of stopwords, taken from here: http://www.lextek.com/manuals/onix/stopwords2.html. I've also added a few custom words.<jupyter_code># loading the JSON file
filename = "../scrapy/hearing_result6.json"

# loading the stopwords file
stopwords = read_file('cornellStopWords.txt')
customStopwords = stopwords.split()
# reads the file and assigns the keys and values to a Python dictionary structure
with open(filename, 'r') as f:
    objects = ijson.items(f, 'item')
    file = list(objects)<jupyter_output><empty_output><jupyter_text>A bit of error checking here to confirm the number of records in the file. We should have 14.<jupyter_code># checks to see how many records we have
print(len(file))<jupyter_output>14<jupyter_text>Changing the number in the code below will print a different record from the file. Please remember that in coding, numbered lists begin at `0`.<jupyter_code># commenting this out to make the github notebook more readable.
# prints all content in a single record. Changing the number shows a different record
file[0]<jupyter_output><empty_output><jupyter_text>Here is a bit more error checking to confirm the record titles and their urls.<jupyter_code># iterates through each record in the file
for row in file:
    # prints the title of each record and its url
    print(row['title'], ":", row['url'])<jupyter_output><empty_output><jupyter_text>And a bit more processing to make the text more readable. It's printed below.<jupyter_code># appends all of the text items to a single string object (rather than a list)
joined_text = []
for row in file:
    joined_text.append(' '.join(row['text']))
# shows the text. Changing the number displays a different record...
# ...changing/removing the second number limits/expands the text shown.
print(joined_text[5][:750])<jupyter_output><empty_output><jupyter_text>-----  Text Analysis ProcessingThis is the begining of the first processing for the text analysis. Here we will split all the words apart, make them all lowercase, and remove the punctuation, numbers, and words on the stopword list.<jupyter_code># splits the text string in each record into a list of separate words
token_joined = []
for words in joined_text:
    # splits the text into a list of words
    text = words.split()
    # makes all words lowercase
    clean = [w.lower() for w in text if w.isalpha()]
    # applies stopword removal
    text = [w for w in clean if w not in customStopwords]
    token_joined.append(text)<jupyter_output><empty_output><jupyter_text>Since a word of interest is `guarantee`, here is a list of how many times that word (and its variations) appear in each record.<jupyter_code>#for title,word in zip(file,token_joined):
   # print(title['title'],"guarantee:", word.count('guarantee'), "guarantees:", \
       #   word.count('guarantees'), "guaranteed:", word.count('guaranteed'))
for title,word in zip(file,token_joined):
    print(title['title'],"service:", word.count('service'),"services:", word.count('services'))<jupyter_output>Transcript, Hearing April 11, 2016  service: 210 services: 103
Transcript, Hearing April 12, 2016  service: 166 services: 74
Transcript, Hearing April 13, 2016  service: 98 services: 49
Transcript, Hearing April 14, 2016  service: 98 services: 31
Transcript, Hearing April 15, 2016  service: 76 services: 31
Transcript, Hearing April 18, 2016  service: 215 services: 79
Transcript, Hearing April 19, 2016  service: 90 services: 46
Transcript, Hearing April 20, 2016  service: 118 services: 60
Transcript, Hearing April 21, 2016  service: 137 services: 52
Transcript, Hearing April 22, 2016  service: 30 services: 30
Transcript, Hearing April 25, 2016  service: 208 services: 60
Transcript, Hearing April 26, 2016  service: 157 services: 60
Transcript, Hearing April 27, 2016  service: 62 services: 28
Transcript, Hearing April 28, 2016  service: 34 services: 20<jupyter_text>ConcordanceIt looks like record number 5 has the most occurences of the word `guarantee`. The code below isolates the record and creates a concordance based on the selected word.<jupyter_code># splits the text from the record into a list of individual words
words = joined_text[0].split()
#assigns NLTK functionality to the text
text = nltk.Text(words)
# prints a concordance output for the selected word (shown in green)
print(text.concordance('services', lines=25))
#creates a new file that can be written by the print queue
fileconcord = codecs.open('April11_service_concord.txt', 'w', 'utf-8')
#makes a copy of the empty print queue, so that we can return to it at the end of the function
tmpout = sys.stdout
#stores the text in the print queue
sys.stdout = fileconcord
#generates and prints the concordance, the number pertains to the total number of bytes per line
text.concordance("service", 79, sys.maxsize)
#closes the file
fileconcord.close()
#returns the print queue to an empty state
sys.stdout = tmpout<jupyter_output><empty_output><jupyter_text>Below is what the text looks like after the initial processing, without punctuation, numbers, or stopwords.<jupyter_code># shows the text list for a given record. Changing the first number displays a... 
# ...different record, changing/removing the second number limits/expands the text shown
print(token_joined[5][:50])<jupyter_output><empty_output><jupyter_text>LemmatizationSome more preparation for the text processing. The code below works on the all of the records, creating one master list of words which is then lemmatized.<jupyter_code># creates a variable for the lemmatizing function
wnl = WordNetLemmatizer()

# lemmatizes all of the verbs

lemm = []
for record in token_joined:
        for word in record:
            lemm.append(wnl.lemmatize(word, 'v'))
'''
lemm = []
for word in token_joined[13]:
        lemm.append(wnl.lemmatize(word, 'v'))
'''

# lemmatizes all of the nouns 
lems = []
for word in lemm:
    lems.append(wnl.lemmatize(word, 'n'))<jupyter_output><empty_output><jupyter_text>Here we are checking to make sure the lemmatizer has worked. Now the word `guarantee` only appears in one form.<jupyter_code># just making sure the lemmatizer has worked
#print("guarantee:", lems.count('guarantee'), "guarantees:", \
         # lems.count('guarantees'), "guaranteed:", lems.count('guaranteed'))
print("service:", lems.count('service'), lems.count('services'))<jupyter_output>service: 2435 0<jupyter_text>Word FrequencyHere is a count of the number of words in each record. While this data isn't terribly useful 'as is', we can make a few assumptions about the text here. Notably that some of the hearings were much longer than others.<jupyter_code># counting the number of words in each record 
for name, each in zip(file,token_joined):
    print(name['title'], ":",len(each), "words")<jupyter_output>Transcript, Hearing April 11, 2016  : 16664 words
Transcript, Hearing April 12, 2016  : 12891 words
Transcript, Hearing April 13, 2016  : 8423 words
Transcript, Hearing April 14, 2016  : 8319 words
Transcript, Hearing April 15, 2016  : 4840 words
Transcript, Hearing April 18, 2016  : 13523 words
Transcript, Hearing April 19, 2016  : 10184 words
Transcript, Hearing April 20, 2016  : 8541 words
Transcript, Hearing April 21, 2016  : 12865 words
Transcript, Hearing April 22, 2016  : 3400 words
Transcript, Hearing April 25, 2016  : 14791 words
Transcript, Hearing April 26, 2016  : 11804 words
Transcript, Hearing April 27, 2016  : 7454 words
Transcript, Hearing April 28, 2016  : 6803 words<jupyter_text>Here we will count the five most common words in each record.<jupyter_code>docfreq = []
for words in token_joined:
    docfreq.append(nltk.FreqDist(words))
for name, words in zip(file_obj, docfreq):
    print(name['title'], ":", words.most_common(5))<jupyter_output><empty_output><jupyter_text>These are the 10 most common word pairs in the text.<jupyter_code># prints the 10 most common bigrams
colText = nltk.Text(lems)
colText.collocations(10)<jupyter_output>service provider; market force; digital literacy; basic service; data
cap; eastern ontario; fix wireless; private sector; low income; rural
remote<jupyter_text>Error checking to make sure the code is processing the text properly.<jupyter_code># creates a list of bigrams (ngrams of 2), printing the first 5
colBigrams = list(nltk.ngrams(colText, 2)) 
colBigrams[:5]<jupyter_output><empty_output><jupyter_text>More error checking.<jupyter_code># error checking. There should be one less bigram than total words
print("Number of words:", len(lems))
print("Number of bigrams:", len(colBigrams))<jupyter_output><empty_output><jupyter_text>Below is a frequency plot showing the occurence of the 25 most frequent words.<jupyter_code># frequency plot with stopwords removed
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 10.0)
fd = nltk.FreqDist(colText)
fd.plot(25)<jupyter_output><empty_output><jupyter_text>CollocationsHere we are preparing the text to search for bigrams containing the word `guarantee`. This code searches for words appearing before and after `guarantee` with a window size of two words on either side.<jupyter_code># loads bigram code from NLTK
bigram_measures = nltk.collocations.BigramAssocMeasures()
# bigrams with a window size of 2 words
finder = BigramCollocationFinder.from_words(lems, window_size = 2)
# ngrams with 'word of interest' as a member
word_filter = lambda *w: 'service' not in w
# only bigrams that contain the 'word of interest'
finder.apply_ngram_filter(word_filter)
# filter results based on statistical test

# calulates the raw frequency as an actual number and percentage of total words
act = finder.ngram_fd.items()
raw = finder.score_ngrams(bigram_measures.raw_freq)
# log-likelihood ratio
log = finder.score_ngrams(bigram_measures.likelihood_ratio)<jupyter_output><empty_output><jupyter_text>Research shows that this is the most reliable statistical test for unreliable data. **Log-Likelihood Ratio**The Log-likelihood ratio calculates the size and significance between the observed and expected frequencies of bigrams and assigns a score based on the result, taking into account the overall size of the corpus. The larger the difference between the observed and expected, the higher the score, and the more statistically significant the collocate is.The Log-likelihood ratio is my preferred test for collocates because it does not rely on a normal distribution, and for this reason, it can account for sparse or low frequency bigrams. It does not over-represent low frequency bigrams with inflated scores, as the test is only reporting how much more likely it is that the frequencies are different than they are the same. The drawback to the Log-likelihood ratio is that it cannot be used to compare scores across corpora. An important note here that **words will appear twice** in the following list. As the ngrams can appear both before and after the word, care must be taken to identify duplicate occurences in the list below and then combine the totals.<jupyter_code># prints list of results. 
print(tabulate(log, headers = ["Collocate", "Log-Likelihood"], floatfmt=".3f", \
               numalign="left"))
# prints list of results. 
print(tabulate(act, headers = ["Collocate", "Actual"], floatfmt=".3f", \
               numalign="left"))
with open('digital-literacy_collocate_Act.csv','w') as f:
    w = csv.writer(f)
    w.writerows(act)<jupyter_output><empty_output><jupyter_text>Here is an example of words appearing twice. Below are both instances of the ngram 'quality'. The first instance appears before 'guarantee' and the second occurs after.<jupyter_code>Collocate                     Log-Likelihood
----------------------------  ----------------
('quality', 'guarantee')      76.826
('guarantee', 'quality')      3.955<jupyter_output><empty_output><jupyter_text>A bit more processing to clean up the list.<jupyter_code>##################################################################
############### sorts list of log-likelihood scores ##############
##################################################################

# group bigrams by first and second word in bigram                                        
prefix_keys = collections.defaultdict(list)
for key, l in log:
    # first word
    prefix_keys[key[0]].append((key[1], l))
    # second word
    prefix_keys[key[1]].append((key[0], l))
    
# sort bigrams by strongest association                                  
for key in prefix_keys:
    prefix_keys[key].sort(key = lambda x: -x[1])

    # prints top 80 results
logkeys = prefix_keys['service'][:80]<jupyter_output><empty_output><jupyter_text>Here is a list showing **only** the collocates for the word `guarantee`. Again, watch for duplicate words below.<jupyter_code>from tabulate import tabulate
print(tabulate(logkeys, headers = ["Collocate", "Log-Likelihood"], floatfmt=".3f", \
               numalign="left"))
with open('service_collocate_Log.csv','w') as f:
    w = csv.writer(f)
    w.writerows(logkeys)<jupyter_output><empty_output><jupyter_text>---<jupyter_code># working on a regex to split text by speaker
diced = []
for words in joined_text:
    diced.append(re.split('(\d+(\s)\w+[A-Z](\s|.\s)\w+[A-Z]:\s)', words))
print(diced[8])
init_names = []
for words in joined_text:
    init_names.append(set(re.findall('[A-Z]{3,}', words)))
print(init_names)
with open('initialNames.csv','w') as f:
    w = csv.writer(f)
    w.writerows(init_names)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10097431355181577, 'fraction_numerical': 0.02541391292498467, 'mean_word_length': 4.954563894523327, 'pattern_counts': {'":': 3, '<': 92, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '353823', 'license': 'MIT', 'n_tokens_mistral': 4696, 'n_tokens_neox': 4227, 'n_words': 1700, 'path': '_code/notebooks/CRTC-Hearing-TextAnalysis.ipynb', 'repo_name': 'telecom-research/crtc-scraper', 'size': '14677'}