<jupyter_start><jupyter_text>Parsing Text and the LDA output  a.1) Opening pdfs and extracting their text Under the material for Lecture 3 I have added a folder called FOMC_pdf. This folder contains the transcripts of all the meetings that took place during the [Greenspan](https://en.wikipedia.org/wiki/Alan_Greenspan) era (August 11, 1987 to January 31st, 2006). We are some lines of code to parse those pdfs.<jupyter_code>#load operating system module

import os<jupyter_output><empty_output><jupyter_text>This module is used to conduct operating system like tasks (such as opening a file, or listing the contents of a directory).<jupyter_code>#Define the base directory containing the FOMC statements

base_directory = "../../../../collection/python/data/transcript_raw_text"
#Return a list containing the name of the files in the directory

raw_doc        = os.listdir(base_directory)
#Sort the list in ascending order

filelist       = sorted(raw_doc)[:-1]
filelist<jupyter_output><empty_output><jupyter_text>To parse the text in the pdfs I will use the PyPDF2 module (but there are other ways to do it. See for example the Tika module). Warning: Depending on your Python configuration, you might not be able to use PyPDF2 directly. I downloaded Python using the Anaconda distribution and I needed to type directly on the terminalconda -forge pypdf2After doing this, I was able to upload the PyPDF2 module  a.2) Organizing the information in a data frame<jupyter_code>import pandas as pd
#load re to split the content of the pdfs by the occurrence of a pattern

import re
#Creates a data frame containing the date|s of the FOMC meetings

date        = pd.Series(data=filelist).apply(lambda x: x[0:10])
print(date)
documents = []
for doc in filelist:
    with open("{}/{}".format(base_directory,doc),"r") as f:
        documents += f.read()
for i in range(len(document)):

    interjections    = re.split('MR. |MS. |CHAIRMAN |VICE CHAIRMAN ', document[i]) 
                        #Split the doc by interjections
    
    temp_df          = pd.DataFrame(columns=['Date','Speaker','content'],index=range(len(interjections)))          
                        #Temporary data frame
          
    for j in range(len(interjections)):
                                    
    
        interjection           = interjections[j].replace('\n',' ') 
                                 #Replace page break (\n) with space
        
        temp_df['Date'].loc[j]    = date[i]
           
        temp_df['Speaker'].loc[j] = interjection.split('.')[0]
        
        temp_df['content'].loc[j] = ''.join(interjection.split('.')[1:])
        
    parsed_text = pd.concat([parsed_text,temp_df],ignore_index=True)                     
            
parsed_text<jupyter_output><empty_output><jupyter_text>We will focus only on Greenspan's interjections.<jupyter_code>Greenspan_text = parsed_text.loc[parsed_text['Speaker'] == 'GREENSPAN']
Greenspan_text.index = range(sum(parsed_text['Speaker'] == 'GREENSPAN'))
Greenspan_text<jupyter_output><empty_output><jupyter_text>a.3) Bag of Words<jupyter_code>Greenspan_corpus = list(Greenspan_text['content'])
len(Greenspan_corpus)
Greenspan_corpus[0]
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
term_doc_matrix = vectorizer.fit_transform(Greenspan_corpus[0:1]).todense()
vectorizer.get_feature_names()
term_doc_matrix<jupyter_output><empty_output><jupyter_text>a.4) Cloud of words If you got the Anaconda distribution of Python go the terminal and typeconda install -c conda-forge wordcloud Here is a good tutorial on how to generate words of clouds:https://www.datacamp.com/community/tutorials/wordcloud-python<jupyter_code>from wordcloud import WordCloud

wordcloud = WordCloud(background_color='white', font_step = 3, stopwords='None', relative_scaling=1).generate(Greenspan_text['content'].loc[0])

Greenspan_text['content'].loc[0]
Greenspan_text.content[0]<jupyter_output><empty_output><jupyter_text>Display the generated image<jupyter_code>import matplotlib.pyplot as plt
plt.imshow(wordcloud, interpolation='bilinear')

plt.axis("off")

plt.show()<jupyter_output><empty_output><jupyter_text>Let's do it for the whole Greenspan interjections<jupyter_code>text_aux = " ".join(interjection for interjection in Greenspan_text.content)
len(text_aux)
wordcloudG = WordCloud(background_color='white', font_step = 3, relative_scaling=1).generate(text_aux)
plt.imshow(wordcloudG, interpolation='bilinear')

plt.axis("off")

plt.show()<jupyter_output><empty_output><jupyter_text>a.5) LDA  Tokenize<jupyter_code>import nltk
from nltk.tokenize import RegexpTokenizer

tokenizer = RegexpTokenizer(r'\w+')
tokens_example = tokenizer.tokenize(Greenspan_text['content'].loc[0])
Greenspan_text['content'].loc[0]
tokens_example<jupyter_output><empty_output><jupyter_text>Remove Stop Words<jupyter_code>from nltk.corpus import stopwords

nltk.download('stopwords')

stopwords = stopwords.words('english')
len(stopwords)
stopped_tokens = [i for i in tokens_example if not i in stopwords]
stopped_tokens<jupyter_output><empty_output><jupyter_text>Stems<jupyter_code>from nltk.stem.porter import PorterStemmer

# Create p_stemmer of class PorterStemmer
p_stemmer = PorterStemmer()
p_stemmer.*?
texts_G = [p_stemmer.stem(i) for i in stopped_tokens]
texts_G<jupyter_output><empty_output><jupyter_text>Loop to Tokenize, Remove Stop Words, Stem<jupyter_code>texts = []
for i in range(0,len(Greenspan_text['content'])):

    tokens         = tokenizer.tokenize(Greenspan_text['content'].loc[i])
    
    stopped_tokens = [j for j in tokens if not j in stopwords]
    
    texts.append([p_stemmer.stem(j) for j in stopped_tokens])
    
Greenspan_text['content'].loc[1]
texts
len(Greenspan_text['content'])<jupyter_output><empty_output><jupyter_text>LDA conda install -c anaconda gensim<jupyter_code>import gensim
from gensim import corpora, models

dictionary = corpora.Dictionary(texts)
dictionary.*?
corpus = [dictionary.doc2bow(text) for text in texts]
corpus[2]
ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=2, id2word = dictionary, passes=20)
for t in range(2):

    plt.figure()
    
    plt.imshow(WordCloud(background_color='white').fit_words(dict(ldamodel.show_topic(t,200))),interpolation='bilinear')
    
    plt.axis('off')
    
    plt.show<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10541220893643802, 'fraction_numerical': 0.008181246066708621, 'mean_word_length': 4.8589861751152075, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '4513', 'license': 'MIT', 'n_tokens_mistral': 2135, 'n_tokens_neox': 2057, 'n_words': 504, 'path': 'src/analysis/python/scripts/indirect/.ipynb_checkpoints/parse_transcripts-checkpoint.ipynb', 'repo_name': 'jm4474/FOMCTextAnalysis', 'size': '6356'}