<jupyter_start><jupyter_text>Basic Keywords Search - it is an simple example of how to search through document, find keyworkds, and get the paragraph context. - we will use all Staff Reports from 2000 - 2016. Most data came from COM xml database, but I also patched some missing years with IR documents. That is why you will see we are going to process two different data srouce in the code- a metadata sheet is also available to match all our documents with proper metadat info.<jupyter_code>### import some modules we gonna use 
import os 
python_root = '..'
import sys
sys.path.insert(0, python_root)
import data_util
import pickle
import csv
import re
from collections import Counter<jupyter_output><empty_output><jupyter_text>- make sure you have pandas and nltk installed<jupyter_code>import pandas as pd
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer<jupyter_output><empty_output><jupyter_text>Donload preprocessed data  - In 1_process_xmls, we have shown one way of processing COM's xml data base. Here, I have already compiled a small database, with all article IV document i can possible find from 1990 to 2016.  - All documents has been preprocessed into a document object that we defined earlier in 1_process_xmls, detailed steps can be found in ../data_util. - From the link, you can download all preprocessed data and also raw data in case you want to process them in a different way.<jupyter_code>### specify download path and extract path 
download_path = "staff_reports_full_sample.zip"
download_link = "https://www.dropbox.com/s/23vdujvh67io2iz/staff_reports_full_sample.zip?dl=1"
extract_path = "./data"  # place data in Python project root folder
## detailed of the download_data function is in data_util module in python_root 
## if you do not yet have the data, run this code, it will set up a data folder under ./Python folder 
data_util.download_data(download_path,download_link,extract_path)<jupyter_output>article iv: 589MB [02:31, 3.87MB/s]<jupyter_text>1. Search - Load xml documents and txt documents from pickle<jupyter_code>## laod xml_dict and txt_dict 
xml_dict = pickle.load(open('data/xlm_docs.p', "rb")) 
txt_dict = pickle.load(open('data/txt_docs.p', "rb")) 

## merge them together 
# if you are running python 3.5 or later, you can use this one line to merge them 
doc_dict = {**xml_dict,**txt_dict}

## print some information 
doc_ids = list(doc_dict.keys())
print("number of documents: ", len(doc_dict))
print("\nOne example: \n\ndoc_id: {} \n\nFirst paragraph: \n{}".format(doc_ids[0],doc_dict[doc_ids[0]].paras[0]))<jupyter_output>number of documents:  2960

One example: 

doc_id: 9781451821079 

First paragraph: 
1. At the conclusion of the last Article IV consultation in March 2002 (SUR/02/33, 3/19/02), Directors commended Kenya for achieving a measure of macroeconomic stability during recent years in difficult circumstances. Directors, however, were concerned that the macroeconomic and financial situation remained fragile, and that investor confidence was low. They stressed the importance of implementing a comprehensive medium-term economic and structural reform program and undertaking measures to address the governance problems that had stalled progress to date. Directors stressed that it was important for Kenya to implement the prior actions needed to resume the Poverty Reduction and Growth Facility (PRGF)-supported program and to help restore confidence.1<jupyter_text>- get keywords, i am reading it from a txt file. Of course you can simply define it as a list in your script<jupyter_code>### get keyword list 
def read_keywords(file):
    """
    file: csv file with keyword list
    """
    with open(file,'r') as f:
        reader = csv.reader(f)
        mylist = list(reader)
    return mylist

key_file = 'search_keywords.csv'
keywords = read_keywords(key_file)
keywords = [k[0].replace('_',' ') for k in keywords] ## in keywords we used _ as [space], change it back 
print('Keywords: ', keywords[:10])<jupyter_output>Keywords:  ['asset quality', 'bad asset', 'bad debt', 'bad loan', 'balance sheet mismatch', 'balance-sheet mismatch', 'bank profitability', 'bankruptcies', 'bankruptcy', 'bond spread']<jupyter_text>- Use regular expression to find keywords in your text - regular expression can get really complicated, we may have a stand alone tutorial for regular expression. - Here what we are trying to achieve is to feed in a list of keywords, and return a list of keywords found, and also the frequency<jupyter_code>## define a function to loacte keywords
def construct_rex(keywords):
    r_keywords = [r'\b' + re.escape(k) + r'(s|es)?\b'for k in keywords]    # tronsform keyWords list to a patten list, find both s and es 
    rex = re.compile('|'.join(r_keywords),flags=re.I)                        # use or to join all of them, ignore casing
    #match = [(m.start(),m.group()) for m in rex.finditer(content)]          # get the position and the word
    return rex

def find_exact_keywords(content,keywords,rex=None):
    if rex is None: 
        rex = construct_rex(keywords)
    content = content.replace('\n', '').replace('\r', '')#.replace('.',' .')
    match = Counter([m.group() for m in rex.finditer(content)])             # get all instances of matched words 
                                                                             # and turned them into a counter object, to see frequencies
    return match<jupyter_output><empty_output><jupyter_text>- Let's see how ti works<jupyter_code>content = doc_dict[doc_ids[100]].paras[9]
print("Content: \n{}".format(content))
rex = construct_rex(keywords)
results = find_exact_keywords(content.lower(),keywords,rex)
print("\nkeywords found: \n {}".format(results))<jupyter_output>Content: 
10. Remarkable progress has been made in consolidating the peace process culminating in the agreement signed in Pretoria on December 17,2002. In line with the Lusaka accords of 1999, peace agreements were signed with Rwanda (end-July 2002) and Uganda (early September 2002), and these countries have all but completed the withdrawal of their troops. On the government side, Angola is also completing the withdrawal of its troops, while Namibia and Zimbabwe have already done so. On November 11, 2002, Presidents Kabila and Kagame agreed to extend the period envisaged in the original peace agreement by three months to allow for the disarming and repatriation of ex-Rwandan Hutu soldiers. Meanwhile, Phase III of the United Nations Observation Mission (MONUC) to the DRC continues and the regional demobilization and reintegration program is gradually being put in place, with the help of the United Nations (UN) and the World Bank. On December 5, 2002, the UN Security Council passed Resol[...]<jupyter_text>Now, let's loop through all document we have and put all keywords frequencies together- depends on how many documents you have, this may take a while - you are more then welcome to modify this to seep it up. Depends on your need, if all you care about is a document level statistics, you can concatinate all paragraphs and do one regex search, it will be much faster. - We are doing it in paragaph level, just in case you will need paragraph context in the future. - you can also try multi thread it, if time is really precious to you. You can find some usefuly information here: https://pymotw.com/2/multiprocessing/basics.html<jupyter_code>total_doc_num = len(doc_dict.items())
df = pd.DataFrame()               ## define a empty data frame, will populat it as we loop though documents
rex = construct_rex(keywords)     ## construct regular expression 

for idx,(ite) in enumerate(doc_dict.items()):
    #if idx > 500: break
    key,doc = ite     ## get id and doc
    val = doc.paras   ## read all paragraphs 
    if len(val)==0:   ## skip if document has no paragraph 
        continue
    for idxp,content in enumerate(val):                                    ## loop though each paragaph in a document 
        results = find_exact_keywords(content.lower(),keywords,rex)        ## get keywords frequency in each paragraph 
        if len(results) == 0:                                              ## if nothing found, skip 
            continue
        results['context'] = content                                    ## if keyword found, assign context to be the paragaph
        results['doc_id'] = str(key)                                    ## assign documetn id 
        results['para_id'] = str(key) + '_' + str(idxp)                 ## assigin paragaph id, we may not use this at all, but just in case
        results['para_word_count'] = len(content.split())               ## get paragaph word count 
        df = df.append(results, ignore_index=True)
    
    if idx%100 == 0 :                                     ## print every 100 iterations 
        print('{}/{}'.format(idx,total_doc_num))
## order doc id and context to first 3 columns 
keys = df.columns
meta = ['doc_id','para_id','context','para_word_count']
keys = [i for i in keys if i not in meta]
meta.extend(keys)
df = df[meta]
df.head()
### save results to csv 
df.to_csv('data/search_results.csv',encoding='utf-8')
print('finished')<jupyter_output>finished
{'fraction_non_alphanumeric': 0.08277429127894395, 'fraction_numerical': 0.013633412681237828, 'mean_word_length': 3.7448665297741273, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '239483', 'license': 'MIT', 'n_tokens_mistral': 2699, 'n_tokens_neox': 2493, 'n_words': 1100, 'path': 'Python/3_basic_search/1_search_keywords.ipynb', 'repo_name': 'johnsonice/IMF_Textmining', 'size': '9242'}