<jupyter_start><jupyter_text>Script for preprocessing tweets for bi-LSTM training **Author:** [<NAME>](https://www.linkedin.com/in/andrew-larkin-525ba3b5/) **Affiliation:** [Oregon State University, College of Public Health and Human Sciences](https://health.oregonstate.edu/) **Date Created:** January 7, 2019 **Summary** Add a regional context indicator variable for words that have regional meaning.  The regional context indicator was added after datasets were preprocssed, which is why these functions are in a separate script from the PreprocessTweets notebook.  Regional indicators are the word indices within a tweet where a key reigonal indicator phrase is present  setup: import packages, define filepaths and global constants<jupyter_code>import pandas as ps
import pickle
import numpy as np
# define input and output filepaths
parentFolder = "C:/Users/larkinan/Desktop/DBTraining/"
dataset = parentFolder + "preprocessingOutput/"
performFolder = parentFolder + "modelTrainingPerformance/"
# pickled datasets to load
datasetPickleParams = { 
    "trainDictPicklePath":dataset + "trainDict.p",
    "devDictPicklePath":dataset + "devDict.p",
    "testDictPicklePath":dataset + "testDict.p",
    "allDictPicklePath":dataset + "allDict.p",
    "embeddingMatrixPicklePath":dataset + "embeddingMatrix.p",
    "word2IndexPicklePath":dataset + "word2Index.p",
    "NYC_DictPicklePath":dataset + "NYCDict.p"
}
# words in Portland that have regionally unique context
keyWordsPDX = ['Enchanted Forest','Splash Mountain','Bull Mountain','Sun River','Lake Oswego','Menlo Park','Forest Heights','Salt Lake',
               'Fire on the Mountain','Lake Tahoe','Garden State','CenturyLink Field','Mountain View','Hood River','Cooper Mountain',
               'Highland Park','Columbia River','Meridian Park','Merlo Park','Space Mountain','ATandT Park','Maywood Park','Fenway Park',
               'Coors Field']

# tweets in NYC that have reigonally unique context
keyWordsNYC = ['Sunset Park','Belmont Park','Forest Hills','Garden City','the Garden','Borough Park',
               'Mad Square Garden','Madsquaregarden', 'square garden','Massapequa Park','Citi Field',
               'Lone Star Park','Deer park','Pearl River', 'Kings Park','Floral Park','Ozone Park',
               'rego park', 'rockaway park','Hyde Park','Fenway Park','PNC Park','Marine Park','Harlem River',
               'Hutchinson River','Lake George','Park Slope','Massapequa Park','Bronx River','Nats Park',
               'Mountain View','Nationals Park','Salt Lake','Madison Sq','Florham Park','Morris Park',
               'Forest Park','Finsbury Park','Hershey Park','Silver Lake' ]

padKey = "<KEY> "
# load pickled preprocessed data
def loadDatasets(pickleParams):
    trainDict = pickle.load(open(pickleParams['trainDictPicklePath'],'rb'))
    devDict = pickle.load(open(pickleParams['devDictPicklePath'],'rb'))
    testDict = pickle.load(open(pickleParams['testDictPicklePath'],'rb'))
    NYC_Dict = pickle.load(open(pickleParams['NYC_DictPicklePath'],'rb'))
    embeddingMatrix = pickle.load(open(pickleParams['embeddingMatrixPicklePath'],'rb'))
    word2IndexMap = pickle.load(open(pickleParams['word2IndexPicklePath'],'rb'))
    return(trainDict,devDict,testDict,NYC_Dict,embeddingMatrix,word2IndexMap)
# test whether substring variable is a subset of the wholestring variable.  Used in a map-reduce function
def testForSubstring(wholeString,subString):
    if(subString.lower() in wholeString):
        return subString
    return("None")
# replace instances of keyWords in the wholeString with the padKey string
def replaceKeywords(wholeString,keyWords,padKey):
    newString = wholeString.lower()
    for word in keyWords:
        if word.lower() in newString:
            wordLength = len(word.split())
            padding = padKey*wordLength
            startIndex = newString.find(word.lower())
            endIndex = newString.find(" ",startIndex+len(word))
            stringToReplace = newString[startIndex:endIndex]
            newString = newString.replace(stringToReplace,padding[:-1])
    return(newString)
def compareWords(padKey,compareWord):
    if(padKey == compareWord):
        return 1
    return 0<jupyter_output><empty_output><jupyter_text>identify regional indicators for one tweet **Inputs**: - **wholeString** (string) - text of single tweet to screen for regional indicators - **keyWords** (string array) - phrases which if present within wholeString should be flagged with a positive indicator variable - **padKey** (string) - intermediate used to identify where within a tweet regional phrases are present **Outputs**: - **loc_indicator** (int array) - series of regional indicators, where each index flags whether the word at index 1 belongs to a regional key phrase or not<jupyter_code># add location indicator to a single 
def createSingleLocInd(wholeString,keyWords,padKey):
    codedString = replaceKeywords(wholeString,keyWords,padKey)
    parsedCode = codedString.split()
    padKeyArray = [padKey[:-1]]*len(parsedCode)
    locResults = list(map(compareWords,parsedCode,padKeyArray))
    loc_indicator = np.asarray(locResults)
    return(loc_indicator)<jupyter_output><empty_output><jupyter_text>create regional indicators for all tweets in a single dataset **Inputs**: - **textArray** (string array) - all texts in the dataset - **keyWords** (string array) - key phrases that are regional indciators.  Indices of the words in these phrases are where regional indicator flags should be positive - **padKey** (string) - intermediate to communicate where regional indicators where indentified within a text **Outputs**:- **padResults** (array of int arrays) - the ith int array corresponds to the reiongal indicator flags for all words in the ith text string<jupyter_code>def createAllLocInd(textArray,keyWords,padKey):
    keyWordMap = [keyWords]*len(textArray)
    padKeyMap = [padKey]*len(textArray)
    padResults = list(map(createSingleLocInd,textArray,keyWordMap,padKeyMap))
    return(padResults)<jupyter_output><empty_output><jupyter_text>add regional indicator to multiple datasets  **Inputs**: - **savePaths** (string array) - absolute filepaths where modified datasets should be stored - **dictArray** (dict array) - input dictionaries to add regional indicator variable to - **keyWords** (string array) - keyStrings that are positive indicators - **padKey** (string) - intermediate product - used to identify where in a text keyWords occur and positive regional indicators should be written<jupyter_code># add regional location indicator to datasets 
def addLocToAllDatasets(savePaths,dictArray,keyWords,padKey):

    #testString = "I'm testing to see if Fire on the mountain and merlo Park are picked up by the map function"

    index=0
    for tempDict in dictArray:
        tempDictLocInd = createAllLocInd(tempDict['sent'],keyWords,padKey)
        tempDict['loc_ind'] = tempDictLocInd
        pickle.dump(tempDict,open(savePaths[index],"wb" ))
        index+=1
def main():
    trainDict, devDict, testDict, NYC_Dict,embeddingMatrix, word2IndexMap = loadDatasets(datasetPickleParams)
    PDXDicts = [trainDict,devDict,testDict]
    PDXSavePaths = [
        parentFolder + 'trainDictv2t.p',
        parentFolder + 'devDictv2t.p',
        parentFolder + 'testDictv2t.p'
    ]
    addLocToAllDatasets(PDXSavePaths,PDXDicts,keyWordsPDX,padKey)
    addLocToAllDatasets([parentFolder + 'NYCDictv2t.p'],[NYC_Dict],keyWordsNYC,padKey)
main()<jupyter_output>dict_keys(['x', 'y', 'seqlens', 'hash', 'emot', 'loc_ind', 'sent', 'labels', 'seqLens'])
dict_keys(['x', 'y', 'seqlens', 'hash', 'emot', 'loc_ind', 'sent', 'labels', 'seqLens'])
dict_keys(['x', 'y', 'seqlens', 'hash', 'emot', 'loc_ind', 'sent', 'labels', 'seqLens'])
dict_keys(['labels', 'seqLens', 'sent', 'hash', 'emot', 'loc_ind'])
{'fraction_non_alphanumeric': 0.11911463664278403, 'fraction_numerical': 0.0034544524053224156, 'mean_word_length': 5.140612725844462, 'pattern_counts': {'":': 7, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '99446', 'license': 'MIT', 'n_tokens_mistral': 2339, 'n_tokens_neox': 2250, 'n_words': 645, 'path': 'DataPreprocessing/AddRegionalContextIndicator.ipynb', 'repo_name': 'larkinandy/GreenTweet_MultipleBiLSTM', 'size': '7836'}