<jupyter_start><jupyter_text>Analyze Baltimore Riot Twitter data This process takes a subset of Twitter data related to the Baltimore Riots and conducts analysis on the tweet text to determine if the areas most affected by the Riots can be recreated.  This analysis uses a combination of a trained Long Short Term Memory (LSTM) Recurrent Neural Network (RNN) and Natural Language Processing Techniques.  The first step is to import the necessary Python Packages.<jupyter_code>import os 
import json
import requests
import textblob
import time
import tweepy
import re

import numpy as np
import pandas as pd
import keras.preprocessing.text as kpt
import matplotlib.pyplot as plt

from keras.models import load_model
from arcgis.gis import GIS
from arcgis.features import SpatialDataFrame
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from textblob.sentiments import NaiveBayesAnalyzer
from textblob.np_extractors import ConllExtractor
from textblob import TextBlob
from collections import Counter
from operator import itemgetter
csv_path = r'Baltimore Riots Tweets/baltimore_twitter.csv'
model = load_model('models/Twitter_SA_Model.h5')<jupyter_output><empty_output><jupyter_text>The vocabulary that was used to train and test the model from the previous steps must be loaded into memory.  Following this, a new Tokenizer must be instantiated that has the length as the vocabulary.  Once the Tokenizer has been created, a function was created to convert the text to an index array.  This populates the new dictionary that will be used in the prediction of the sentiment of the tweet based on the Keras Model.<jupyter_code>with open('dictionary/dictionary.json', 'r') as dictionary_file:
    dictionary = json.load(dictionary_file)
tokenizer = Tokenizer(num_words=20000)
def convert_text_to_index_array(text):
    words = kpt.text_to_word_sequence(text)
    wordIndices = []
    for word in words:
        if word in dictionary:
            wordIndices.append(dictionary[word])
        else:
            pass
    return wordIndices<jupyter_output><empty_output><jupyter_text>A Sentiment function must be created that takes as an input the text of the tweet, calls the function that turns the text into an array, and predicts the sentiment based on the Keras trained model.<jupyter_code>def Sentiment(tweet_text):
    labels = ['positive', 'negative']
    testArr = convert_text_to_index_array(tweet_text)
    twt = tokenizer.sequences_to_matrix([testArr], mode='binary')
    twt = pad_sequences(twt, maxlen=86, dtype='int32', padding='post', truncating='post', value=0)
    sentiment = model.predict(twt)
    accuracy = sentiment[0][np.argmax(sentiment)] * 100
    tweetSent = labels[np.argmax(sentiment)]
    return tweetSent, accuracy<jupyter_output><empty_output><jupyter_text>Other measures of sentiment will be calculated based off the TextBlob Naïve Bayes analyzer and the Polarity and Subjectivity calculator.  To calculate these measures, the tweet text must be instantiated as a TextBlob object, or blob.  This blob can then be used in the various classifiers and analyzers associated with TextBlob.  Extracting Noun-phrases and verbs from Baltimore Riot Twitter Dataset:  TextBlob provides a built-in function that allows for a user to call a part of sentence tagger.  This in turn returns a list of tuples of the word and type of word (noun, verb, etc…).<jupyter_code>def calculate_sentiment(object_id, text_to_analyze):
    sent_sp = TextBlob(text_to_analyze)
    #nba_sent = TextBlob(text_to_analyze, analyzer=NaiveBayesAnalyzer())
    subjectivity = sent_sp.sentiment.subjectivity
    polarity = sent_sp.sentiment.polarity
    sentiment_tf = Sentiment(text_to_analyze)
    classification_tf = sentiment_tf[0]
    if classification_tf == 'positive':
        classification_num = 1
    else:
        classification_num = 0
    accuracy_tf = sentiment_tf[1]
    nba_sentiment = ""
    nouns = []
    verbs = []
    for part in sent_sp.tags:
        if part[1].startswith("V"):
            verbs.append(part[0])
        elif part[1].startswith("N"):
            nouns.append(part[0])
        
    for noun in sent_sp.noun_phrases:
        if noun not in nouns:
            nouns.append(noun)
    
    return object_id, subjectivity, polarity, classification_tf, accuracy_tf, classification_num, nouns, verbs, nba_sentiment<jupyter_output><empty_output><jupyter_text>To correctly calculate the sentiment of each individual tweet, the Baltimore Riots Twitter dataset must be loaded into memory using a Pandas Dataframe.  Then an iterator is created to go through the various rows of the dataset, passing the Tweet text to the various functions that have been outlined above.  Once these classifiers conduct their analysis, these values are loaded into an in-memory list, that in turn is used to create a new Pandas Dataframe with the Tweet text and associated sentiment.  This dataset is then in turn appended to the original Baltimore Riots Twitter dataset.<jupyter_code>df = pd.read_csv(csv_path)
df.dropna()
text_df = df[['OBJECTID', 'text']]
df.head()<jupyter_output><empty_output><jupyter_text>Regular expressions are used to go through the body of text and extract out all Twitter user names and hashtags that are located within the Tweet text.<jupyter_code>mentions = []
mentions_dict = {}
hashtags = {}

for row in df.iterrows():
    oid = row[1]['OBJECTID']
    text = row[1]['text']
    user = row[1]['user_name']
    match = re.findall(r'@(?i)[a-z0-9_]+', text)
    if len(match) > 0:
        mentions_dict[oid] = match
        for handle in match:
            mentions.append(["@" + user, handle])
    hash_match = re.findall(r'#(?i)[a-z0-9_]+', text)
    if len(hash_match) > 0:
        hashtags[oid] = hash_match
print(len(mentions))
print(len(hashtags))
mentions_columns = ['User', 'Mentioned']
mentions_df = pd.DataFrame(mentions, columns=mentions_columns)
mentions_df.head()
mentions_dict
mentions_df.to_csv('output/TwitterMentions.csv')<jupyter_output><empty_output><jupyter_text>Map the Mentions and Hashtags back to the original dataframe.<jupyter_code>df['Mentions'] = df['OBJECTID'].map(mentions_dict)
df['Hashtags'] = df['OBJECTID'].map(hashtags)
df.head(10)<jupyter_output><empty_output><jupyter_text>Apply lambda functions to make the Tweet text lower case and remove any special characters.<jupyter_code>df['text'] = df['text'].apply(lambda x: x.lower())
df['text'] = df['text'].apply((lambda x: re.sub('[^a-zA-z0-9\s]','',x)))
df.head()
df_shape = df.shape
print("Total Rows:  " + repr(df_shape[0]))
print("Total Columns:  " + repr(df_shape[1]))<jupyter_output>Total Rows:  23137
Total Columns:  9<jupyter_text>Iterates through the Pandas DataFrame, applies the sentiment classification function to each of the texts.  The result of this is appended to different dictionaries that are used for mapping back to the source DataFrame.<jupyter_code>subj_dict = {}
pol_dict = {}
class_tf_dict = {}
acc_tf_dict = {}
class_num_dict = {}
nouns_dict = {}
verbs_dict = {}
nba_dict = {}

start_time = time.time()
print(start_time)

errors = 0
count = 0

for row in text_df.iterrows():
    oid = row[1]['OBJECTID']
    text = row[1]['text']
    if text != None:
        try:
            oid, subj, pol, class_tf, acc_tf, class_num, nouns, verbs, nba = calculate_sentiment(oid, text)
            subj_dict[oid] = subj
            pol_dict[oid] = pol
            class_tf_dict[oid] = class_tf
            acc_tf_dict[oid] = acc_tf
            class_num_dict[oid] = class_num
            nouns_dict[oid] = nouns
            verbs_dict[oid] = verbs
            nba_dict[oid] = nba
            if count % 100 == 0:
                print(oid, subj, pol, class_tf, acc_tf, class_num, nouns, verbs, nba)
        except:
            errors +=1
            print("Error on oid " + str(oid))
        count +=1
        
        
    
end_time = time.time()
print(end_time - start_time)
print("Process completed with an error percentage of " + repr((errors/df_shape[0]) * 100))<jupyter_output><empty_output><jupyter_text>Map the dictionaries created in the previous step back to the source DataFrame.<jupyter_code>df['Subjectivity'] = df['OBJECTID'].map(subj_dict)
df['Polarity'] = df['OBJECTID'].map(pol_dict)
df['Classification (Tensorflow)'] = df['OBJECTID'].map(class_tf_dict)
df['Accuracy (Tensorflow)'] = df['OBJECTID'].map(acc_tf_dict)
df['Classification Number'] = df['OBJECTID'].map(class_num_dict)
df['Nouns'] = df['OBJECTID'].map(nouns_dict)
df['Verbs'] = df['OBJECTID'].map(verbs_dict)
df['NBA'] = df['OBJECTID'].map(nba_dict)
df.to_csv('output/SentimentData.csv')
df.head()
enriched_df = pd.read_csv('output/Twitter_locations_Neighborhoods.csv')
enriched_df.head()
neighborhood_nouns = {}
neighborhood_verbs = {}
neighborhood_hashtags = {}
for row in enriched_df.iterrows():
    neighborhood = row[1]['LABEL']
    if neighborhood not in neighborhood_nouns.keys():
        neighborhood_nouns[neighborhood] =[]
    if neighborhood not in neighborhood_verbs.keys():
        neighborhood_verbs[neighborhood] =[]
    if neighborhood not in neighborhood_hashtags.keys():
        neighborhood_hashtags[neighborhood] =[]
for row in enriched_df.iterrows():
    neighborhood = row[1]['LABEL']
    noun_list = row[1]['Nouns']
    verb_list = row[1]['Verbs']
    hashtag_list = row[1]['Hashtags']
    nouns = noun_list[1:-1].split(',')
    verbs = verb_list[1:-1].split(',')
    
    
    for noun in nouns:
        neighborhood_nouns[neighborhood].append(noun)
    for verb in verbs:
        neighborhood_verbs[neighborhood].append(verb)
    if type(hashtag_list) != float:
        hashtags = hashtag_list[1:-1].split(',')
        for hashtag in hashtags:
            neighborhood_hashtags[neighborhood].append(hashtag)
top_nouns_neighborhood = {}
top_verbs_neighborhood = {}
top_hashtags_neighborhood = {}
for key, value in neighborhood_nouns.items():
    noun_counts = dict(Counter(value))
    sorted_noun_counts = sorted(noun_counts.items(), key=itemgetter(1))
    top_nouns_neighborhood[key] = sorted_noun_counts[len(sorted_noun_counts)-5:]
    print(key, sorted_noun_counts[len(sorted_noun_counts)-5:])
for key, value in neighborhood_hashtags.items():
    ht_counts = dict(Counter(value))
    sorted_ht_counts = sorted(ht_counts.items(), key=itemgetter(1))
    top_hashtags_neighborhood[key] = sorted_ht_counts[len(sorted_ht_counts)-5:]
    print(key, sorted_ht_counts[len(sorted_ht_counts)-5:])
for key, value in neighborhood_verbs.items():
    verbs_counts = dict(Counter(value))
    sorted_verb_counts = sorted(verbs_counts.items(), key=itemgetter(1))
    top_verbs_neighborhood[key] = sorted_verb_counts[len(sorted_verb_counts)-5:]
    print(key, sorted_verb_counts[len(sorted_verb_counts)-5:])
sorted_noun_counts[1:]
gis = GIS("http://esrifederal.maps.arcgis.com", "james_jones_federal", "QWerty654321@!")

sdf = pd.DataFrame.spatial.from_featureclass("Baltimore_neighborhoods/nhood_2010.shp")
sdf.head()
sdf['Most_Common_Nouns'] = sdf['LABEL'].map(top_nouns_neighborhood)
sdf['Most_Common_Verbs'] = sdf['LABEL'].map(top_verbs_neighborhood)
sdf['Most_Common_Hashtags'] = sdf['LABEL'].map(top_hashtags_neighborhood)
sdf.head()
sdf_out = sdf[['LABEL', "Most_Common_Nouns", "Most_Common_Verbs", "Most_Common_Hashtags"]]
sdf_out.to_csv('output/sdf_neighborhoods.csv')
sdf_out.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10523541813070977, 'fraction_numerical': 0.007817990161630359, 'mean_word_length': 4.6925, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '573824', 'license': 'MIT', 'n_tokens_mistral': 3818, 'n_tokens_neox': 3691, 'n_words': 966, 'path': 'Analyze Twitter Data.ipynb', 'repo_name': 'GeoJamesJones/GIS-Final', 'size': '11384'}