<jupyter_start><jupyter_text>An analysis of the Twitter of Trump and Obama Authors: <NAME>, <NAME> and <NAME>  AbstractThe purpose of the repository is analyzing the Twitter use of Donald Trump and Barack Obama. First, we briefly plot the usage of the twitter for both presidents,e.g frequency of tweets, popularity change, etc. Then we did the text processing with NLTK to explore aggregate text statistics of the speeches and visualize some of their properties.Exploring some of the relations between the properties of the tweets, time, provide a novel visualization of the relevant information. Next, we analyzed the topic change between the president's Obama last 10 Months and president's Trump first 10 twitter. We used the word cloud for most frequent word, and LDA for the analysis of the topic change. Lastly, the Presidential Job Approval Rating from UCSB as a response variable to find whether the Approval Rating can be predicted by the twitter data.   Visualization of the Raw Data In notebook 1, we count the frequency of tweets by Mr. Obama and Mr. Trump by Month Category. Then plot the histogram. We think it's a simple way to find something directly. And we plot the hitogram to show the number of Twitters of senators by Month. We could know when will senators post more Twitters.<jupyter_code>import pandas as pd
from PIL import Image
import codecs
import webbrowser
import shelve
senator = Image.open('fig/senator_month.png')
senator<jupyter_output><empty_output><jupyter_text>From this graph, it seems senators post less Twitters from October to December.<jupyter_code>obama = Image.open('fig/obama_month.png')
obama
trump = Image.open('fig/trump_month.png')
trump<jupyter_output><empty_output><jupyter_text>We try to connect some important political event with those time. The United States presidential election of 2016 was the 58th quadrennial American presidential election, held on Tuesday, November 8, 2016. In addition, the number of Twitters of Donald Trump in October is much more than the other month. It seems the political events would influence the number of Twitters for Donald Trump. And for Brack Obama, there is no any month that the number of Twitters is much more than the other month. Next, we want to see how the popular trends for both Barack Obama and <NAME>ump when they are as president. Thus, we plot the series plot to compare their popularity in Twitter. And it's also easy to know how their popularity change:<jupyter_code>obamapopul = Image.open('fig/obama_popularity_time.png')
obamapopul
trumppopul = Image.open('fig/trump_popularity_time.png')
trumppopul<jupyter_output><empty_output><jupyter_text>As we can see, the popularity of Donald Trump is much more than Barack Obama's. Or we can say, the tweets posted by <NAME> raised more concerns. At first, we thought it may be the Twitter users is not that much 3 years ago. However, it makes no differences for the amount of Twitter users between right now and 3 years ago. It is easy to find the data with google search for that know how many Twitter users in the past 5 years. Then, we would like to see which tweet got many likes, retweets and replies, such as the tweets of Mr. Obama from 2016-12 to 2017-08, and the tweets of <NAME> in 2016-11 and 2017-07. Since the United States presidential election of 2016 was in November, 2016, we could know it's the reason why Mr. Trump's tweet got many attentions.   Text Processing and Novel Visualization of the Relevant Information First of all, we cleaned up the words they tweeted. Then, we count all the number of words, unique words, and number of characters they used and restore them in the datafram. And we describe it with count, mean, sd, min, max. We try to explore the dataset and find the useful information. Here is the dataframe:<jupyter_code>obama = pd.read_hdf('result/n2.h5','obama')
obama.head()
obama.describe()
trump = pd.read_hdf('result/n2.h5','trump')
trump.head()
trump.describe()<jupyter_output><empty_output><jupyter_text>Next, we did some text processing with NLTK to compute aggregate text statistics of the speeches as well as visualize some of their properties. Explore some of the relations between the properties of the tweets, time, their authors, provide a novel visualization of the relevant information. LHS is the visulization of Tweets by Mr. Obama. RHS is by <NAME>. Since we have a whole vector with creation dates, we can construct time series respect tweets lengths, likes and retweets. And if we want to plot the time series, pandas already has its own method in the object. We plot a time series as follows:<jupyter_code>nword = Image.open('fig/n_word_comparison.png')
nword
nuword = Image.open('fig/n_uword_comparison.png')
nuword
nchar = Image.open('fig/n_char_comparison.png')
nchar<jupyter_output><empty_output><jupyter_text>We try to compare the time series when they are president. It seems the the twitters quality(from word, unique word, characters) by Mr. Obama is more steady than Mr. Trump, since Mr.Obama is the United States president until 2016-12. From RHS in those visualization, the frequency of word, unique word, and character have a large fluctuation.   Topic change between the president's Obama last 10 Months and president's Trump first 10 twitter *Please see Notebook 3 for LDA Visualization We will use the LDA to analyze the topic of the president's Obama last 10 Months and president's Trump first 10 twitter.<jupyter_code>topicobama = pd.read_hdf('result/n3.h5','topicobama')
topicobama
topictrump = pd.read_hdf('result/n3.h5','topictrump')
topictrump<jupyter_output><empty_output><jupyter_text>Compare the topic change by LDA The important word from the top 5 topics for Obama- First topic are **fair, suprem, peopl, continu** - Second topic is **court, time, chang, live, economi** - Third topic is **senat, will, leader, think** - Forth topic is **judg, climat, american, obamacar** - Fifth topic is **doyourjob, new, women, gun**  The important word from the top 5 topics for Trump- First topic are **great, border, presid, iran, terribl** - Second topic is **fake, new, interview, tax, administr,hard** - Third topic is **countri, just, come, made** - Forth topic is **job, democrat, vote, white** - Fifth topic is **work, execut, women, help** Most topics from Obama's tweets are related to social issue and the communtiy, e.g. peopl, fair, obamacar, economi, senat, climat, women, gun. I can guess from LDA analysis, his topics more about the economic, climate change, the women rights, obama care, gun control. Actually, he worked on this topic in his presidency.Most topics from trump's tweets are related to national security, tax, media and his slogan "Make America Great Again" , e.g. great, border, iran, fake, new, interview, tax. I can guess from LDA analysis, his topics more about the his slogan "Make America Great Again", building wall along the Mexican border, blaming the media, Iran nuclear deal. Actually, he worked on this topic in his presidency. From the above result, we can say that the LDA perform well analyze the topic of twitter data. From the above result, we can say that the LDA perform well analyze the topic of twitter data.  WordCloud - Most Frequent Stemmed Words<jupyter_code>obamaclond = Image.open('fig/word_clond_obama.png')
obamaclond
trumpclond = Image.open('fig/word_clond_trump.png')
trumpclond<jupyter_output><empty_output><jupyter_text>Compare the word cloudWe can see that the most most frequent stemmed words of Obama's Twitter are *presid*, *senat* , *Obama*, *need*, *change*, *job*, *leader*, *American*.We can see that the most most frequent stemmed words of Trump's Twitter changed to  *will*, *new* , *great*, *thank*, *job*, *fake*, *presid*, *today*. We know 2008 is a huge challenge for the United States. Job problem, terrorism problem, and climate problem was what Mr. Obama try to focus. And for Mr. Trump, we know "Make America Great Again" (MAGA) is a campaign slogan used in American politics that was popularized by President <NAME> in his 2016 presidential campaign.  Sentiment Anaylsis and Predicting the Approval Rating We found the Presidential Job Approval Rating from UCSB as a response variable to evaluate our model. And then we do a sentiment anaylsis of Trump and Obama twitter data by using the Sentiment Anaylzer from the NLTK package. We use the sentiment anaylsis to get the positive and negative on their tweets , and compare their sentiment by the twitter data. We will use the OLS regresion to find out whether the Approval Rating can be predicted.  Sentiment AnalysisAn anaylsis of Trump and Obama twitter data by using the Sentiment Anaylzer from the NLTK package. We use the sentiment anaylsis to get the positive and negative on their tweets , and compare their sentiment by the twitter data.  The VADER algorithm outputs sentiment scores to 4 classes of sentimentsThe sentiment score from 0 to 1 measuring the subjectivness of the text. 0 is objective, 1 is subjective.- Postive:The score of measuring positive words- Negative:The score of measuring negative words- Neutral:The score of neutral words- Compound: The aggregated score which is normalized of the sum of all scores .  Summary of the Sentiment Analysis for Obama<jupyter_code>pd.read_hdf('result/n4.h5','obama_describe')<jupyter_output><empty_output><jupyter_text>Summary of the Sentiment Analysis for Trump<jupyter_code>pd.read_hdf('result/n4.h5','trump_describe')<jupyter_output><empty_output><jupyter_text>Comparing the Sentiment Analysis  of Trump and ObamaFrom the above results, we can see the compound value is for trump is 0.218 while obama is 0.221. The means of postive and negative value of Trump's tweet are higher than Obama. Obama's mean of neutral is higher than Trump. Therefore, we can see Obama' tweets trend to neutral .  Visualization of the Sentiment Analysis Counting the postive and negative tweets<jupyter_code>Image.open('fig/comparison.png')<jupyter_output><empty_output><jupyter_text>Plot the sentiment analysis<jupyter_code>Image.open('fig/Obama_Sentiment_Analysis.png')
Image.open('fig/Trump_Sentiment_Analysis.png')<jupyter_output><empty_output><jupyter_text>Analysis of Tweeter Data for Prediction Correlation matrix of Obama<jupyter_code>Image.open('fig/obama_cor.png')<jupyter_output><empty_output><jupyter_text>Correlation matrix of Trump<jupyter_code>Image.open('fig/trump_cor.png')<jupyter_output><empty_output><jupyter_text>Plot the Approval Rating, Compound and Favorites(Standardized the data)<jupyter_code>Image.open('fig/Obama_approval_relation.png')
Image.open('fig/Trump_approval_relation.png')<jupyter_output><empty_output><jupyter_text>OLS Model<jupyter_code>with shelve.open('result/vars3') as db:
    result1 = db['result1']
    result2 = db['result2']<jupyter_output><empty_output><jupyter_text>OLS regression for obama<jupyter_code>print(result1.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:               approval   R-squared:                       0.158
Model:                            OLS   Adj. R-squared:                  0.143
Method:                 Least Squares   F-statistic:                     10.57
Date:                Wed, 13 Dec 2017   Prob (F-statistic):           7.27e-08
Time:                        22:19:41   Log-Likelihood:                -541.96
No. Observations:                 231   AIC:                             1094.
Df Residuals:                     226   BIC:                             1111.
Df Model:                           4                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>As we can see the r-squared is 0.158 which is really small. It cannot show there is a significant correlation between the predictor variables and the response variable. P-value for Compound (0.703) and Replies (0.718) are much greater than the common alpha level of 0.05, which indicates that it is not statistically significant. Coefficient for Favorites( 0.0019) and Retweets(-0.0020) are really small which may not be statistically significant. It also indicatesthat there arestrong multicollinearity or other numerical problems, it probably caused by the replies, favorites , retweets. Because most people do these three things at the same time, and there is a strong correlation between these three things.   OLS regression for Trump<jupyter_code>print(result2.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:               approval   R-squared:                       0.140
Model:                            OLS   Adj. R-squared:                  0.127
Method:                 Least Squares   F-statistic:                     10.71
Date:                Wed, 13 Dec 2017   Prob (F-statistic):           4.63e-08
Time:                        22:19:51   Log-Likelihood:                -612.73
No. Observations:                 269   AIC:                             1235.
Df Residuals:                     264   BIC:                             1253.
Df Model:                           4                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]
{'fraction_non_alphanumeric': 0.08523101706084792, 'fraction_numerical': 0.016914402870322913, 'mean_word_length': 3.69993117687543, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '348456', 'license': 'MIT', 'n_tokens_mistral': 3840, 'n_tokens_neox': 3507, 'n_words': 1676, 'path': 'School Work/Twitter_Analysis/main.ipynb', 'repo_name': 'rickchen123/Portfolio', 'size': '13693'}