<jupyter_start><jupyter_text>Assignment 2, Data Science for Design 2017 <NAME>, s1635468  Bot on twitter during BrexitDuring the brexit, many people talk about it through social media. one social media platform that was used by people is twitter. usually, this platform is used by the researcher as their data sources. furthermore, they use it to identify the social condition like expectation and preception about some topic, in this case is Brexit.Before we gain insight about something using twitter data. can we trust this data source to identify social behaviour? Since there is not only human that sharing their idea through this platform. but also A bot. Based on BotOMeter researcher, <NAME>, Ph.D, explained, “We found that approximately 15% of the users active in the political conversation one month prior to the 2016 election were likely bots… and they were responsible to about one in five such political tweets (nearly 20%).”*. Commonly, this bot was created by human to achieve the bot creator purpose, such as spamming and influencing the social media to create perception about something. If we want to talk about people perception by using twitter data. Should we exclude the bot from the dataset? but they might taking part on shaping people perception.the main analysis on this notebook to identify the number of bots from twitter data. and also to identify what sentiment being brought by the bot. source:- "Social  Twitter  bots  fake news Bot-hunting Twitter bot sniffs out bogus political tweets"https://techcrunch.com/2017/10/25/botometer-twitter-bot-hunting-probabot/ Data Description and PreparationData Preparation 1: Import twitter dataThe data used in this project are tweets exported from Twitter in the context of the UK brexit.The data are generated from the following hashtags: eureferendum, euref, brexit, no2eu, yes2eu, notoeu, yestoeu, betteroffout, betteroffin, voteout, votein, eureform, ukineu, Bremain ,EUpoll, UKreferendum, UKandEU, EUpol, ImagineEurope, EdEUref, MyImageOfTheEU eu, referendum, europe, UKRef, ref, migrant, refugee strongerin, leadnotleave, voteremain, britainout, 'leaveeu, voteleave, beleave, loveeuropeleaveeu, greenerin, britin, eunegotiation, eurenegotiation, grassrootsout, projectfear, projectfact, remaineu, europeanunion, brexitfears, remain, leave, takecontrol, euinorout, leavechaos, labourin, conservatives, bregret, brexitvote, brexitin5words, labourcoup, eurefresults, projectfear, VoteLeaveLoseControl, regrexit, wearethe48, scexit, niineurope, scotlandineurope, article50, scotlandineuThe generated data are stored as JSON files. In total they are approximately 18GB of compressed Tar files.The dataset is collected on 5 days around brexit.- 21 June 2016- 22 June 2016- 23 June 2016- 22 September 2017- 23 September 2017For each day there are JSON files for tweets about brexit collected every hour in 24 hours.Each JSON files varies with one another in size, ranging from 4MB in to 500MB.In this notebook, we use only one hour time frame on 22 September 2017 between 12 AM to 13 PM.<jupyter_code>%matplotlib inline

import tarfile
import json
import gzip
import pandas as pd
import re
from textblob import TextBlob
from pandas.io.json import json_normalize
from scipy import stats, integrate
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

tar = tarfile.open("../input/2017-09-22.tar.gz", "r:gz")
data = pd.DataFrame()
count=0

'''
        Extracting from twitter data.
'''

for members in tar.getmembers():
    if (count==13):
        f = tar.extractfile(members)
        data = data.append(pd.read_json(f, lines=True))
    count = count + 1
print("number of rows :", data.shape[0])
print("number of column :",data.shape[1])
print("")
print("raw twitter data columns :")
print("")
print(data.iloc[0])<jupyter_output>('number of rows :', 4369)
('number of column :', 36)

raw twitter data columns :

contributors                                                               NaN
coordinates                                                               None
created_at                                                 2017-09-22 12:05:02
display_text_range                                                    [0, 140]
entities                     {u'user_mentions': [], u'symbols': [], u'hasht...
extended_entities                                                          NaN
extended_tweet               {u'display_text_range': [0, 135], u'entities':...
favorite_count                                                               0
favorited                                                                False
filter_level                                                               low
geo                                                                       None
id                                              [...]<jupyter_text>Column SelectionNot all of the column is used on this analysis. Since we want to know which account bot, I decided to choose column "screen_name" so I could find analysis this account by using BotOMeter API. and also, I picked "retweet_count", "reply_count", "quote_count", and "retweeted_status" columns to know whether this account got feedback from other users. and column "created_at" to know time series of the data. and extracting hashtag to know which hashtag was being trend during that time. and also lang, to identify language being used by the twitter user.<jupyter_code>'''
        Compiling hashtags result.
'''
count=0
hashtags=[]
for ent in data['entities']:
    str_hsh=''
    for hsh in ent['hashtags']:
        #print(hsh['text'])
        str_hsh=hsh['text']+';'+str_hsh
    hashtags.append(str_hsh.upper())
    count = count+1
    
'''
        Compiling user screen_name result.
'''
namelist=[]
for x in data['user']:
    namelist.append(x['screen_name'])

twt_data=pd.DataFrame({'screen_name':namelist, 'lang':data['lang'] 
                            ,'retweet_count':data['retweet_count']
                            ,'reply_count':data['reply_count']
                            ,'quote_count':data['quote_count']
                            ,'retweeted_status':data['retweeted_status']
                            ,'hashtags':hashtags
                            ,'created_at':data['created_at']
                           })
twt_data.head()<jupyter_output><empty_output><jupyter_text>Data Preparation 2: Import data from previous BotOMeter preprocessing I used BotOMeter API To identify which twitter account is bot. This API investigating the account by using their machine learning system to extracts more than a thousand features in six different classes: users and friends meta-data, tweet content and sentiment, network patterns, and activity time series. I worked on diffrent notebook to get the bot score since this process takes long time to produce the result and dumped into "outputbot_[1:8].csv". So, in this notebook we only unload the csv file into dataframe.botometer preprocessing notebook:http://localhost:8888/notebooks/iqbal/bom_crawler.ipynbBotOMeter documentation : - https://botometer.iuni.iu.edu/- https://github.com/IUNetSci/botometer-pythonReference:- Online Human-Bot Interactions: Detection, Estimation, and Characterization, <NAME> <NAME>, ICWSM 2017.src: https://aaai.org/ocs/index.php/ICWSM/ICWSM17/paper/view/15587/14817<jupyter_code>bot_data = pd.DataFrame()
distinct_bot_data = ()

for i in range(1,8):
    output="outputbot_"+str(i)+".csv"
    bot_data=bot_data.append(pd.read_csv(output))
bot_data.head()<jupyter_output><empty_output><jupyter_text>Column SelectionI chosed column "screen_name" to set is as foreign key so I could join it with twitter data. and also the data itself is distincted  by screen_name. and also I used "englsih" column to identify their similiarness to the bot. the value for this colun is between 0.0 and 1.0.<jupyter_code>#print(len(bot_data))
bot_data=bot_data[['screen_name','english']]
print("Data example :")
print(bot_data.head())
print("\n\nData description:")
print(bot_data.describe())<jupyter_output>Data example :
    screen_name  english
0  allanwhyte81     0.35
1  blueghosties     0.33
2       83Kindy     0.39
3   WheezingWez     0.22
4  abacohosting     0.45


Data description:
           english
count  3120.000000
mean      0.373854
std       0.117017
min       0.040000
25%       0.290000
50%       0.360000
75%       0.450000
max       0.880000<jupyter_text>Data Preparation 3: Import Sentiment based on Naive Bayes dataIn order to get sentiment analysis, I used TextBlob API. This API using Naive Bayes Analyzer to identify sentiment on machine learning based on corpora movie review database. The output result of API is :- Classification - either pos or neg indicating if the text is positive or negative, the value of this column is "neg" or "pos"- P_Pos - a measure of how positive the text is between 0.0 and 1.0- P_Neg - a measure of how negative the text is between 0.0 and 1.0Since it takes long time on processing. I decided to work on different notebook and dump into "sentiment_output.csv".http://localhost:8888/notebooks/iqbal/textblob_naivebayes_sentiment.ipynbDocumentation:- https://textblob.readthedocs.io/en/dev/index.html<jupyter_code>sentiment_data = pd.DataFrame()
sentiment_data = pd.read_csv("sentiment_output.csv")
sentiment_data = sentiment_data[["class","p_neg","p_pos"]]
sentiment_data[0:2]<jupyter_output><empty_output><jupyter_text>Data Merging: Merging the data sourcesJoining the twitter data and the sentiment data, since this data is in the same order and has same rows. I decided to use join function.<jupyter_code>sen_dat=twt_data.join(sentiment_data[['class','p_neg','p_pos']])<jupyter_output><empty_output><jupyter_text>Joining the twitter data and the botometer data. I use 'screen_name" as key to join as left outer join with twitter data.<jupyter_code>compiled_data=pd.merge(sen_dat,bot_data, how='left', on=['screen_name'])

compiled_data=compiled_data.rename(columns = {'english':'botometer'})
old_compiled_data=compiled_data
old_compiled_data=compiled_data<jupyter_output><empty_output><jupyter_text>Data checking process: Checking if the input and the data after joining has the same output. So there is no miss during join process.<jupyter_code>assert (len(compiled_data)==len(data)), "Input data and output data has a different count"<jupyter_output><empty_output><jupyter_text>Describing the Data<jupyter_code>old_compiled_data.describe()
old_compiled_data.iloc[0]<jupyter_output><empty_output><jupyter_text>Cleansing null value on column BotoMeterWhen the column of this botometer is null, it's mean the account has been deactivated. This value will be filled by value 0.<jupyter_code>compiled_data[pd.isnull(compiled_data['botometer'])==True][0:2]
print("there is null value on column botometer, this null indicated this account already deactivate their account :")
len(compiled_data[pd.isnull(compiled_data['botometer'])==True])
print("replacing null value to zero is done")
compiled_data['botometer']=compiled_data.botometer.replace(np.NaN, 0)
print("there is null value on column botometer, this null indicated this account already deactivate their account :")
len(compiled_data[pd.isnull(compiled_data['botometer'])==True])<jupyter_output>replacing null value to zero is done
there is null value on column botometer, this null indicated this account already deactivate their account :<jupyter_text>English Language tweet filteringSince sentiment analysis is used by english language, I decided to excluded non-english language.<jupyter_code>check=compiled_data[['lang','quote_count']]
check=check.rename(columns = {'quote_count':'count'})
check.groupby(['lang'], sort=True, as_index=False).count().sort_values(by='count', ascending=False)
compiled_data=compiled_data[compiled_data['lang']=='en']
print("number of tweet that have non-english language :")
len(compiled_data[compiled_data['lang']!='en'])<jupyter_output>number of tweet that have non-english language :<jupyter_text>Analysis PartAnalysis 1: sentiment analysis of hashtagsIdentifying sentiment analysis of hashtags by finding the sentiment value of tweet where the hashtag available. and then averaging the sentiment analysis.<jupyter_code>txt_tag=[]
sntment_tag=[]
for x in range(len(compiled_data)):
    oo=TextBlob(compiled_data.iloc[x]['hashtags']).words
    for p in oo:
        txt_tag.append(p)
        sntment_tag.append(compiled_data.iloc[x]['p_pos'])

hashtag_analysis=pd.DataFrame({
            'hashtag':txt_tag,
            'sentimenttag':sntment_tag
            })
hashtag_analysis[0:2]
avg=hashtag_analysis.groupby(['hashtag'], as_index=False).mean()
cnt=hashtag_analysis.groupby(['hashtag'], sort=True, as_index=False).count().sort_values(by='sentimenttag', ascending=False)

analysis1=pd.merge(avg,cnt, how='left', on=['hashtag'])
analysis1=analysis1.rename(columns = {
                            'sentimenttag_y':'count',
                            'sentimenttag_x':'p_pos'
                           })
analysis1=analysis1.sort_values(by=['count'], ascending=False)[0:10]
analysis1<jupyter_output><empty_output><jupyter_text>ResultsBased on two barplot below, top 10 hashtags have average positive sentiment analysis on their tweet.<jupyter_code># Plot the total crashes
sns.set_color_codes("pastel")
sns.barplot(x="count", y="hashtag", data=analysis1,
            label="Total", palette="Blues_d")
sns.set_color_codes("pastel")
sns.barplot(x="p_pos", y="hashtag", data=analysis1,
            label="Total", palette="Blues_d")
sns.despine(left=True, bottom=True)
sns.set_color_codes("pastel")
sns.barplot(x="p_pos", y="hashtag", data=analysis1,
            label="Total")<jupyter_output><empty_output><jupyter_text>Analysis 2: the number of botDistribution Plot below show the number of bots. where value above 0.5 is suspected as a bot.but it seems the human still dominate the twitter, since the median falling around between 0.3 and 0.4. but there is a Zero value of botometer, since the user is not found. but they exist during the data time frame. Are they only activate their account during that time? are they human or bot? or are they already indetified as bot by twitter and then the got banned?<jupyter_code>sns.set_style("whitegrid")
ax=sns.distplot(compiled_data['botometer'], kde=False)
ax.set_xlabel("BotOMeter Score")
ax.set_ylabel("Count")<jupyter_output><empty_output><jupyter_text>Analysis 3: Sentiment on the timelineAs it can be seen, there is a lot of the number of positive sentiment rather than the negative sentiment. and they reach a peak at 12:31 PM.<jupyter_code>dims = (14, 4)
fig, axarr = plt.subplots(figsize=dims)
new_labels = ['created_at', 'class']
ii=0

time_analysis=pd.DataFrame({
            'created_at':compiled_data['created_at'].dt.strftime('%H:%M'),
            'class':compiled_data['class'],
            'count':range(len(compiled_data))
            })

time_analysis=time_analysis.groupby(['created_at', 'class'], as_index=False).count()
#time_analysis.head()

sns.set()
sns.set_context("paper")
#sns.set_style("darkgrid", {'xtick.direction': u'out'})
g=sns.pointplot(x="created_at", y="count", hue='class', data=time_analysis,
              palette={"neg": "r", "pos": "g"},
              markers=["x", "o"], linestyles=["-", "--"],
              ax=axarr, scales=0.7, aspect=9.0
             )

for item in g.get_xticklabels():
    item.set_rotation(60)
    item.set_size(7.5)<jupyter_output><empty_output><jupyter_text>Analysis 4: Are the bot spread the positivity or the negativityAs it can be seen, the positive sentiment tweets dominate the conversation during the time frame. based on the column "class", positive sentiments fall for the p_pos above 0.5. where negative sentiment fall below 0.5. and there is a linear positive correlation between positive tweet and the botmeter value. on the other hand, there is a slight neagative correlation between negative tweet and botometer.<jupyter_code>sns.set(style="ticks")
sns.set_context("paper")
y = sns.lmplot(x="botometer", y="p_pos", hue='class', data=compiled_data, 
               markers=["o", "x"],
               palette={"neg": "r", "pos": "g"},
               aspect=.9,
               size=8)
plt.ylim(0, 1.03) 
plt.xlim(-0.05, 0.95)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0952878937007874, 'fraction_numerical': 0.02011564960629921, 'mean_word_length': 3.936835712116611, 'pattern_counts': {'":': 4, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '59447', 'license': 'MIT', 'n_tokens_mistral': 5011, 'n_tokens_neox': 4645, 'n_words': 1580, 'path': '2nd_Assignment/iqbal/bot_analysis.ipynb', 'repo_name': 'indralukmana/twitter-brexit', 'size': '16336'}