<jupyter_start><jupyter_text>Access Twitter APIYou are free to use or adapt this notebook for any purpose you'd like. However, please respect the [Simplified BSD License](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/blob/master/LICENSE.txt) that governs its use.  Twitter API AccessTwitter implements OAuth 1.0A as its standard authentication mechanism, and in order to use it to make requests to Twitter's API, you'll need to go to https://apps.twitter.com/ and create a sample application.Choose any name for your application, write a description and use `http://google.com` for the website. Depending on your country, you may need to first have your phone number updated in your Twitter profile.Click on the **Key and Access Tokens** tab. There are four primary identifiers you'll need to note for an OAuth 1.0A workflow: * consumer key, * consumer secret, * access token, and * access token secret (Click on Create Access Token to create those).Note that you will need an ordinary Twitter account in order to login, create an app, and get these credentials. The first time you execute the notebook, add all credentials so that you can save them in the `pkl` file, then you can remove the secret keys from the notebook because they will just be loaded from the `pkl` file.The `pkl` file contains sensitive information that can be used to take control of your twitter acccount, **do not share it**.<jupyter_code>import pickle
import os
import twitter
import json
from collections import Counter
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
if not os.path.exists('secret_twitter_credentials.pkl'):
    Twitter={}
    Twitter['Consumer Key'] = '' # enter your consumer key here, then after storing it into your pkl file delete it 
    Twitter['Consumer Secret'] = '' # your consumer secret
    Twitter['Access Token'] = ''    # your access token
    Twitter['Access Token Secret'] = '' # your Access Token Secret
    with open('secret_twitter_credentials.pkl','wb') as f:
        pickle.dump(Twitter, f)
else:
    Twitter=pickle.load(open('secret_twitter_credentials.pkl','rb'))<jupyter_output><empty_output><jupyter_text>Authorizing an application to access Twitter account data<jupyter_code># creates an authentication object
auth = twitter.oauth.OAuth(Twitter['Access Token'],
                           Twitter['Access Token Secret'],
                           Twitter['Consumer Key'],
                           Twitter['Consumer Secret'])

# authentication is used to create a Twitter API object
twitter_api = twitter.Twitter(auth=auth)

# Nothing to see by displaying twitter_api except that it's now a
# defined variable

type(twitter_api)<jupyter_output><empty_output><jupyter_text>Retrieving trends in a particular location Trending topics are hashtags (example) or the words that are currently popular. They are categorized by country or city.Twitter identifies locations using the [Yahoo! Where On Earth ID](http://woeid.rosselliot.co.nz). The Yahoo! Where On Earth ID for the entire world is 1.<jupyter_code>WORLD_WOE_ID = 1
US_WOE_ID = 23424977
LOCAL_WOE_ID = 2486340 #The WOEID for Sacramento - you can change it to your local WOEID<jupyter_output><empty_output><jupyter_text>Returns top 50 popular trends in the specified location:<jupyter_code>world_trends = twitter_api.trends.place(_id=WORLD_WOE_ID)
us_trends = twitter_api.trends.place(_id=US_WOE_ID)
local_trends = twitter_api.trends.place(_id=LOCAL_WOE_ID)
print type(us_trends)

trends = us_trends[0]
print type(trends)
print trends.keys()
print trends['locations']
print trends['created_at']
print trends['as_of']

trends_tr = trends['trends']
print type(trends_tr)
print 'Number of trends:', len(trends_tr)
print 'The most popular trend:', trends_tr[0]<jupyter_output>[{u'woeid': 23424977, u'name': u'United States'}]
2018-05-04T23:13:18Z
2018-05-04T23:19:10Z
<type 'list'>
Number of trends: 50
The most popular trend: {u'url': u'http://twitter.com/search?q=%22Matt+Harvey%22', u'query': u'%22Matt+Harvey%22', u'tweet_volume': 18595, u'name': u'<NAME>', u'promoted_content': None}<jupyter_text>Displaying API responses as JSON format The API response is in JSON format, which is a data format used to transfer data on the web. It is roughly equivalent to nested Python dictionaries and lists.<jupyter_code>print((json.dumps(trends_tr, indent=1))) # indent every paranethesis by a character<jupyter_output>[
 {
  "url": "http://twitter.com/search?q=%22Matt+Harvey%22", 
  "query": "%22Matt+Harvey%22", 
  "tweet_volume": 18595, 
  "name": "<NAME>", 
  "promoted_content": null
 }, 
 {
  "url": "http://twitter.com/search?q=%22DJ+Khaled%22", 
  "query": "%22DJ+Khaled%22", 
  "tweet_volume": 102294, 
  "name": "<NAME>", 
  "promoted_content": null
 }, 
 {
  "url": "http://twitter.com/search?q=%22Monomoy+Girl%22", 
  "query": "%22Monomoy+Girl%22", 
  "tweet_volume": null, 
  "name": "<NAME>", 
  "promoted_content": null
 }, 
 {
  "url": "http://twitter.com/search?q=%23MillennialBedtimeStories", 
  "query": "%23MillennialBedtimeStories", 
  "tweet_volume": null, 
  "name": "#MillennialBedtimeStories", 
  "promoted_content": null
 }, 
 {
  "url": "http://twitter.com/search?q=%23NRAConvention", 
  "query": "%23NRAConvention", 
  "tweet_volume": 34500, 
  "name": "#NRAConvention", 
  "promoted_content": null
 }, 
 {
  "url": "http://twitter.com/search?q=%23YeahYoureWelcome", 
  "query[...]<jupyter_text>Computing the intersection of two sets of trends Collect the names of most popular trends:<jupyter_code>trends_set = {}
trends_set['world'] = set([trend['name'] 
                        for trend in world_trends[0]['trends']])

trends_set['us'] = set([trend['name'] 
                     for trend in us_trends[0]['trends']]) 

trends_set['local'] = set([trend['name'] 
                     for trend in local_trends[0]['trends']])<jupyter_output><empty_output><jupyter_text>Print the trend names:<jupyter_code>for loc in ['world','us','local']:
    print '-'*10,loc
    print(','.join(trends_set[loc]))
print(( '='*10,'intersection of world and US'))
print((trends_set['world'].intersection(trends_set['us'])))

print(('='*10,'intersection of US and local'))
print((trends_set['local'].intersection(trends_set['us'])))<jupyter_output>('==========', 'intersection of world and US')
set([u'<NAME>', u'#MillennialBedtimeStories', u'#NRAConvention', u'Monomoy Girl', u'#YeahYoureWelcome', u'<NAME>', u'Brighton'])
('==========', 'intersection of US and local')
set([u'<NAME>', u'<NAME>', u'#WPXIStorm', u'Smash Mouth', u'Amber Alert', u'<NAME>', u'<NAME>', u'#MayThe4BeWithYou', u'<NAME>', u'<NAME>', u'<NAME>', u'<NAME>', u'Manafort', u'Rushing Fall', u'#MillennialBedtimeStories', u'#VRLA2018', u'#NRAConvention', u'#USOW2018', u'#JEGS200', u'<NAME>', u'Monomoy Girl', u'<NAME> and CBS News', u'<NAME>', u'Brighton', u'#CarbLoadASongOrBand', u'57,000 Hondurans', u'Coach Rocks', u'#CrimeCon', u'#GGGVanes', u'#JunotDiaz', u'#SouthernNats', u'<NAME>', u'<NAME>', u'#YeahYoureWelcome', u'Waymo', u'#BHAMUN', u'<NAME>', u'<NAME>', u'#ECU18', u'<NAME>', u'#MexicoSeries', u'<NAME>', u'#dearwhitepeopleseason2'])<jupyter_text>Collecting search results Set the variable `q` to a trending topic, or anything else for that matter.<jupyter_code>topic = '#MillennialBedtimeStories' 
number = 100

search_results = twitter_api.search.tweets(q=topic, count=number)
statuses = search_results['statuses']
print type(search_results)
print search_results.keys()
print len(statuses)
print statuses[0].keys()
print statuses[0]['text']<jupyter_output><class 'twitter.api.TwitterDictResponse'>
[u'search_metadata', u'statuses']
100
[u'contributors', u'truncated', u'text', u'is_quote_status', u'in_reply_to_status_id', u'id', u'favorite_count', u'entities', u'retweeted', u'coordinates', u'source', u'in_reply_to_screen_name', u'in_reply_to_user_id', u'retweet_count', u'id_str', u'favorited', u'retweeted_status', u'user', u'geo', u'in_reply_to_user_id_str', u'possibly_sensitive', u'lang', u'created_at', u'in_reply_to_status_id_str', u'place', u'extended_entities', u'metadata']
RT @MrRaceBannon: The Three Little Pigs  #MillennialBedtimeStories https://t.co/wSsicFEeQx<jupyter_text>Twitter often returns duplicate results, we can filter them out checking for duplicate texts:<jupyter_code>all_text = []
filtered_statuses = []
for s in statuses:
    if not s["text"] in all_text:
        all_text.append(s["text"])
        filtered_statuses.append(s)
statuses = filtered_statuses     
len(statuses)
all_text = [s['text'] for s in statuses]
all_text[:10]<jupyter_output><empty_output><jupyter_text>Explore one of the tweets:<jupyter_code># Show one sample search result by slicing the list
t = statuses[0]
print(json.dumps(t, indent=1))
print(t['retweet_count'])
print(t['retweeted'])
## Extracting text, screen names (usernames of twitter account), and hashtags from tweets

status_texts = [status['text'] for status in statuses ]

screen_names = [user_mention['screen_name'] for status in statuses for user_mention in status['entities']['user_mentions'] ]

hashtags = [hashtag['text'] for status in statuses for hashtag in status['entities']['hashtags'] ]
print 'Texts:\n', status_texts[:5]
print '\nUsernames:', screen_names[:5]
print '\nHashtags:', hashtags[:5]
# collection of all words from all tweets
words = [w for t in status_texts for w in t.split() ]
print words[:15]<jupyter_output>[u'RT', u'@MrRaceBannon:', u'The', u'Three', u'Little', u'Pigs', u'#MillennialBedtimeStories', u'https://t.co/wSsicFEeQx', u'#MillennialBedtimeStories', u'Once', u'upon', u'a', u'ending', u'millennium', u'convinced']<jupyter_text>Basic frequency distribution of the words in tweets<jupyter_code>word_counter = Counter(words)
word_counter.most_common()[:10]
sorted_word_counts = sorted(word_counter.values(), reverse=True)
for item in [words, screen_names, hashtags]:
    c = Counter(item)
    print(c.most_common()[:10]) # top 10
    print
##Create a prettyprint function to display tuples in a nice tabular format

def prettyprint_counts(label, list_of_tuples):
    # the header of the table: first write the "label", pad it to 20 spaces and center-align the label.
    # then, write the word "Count", pad it to 6 spaces and center-align it.
    print("\n{:^20} | {:^6}".format(label, "Count"))
    # print * for 40 times
    print("*"*40)
    for k,v in list_of_tuples:
        # for each word and its count, print the word as left-aligned and the count as right-aligned. 
        print("{:20} | {:>6}".format(k,v))
for label, data in (('Word', words), ('Screen Name', screen_names), ('Hashtag', hashtags)):
    c = Counter(data)
    prettyprint_counts(label, c.most_common()[:10])<jupyter_output>Word         | Count 
****************************************
#MillennialBedtimeStories |     66
RT                   |     41
the                  |     31
The                  |     24
and                  |     18
a                    |     11
A                    |      9
#MillennialBedTimeStories |      8
Little               |      8
to                   |      6

    Screen Name      | Count 
****************************************
eatpraystyle         |      6
Nalahru              |      6
Rizember             |      6
FioNoire             |      6
BrushingOff          |      2
MrRaceBannon         |      2
billy4ever9          |      2
JeffHendrix88        |      1
Superbokka           |      1
kevinwxgg            |      1

      Hashtag        | Count 
****************************************
MillennialBedtimeStories |     66
MillennialBedTimeStories |      8
WhatStarWarsTaughtMe |      1
Vacuum               |      1
Robot                |      1
millennialbedtim[...]<jupyter_text>Finding the most popular retweets<jupyter_code>retweets = [
            # Store out a tuple of these three values ...
            (status['retweet_count'], 
             status['retweeted_status']['user']['screen_name'],
             status['text'].replace("\n","\\")) #replace a new line with backslash
            
            # ... for each status ...
            for status in statuses 
            
            # ... so long as the status meets this condition.
                if 'retweeted_status' in status
           ]
retweets[:5]
sorted_retweets = sorted(retweets, reverse=True)
sorted_retweets[:5]
##Create a prettyprint function to display tweets with their retweet counts

row_template = "{:^7} | {:^15} | {:50}"
def prettyprint_tweets(list_of_tuples):
    print()
    print(row_template.format("Count", "Screen Name", "Text"))
    print("*"*60)
    for count, screen_name, text in list_of_tuples:
        print(row_template.format(count, screen_name, text[:50].encode('utf-8')))
        # split the text of the tweet in up to 3 lines, if needed
        if len(text) > 50:
            print(row_template.format("", "", text[50:100].encode('utf-8')))
            if len(text) > 100:
                print(row_template.format("", "", text[100:].encode('utf-8')))

prettyprint_tweets(sorted_retweets)<jupyter_output>()
 Count  |   Screen Name   | Text                                              
************************************************************
  207   | DaSkrambledEgg  | RT @DaSkrambledEgg: The Little engine that couldn'
        |                 | t even #MillennialBedTimeStories                  
  121   |   BrushingOff   | RT @BrushingOff: Bi-Curious George. #MillennialBed
        |                 | TimeStories https://t.co/4oQXYhsND3               
  85    |  MrRaceBannon   | RT @MrRaceBannon: Hansel and Regretel #MillennialB
        |                 | edtimeStories https://t.co/U06Pr4Ou60             
  79    |  colbywinters   | RT @colbywinters: Little red riding hood AF\ #Mill
        |                 | ennialBedtimeStories                              
  62    | iamalmostlegend | RT @iamalmostlegend: Organic Green Eggs &amp; Free
        |                 |  Range Ham \#MillennialBedtimeStories             
  57    |   BrushingOff   | RT @BrushingOff: Charlotte's Webcam. #M[...]
{'fraction_non_alphanumeric': 0.14181973515591628, 'fraction_numerical': 0.0199345009255304, 'mean_word_length': 3.8488091128753883, 'pattern_counts': {'":': 26, '<': 67, '<?xml version=': 0, '>': 70, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '89976', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 4819, 'n_tokens_neox': 4470, 'n_words': 1215, 'path': 'TwitterAPI.ipynb', 'repo_name': 'pirmoradian/TwitterAPI', 'size': '14168'}