<jupyter_start><jupyter_text>Search for tweets — _Using Twitter API_ — ⚠️ This notebook was not useful to retrieve historic tweets.Considering that Trudeau's announcement was done on March 27th, I was willing to get tweets from March 1st until April 30th, to look for messages before and after the announcement and compare how much the sentiment of people changed over this period.However, the standard twitter developer's account only gives access to search historic databases with a 7-day limit, which means that I was not able to find tweets for a date older than one week. For this reason, this notebook which uses the standard twitter developer's account is not useful for the objective of our project. However, this notebook can be used as reference; to use it you will need to get a Twitter developer's account. Instead, we invite you to look at the notebook **Canada_Twitter.ipynb**, located in the same [_src_ folder](https://github.com/vcuspinera/Canada_response_covid/blob/master/src), where we successfully download the historic tweets used for this project._Author: <NAME> ([vcuspinera](https://github.com/vcuspinera))_  Import libraries<jupyter_code>import numpy as np
import pandas as pd
import twitter
import time
import collections
from datetime import datetime, timedelta
from pytz import timezone<jupyter_output><empty_output><jupyter_text>Bring your twitter's keys<jupyter_code># Call your twitter keys
keys = ! ../keys/twitter_config.py

# Save keys to use them later
api = twitter.Api(consumer_key = keys[0],
                  consumer_secret = keys[1],
                  access_token_key = keys[2],
                  access_token_secret = keys[3],
                  sleep_on_rate_limit=True #
                 )<jupyter_output><empty_output><jupyter_text>Parameters<jupyter_code># twitter accounts
accounts = ("CanadianPM", "Canada", "OpenGovCan", "JustinTrudeau")

# dates
# format YYYY-MM-DD, and has a 7-day limit (no tweets will be found 
# for a date older than one week.)
today = datetime.now()#timezone('US/Pacific'))
dates = list()
for d in range(7, -2, -1):
    aux = today - timedelta(days=d)
    dates.append(aux.strftime("%Y-%m-%d"))

today = today.strftime("%Y-%m-%d")

# tweets
num_tweets = 100
max_tweets = 180
time_epoc = 900 # 15 minutes = 900 seconds
time_waiting = time_epoc / max_tweets<jupyter_output><empty_output><jupyter_text>RestrictionsThe standard API rate limits described in the [Twitter developer's documents](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets) related with GET (read) endpoints are shown in the next table.  Note that endpoints not listed in the chart default to 15 requests per allotted user. All request windows are *15 minutes in length*.  These rate limits apply to the standard API endpoints only.| Endpoint | Resource family | Requests / window (user auth) | Requests / window (app auth) ||---|:--:|:---:|:---:||GET search/tweets | search | 180 | 450 |Also, the search index has a 7-day limit, which means that no tweets will be found for a date older than one week.  Retrieve tweets<jupyter_code># dictionary of dictionaries
results = collections.defaultdict(dict)
print("--- Retrieving tweets ---")
print("\nDATE       : ACCOUNT")
print("---------- : -------")
for ac in accounts:
    for d in range(0, len(dates)-1):
#     for da in dates:
        print(dates[d] + " : " + ac)
        results[ac][dates[d]] = api.GetSearch(
            raw_query = "q=" + ac + 
                        "%20&until=" + dates[d+1] +
                        "&count=" + str(num_tweets)
        )
        time.sleep(time_waiting)
print("\n--- Completed ---")
print("accounts:", accounts)
print("dates:", dates[:-1])
## Uncomment and run this cell to see the retrieved results
# results<jupyter_output><empty_output><jupyter_text>Example:<jupyter_code># First tweet by the public to the 'Canada' account on the last day of the period.
print('Account: Canada')
print('Date:', today)
results['CanadianPM'][today][0]<jupyter_output>Account: Canada
Date: 2020-08-28<jupyter_text>Load dictionaries (if exist) and save tweets<jupyter_code># Load 
#  - retrieve dictionaries if they exist,
#  - if they doesn't exist, it creates them,
#  - finally adds the tweets from the last 7 days.
results_all = collections.defaultdict(dict)
for ac in accounts: 
    try:
        results_all[ac] = np.load('../tweets/non-historic/tweets_' +ac+ '.npy',allow_pickle='TRUE').item()
    except:
        next
    for d in range(0, len(dates)-1):
        results_all[ac][dates[d]] = results[ac][dates[d]]
## Uncomment and run this cell to see all the retrieved results over time
# results_all
# Save
for ac in accounts:
    np.save('../tweets/non-historic/tweets_' + ac + '.npy', results_all[ac])
for ac in accounts:
    print(results_all[ac].keys())<jupyter_output>dict_keys(['2020-07-22', '2020-07-23', '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27', '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31', '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04', '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08', '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12', '2020-08-13', '2020-08-15', '2020-08-16', '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20', '2020-08-21', '2020-08-22', '2020-08-14', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28'])
dict_keys(['2020-07-22', '2020-07-23', '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27', '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31', '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04', '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08', '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12', '2020-08-13', '2020-08-15', '2020-08-16', '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20', '2020-08-21', '2020-08-22', '2020-08-14[...]
{'fraction_non_alphanumeric': 0.15455791638108293, 'fraction_numerical': 0.10572309801233722, 'mean_word_length': 3.8641666666666667, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '579128', 'license': 'MIT', 'n_tokens_mistral': 2388, 'n_tokens_neox': 1919, 'n_words': 586, 'path': 'src/twitter-search_v1_TwitterAPI.ipynb', 'repo_name': 'vcuspinera/Perception_of_Canadian_Government', 'size': '5846'}