<jupyter_start><jupyter_text>Mapping Geotagged Tweets -- Teacher Version This exercise is part of the *Teaching Privacy* curriculum, which you can find at https://teachingprivacy.org.**Teacher Notes: Full background for the exercise can be found in Teaching Privacy Module 1: You're Leaving Footprints. Note that, although the location is slightly obscure, these solutions are public and your students may find them.**  Part 1: Install the Tweepy LibraryTo access the Twitter API, you will use the Tweepy library. To install it, try running the following in your terminal:**pip install tweepy**If this does not work, check the readme file at https://github.com/tweepy/tweepy for the most up-to-date installation instructions. *For more on getting and using **pip**, check out the documentation or the w3schools tutorial.*Run the cell below to import the module.<jupyter_code>import tweepy
from tweepy import TweepError
import json<jupyter_output><empty_output><jupyter_text>Part 2: Create Twitter App and Get Access Tokens Create a Twitter App1. Go to https://apps.twitter.com and click 'Sign In'. If you don't have a Twitter account or don't want to use your current Twitter account, you will need to create one.2. Click on 'Create New App'.3. Give your app a Name, Description, and Website. For the website, you can use a placeholder (such as https://teachingprivacy.org).  Obtain Twitter API Keys When using APIs that require tokens and keys for authentication, it is common practice to have your keys in a separate JSON file, to protect yourself and the app's users. Your key file should not be posted in public repositories, and you should *never* share your keys. Create a new text file named **twitter_keys.json** with the following format:{    "api_key":"",    "api_secret":  "",    "access_token": "",    "access_token_secret": "" }1. Go into the app you created in the previous step and go to the 'Keys and Access Tokens' tab. 2. Copy and paste the tokens and keys for the corresponding variables into your JSON file.     a. You will need to click 'Create my access token' when you first create your app.     b. Make sure you copy and paste the tokens *inside* the quotation marks.3. Run the cell below to assign your keys to the **keys** variable.<jupyter_code>keys_file = 'twitter_keys.json'
with open(keys_file) as file:
    keys = json.load(file)<jupyter_output><empty_output><jupyter_text>Run the cell below to check whether you've correctly set up the keys.<jupyter_code>try:
    auth = tweepy.OAuthHandler(keys["api_key"], keys["api_secret"])
    auth.set_access_token(keys["access_token"], keys["access_token_secret"])
    api = tweepy.API(auth)
    print("You have correctly set up your API keys. Your username is:", api.auth.get_username())
except TweepError as e:
    print("Tweepy found an error. Revisit your twitter_keys.json file and make sure you have the correct keys.")<jupyter_output>You have correctly set up your API keys. Your username is: ImKarloss<jupyter_text>Part 3: Use the Twitter API with Tweepy to Gather Tweet MetadataNow that you've been authenticated to use the Twitter API, it's time to get acquainted with it.With help from the Tweepy documentation, find the 200 most-recent tweets for, say, Twitter user @stevewoz, in the cell below.**Teacher Note: The outputs below used @jack (<NAME>) in summer 2018. We instead suggest students look at Steve Wozniak first because he more frequently allows geotags from more interesting locations.***Hint: Look for a method to return the user timeline under 'API Reference'. http://docs.tweepy.org/en/v3.5.0/api.htmltimeline-methods*<jupyter_code>tweets = api.user_timeline(screen_name="jack", count=200)<jupyter_output><empty_output><jupyter_text>In the cell below, identify the data type of the previous tweet.<jupyter_code>type(tweets)<jupyter_output><empty_output><jupyter_text>The cell above should say we have a tweepy.models.ResultSet, which is a list of Status objects, or tweets. Confirm this in the cell below by indexing the first tweet and checking its type.<jupyter_code>first_tweet = tweets[0]
type(first_tweet)<jupyter_output><empty_output><jupyter_text>RESTful APIs typically send data in JSON format, the same format as our keys file. Using the '._json' attribute, convert the first tweet into a dictionary in the cell below. - *For some background, try this Stackoverflow post with more information about the json attribute: https://stackoverflow.com/questions/27900451/convert-tweepy-status-object-into-json * - *If you have not used dictionaries before, you can check out these videos: https://www.youtube.com/watch?v=daefaLgNkw0 or https://www.youtube.com/watch?v=XCcpzWs-CI4 *<jupyter_code>first_tweet_dict = first_tweet._json
first_tweet_dict<jupyter_output><empty_output><jupyter_text>Looking at the cell above, you should see that we got a nested dictionary. That's the common JSON format, but this in itself is not a JSON file. Explore the result to identify where the tweet's location is listed and under which keys. Use the cell below to print the first tweet's location.*Hint: Not all tweets have locations embedded. Find the first tweet's 'place' tag.*<jupyter_code>first_tweet_location = first_tweet_dict['place']
print('This tweet was tweeted from:', first_tweet_location)<jupyter_output>This tweet was tweeted from: None<jupyter_text>Part 4: Gather Tweet LocationsIn the cell below, find the locations for all the tweets we obtained. *Hint: Not all tweets are geotagged, so figure out how to only append actual tweet locations to the list, and ignore those with no location.*<jupyter_code>locations = []
tweets_with_location = []
for tweet in tweets:
    current_tweet = tweet._json['place']
    if current_tweet is not None:
        tweets_with_location.append(current_tweet)
        locations.append(current_tweet['full_name'])
locations<jupyter_output><empty_output><jupyter_text>Part 5: Install Plotting Libraries We will use the **geoplotlib** library to visualize tweet locations. Geoplotlib requires two other libraries, **NumPy** and **pyglet**, so you will need to install those too, using the following 3 *separate* commands in your terminal:**pip install numpy pip install pyglet pip install geoplotlib **When you're done, run the cell below to import Tweepy and all other necessary Python modules.<jupyter_code>import geoplotlib<jupyter_output><empty_output><jupyter_text>Part 6: Visualize Tweet LocationsNow that we've stored the locations of the user's tweets, we can create a visualization.For each tweet with a location, Twitter stores four pairs of latitude and longitude coordinates, to create a bounding box. For each tweet, store the first pair from each bounding box in an array named 'coords'.<jupyter_code>coords = []
for tweet in tweets_with_location:
    coords.append(tweet['bounding_box']['coordinates'][0][0])<jupyter_output><empty_output><jupyter_text>Geoplotlib has a utils.DataAccessObject that takes in a dictionary or pandas dataframe to create a DataAccessObject. This is the data type the library uses to create its maps.Create a dictionary with three keys: latitude, longitude, and the name of the city, in that order. For each key, the value should be a list with the corresponding values. You should already have the necessary values in previously assigned arrays.Next, use the utils.DataAccessObject method to create the DataAccessObject and use the .dot method to create a dot density map.*Hint: After creating the dot density map, you'll need to call geoplotlib.show() to open a window with the map.*<jupyter_code>lat, lon, name = [], [], locations
lat = [coordinate[1] for coordinate in coords]
lon = [coordinate[0] for coordinate in coords]
loc = {'lat': lat, 'lon': lon, 'name': name}
geo_loc = geoplotlib.utils.DataAccessObject(loc)
geoplotlib.dot(geo_loc)
geoplotlib.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07604272634791455, 'fraction_numerical': 0.00546795523906409, 'mean_word_length': 5.373581847649919, 'pattern_counts': {'":': 4, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '234126', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2282, 'n_tokens_neox': 2174, 'n_words': 987, 'path': 'Mapping_Geotagged_Tweets_Solutions.ipynb', 'repo_name': 'jbernd/AMM-Bits', 'size': '7869'}