<jupyter_start><jupyter_text>What Can You Learn from Cross-Site Aggregation? -- Teacher Version This exercise is part of the *Teaching Privacy* curriculum, which you can find at https://teachingprivacy.org.**Teacher Notes: Full background for the exercise can be found in Teaching Privacy Module 3: Information Is Valuable. Note that, although the location is slightly obscure, these solutions are public and your students may find them.**In this exercise, you will aggregate data about a character named <NAME> from Tumblr and Twitter:https://twitter.com/kaiperochttps://kaiperoc.tumblr.com  Part 1: Install the API Libraries Tumblr APITo access the Tumblr API, you will use the PyTumblr library. To install it, run the following in your terminal:**sudo pip install oauth oauth2 pytumblr**If this doesn't work, check the PyTumblr readme file for the most up-to-date installation instructions. *For more on getting and using **pip**, check out the documentation or the w3schools tutorial.*  Twitter APITo access the Twitter API, you will use the Tweepy library. To install it, try running the following in your terminal:**pip install tweepy**If this doesn't work, check the readme file at https://github.com/tweepy/tweepy for the most up-to-date installation instructions.   ImportNow run the cell below to import the appropriate packages.<jupyter_code>import pytumblr
import tweepy
from tweepy import TweepError
import json
import re<jupyter_output><empty_output><jupyter_text>Part 2: Create Tumblr App and Get Access Tokens Create a Tumblr App1. Go to https://www.tumblr.com/oauth/apps. You will need to create a Tumblr account if you don't have one already or don't want to use your existing account for this purpose.2. Click on 'Register application'.3. Fill out the required fields. For the 'Application website' and 'Callback URL', you can use a placeholder (such as https://teachingprivacy.org); note that the URLs must match.  Obtain Tumblr API KeysGo into the app you created. You should see your oauth consumer key and your secret key.Go to https://api.tumblr.com/console and copy and paste these keys into the appropriate fields to get your oauth token and oauth secret key.When using APIs that require tokens and keys for authentication, it is common practice to keep your keys in a separate JSON file, to protect yourself and the app's users. Your key file should not be posted in public repositories, and you should *never* share your keys. Create a new file called **tumblr_keys.json** with the format below. Paste your keys between the empty quotation marks. (Make sure you don't overwrite the quotation marks!)  {    "api_key": ""    "api_secret":  "",    "oauth_token": "",    "oauth_secret": "" }  Part 3: Create Twitter App and Get Access Tokens Create a Twitter App1. Go to https://apps.twitter.com and click 'Sign In'. If you don't have a Twitter account or don't want to use your current Twitter account, you will need to create one.2. Click on 'Create New App'.3. Give your app a Name, Description, and Website. For the website, you can use a placeholder (such as https://teachingprivacy.org).  Obtain Twitter API Keys Create a new text file named **twitter_keys.json** with the following format:{    "consumer_key":"",    "consumer_secret":  "",    "access_token": "",    "access_token_secret": "" }1. Go into the app you created in the previous step and go to the 'Keys and Access Tokens' tab. 2. Copy and paste the tokens and keys for the corresponding variables into your JSON file.     a. You will need to click 'Create my access token' when you first create your app.     b. Make sure you copy and paste the tokens *inside* the quotation marks.  Part 4: Assign and Verify Your API Keys Assign the KeysRun the cell below to assign your keys to the **keys** variable.<jupyter_code>twitter_keys_file = 'twitter_keys.json'
tumblr_keys_file = 'tumblr_keys.json'
with open(twitter_keys_file) as file:
    twitter_keys = json.load(file)
with open(tumblr_keys_file) as file:
    tumblr_keys = json.load(file)<jupyter_output><empty_output><jupyter_text>Establish Your Tumblr APINow that you have set up your keys, run the cell below to establish your Tumblr API.<jupyter_code>client = pytumblr.TumblrRestClient(
    tumblr_keys['api_key'],
    tumblr_keys['api_secret'],
    tumblr_keys['oauth_token'],
    tumblr_keys['oauth_secret'],
)
client.info()<jupyter_output><empty_output><jupyter_text>Check Twitter KeysRun the cell below to check whether you have set up the Twitter keys correctly.<jupyter_code>try:
    auth = tweepy.OAuthHandler(twitter_keys["api_key"], twitter_keys["api_secret"])
    auth.set_access_token(twitter_keys["access_token"], twitter_keys["access_token_secret"])
    api = tweepy.API(auth)
    print("You have correctly set up your API keys. Your username is:", api.auth.get_username())
except TweepError as e:
    print("Tweepy found an error. Revisit your keys.json file and make sure you have the correct keys.")<jupyter_output>You have correctly set up your API keys. Your username is: ImKarloss<jupyter_text>Part 5: Gather Some DataNow that you've set up the API, let's see what data you can get from someone's social media accounts.  Gather Tumblr PostsWith the help of the PyTumblr documentation, grab all of Kai Peroc's Tumblr posts and store them in the array provided. The example in the cell below contains a line that uses regular expressions (regex) to remove the html tags in posts. *Never dealt with regular expressions? Check out this tutorial.*<jupyter_code>regex = re.compile('<.*?>')
old = "This string has an <html tag>."
new = re.sub(regex, '', old)
print(new)<jupyter_output>This string has an .<jupyter_text>*Hints:**1. Look for the .posts method in the documentation.**2. When looking for posts, note that not all types of posts will have the same tags. Look in the 'body' or 'caption' tags.**3. To get all the posts, store them in a variable like **tumblr_call** when you call the .posts method and iterate through them with a for loop.*<jupyter_code>tumblr_posts = []
tumblr_call = client.posts("kaiperoc")
for post in tumblr_call['posts']:
    if 'body' in post:
        clean_post = re.sub(regex, '', post['body'])
        tumblr_posts.append(clean_post)
    else:
        clean_post = re.sub(regex, '', post['caption'])
        tumblr_posts.append(clean_post)
tumblr_posts<jupyter_output><empty_output><jupyter_text>Gather Twitter PostsNow, with help from the Tweepy documentation, grab <NAME>'s most-recent tweets and store them in the array provided.*Hint: Look for a method to return the user timeline under 'API Reference': http://docs.tweepy.org/en/v3.5.0/api.htmltimeline-methods *<jupyter_code>twitter_posts = []
twitter_call = api.user_timeline(screen_name="kaiperoc", count=100)
for tweet in twitter_call:
    twitter_posts.append(tweet._json['text'])
twitter_posts<jupyter_output><empty_output><jupyter_text>Part 6: Aggregate the DataNow that you have Kai's data from two different social media accounts, it's time to aggregate.<jupyter_code>agg_posts = []
for tweet in twitter_posts:
    agg_posts.append(tweet)
for tumblr_post in tumblr_posts:
    agg_posts.append(tumblr_post)

agg_posts<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08188757807078417, 'fraction_numerical': 0.0036086051353226927, 'mean_word_length': 4.896890343698854, 'pattern_counts': {'":': 8, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '437150', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2162, 'n_tokens_neox': 2035, 'n_words': 900, 'path': 'What_Can_You_Learn_from_Cross-Site_Aggregation_Solutions.ipynb', 'repo_name': 'jbernd/AMM-Bits', 'size': '7211'}