<jupyter_start><jupyter_text>CDSW Week 2: Using APIs to Get Data From the Internet============================== Agenda------**Last session** we learned some basic command in python* Variables* Types* lists* dicts* for loops**This week** we will learn how to find and retrieve data from the internet**Next session** we will learn to do some Data Science on our data. Q: What's the difference between lists and dictionaries? A: Lists are ordered, dictionaries have keys and values.  Warmup------**Lists of dictionaries**<jupyter_code># create a dictionary with age, town, and state.
tommy = {'name':'Tommy', 'age': 27, 'town': 'Seattle', 'state': 'Washington'}
# create a list of all my siblings 
#You can make a list of dictionaries. That's how you order a lot of data from the internet. 

family = [
    {'name':'Tommy', 'age': 27, 'town': 'Seattle', 'state': 'Washington'},
    {'name':'Sue', 'age': 29, 'town': 'Marlboro', 'state': 'New York'},
    {'name':'Tammy', 'age': 21, 'town': 'Newark', 'state': 'New Jersey'},
    
]

for person in family: 
    print(person['name'] + " is " + str(person['age']) + " years old.")<jupyter_output>Tommy is 27 years old.
Sue is 29 years old.
Tammy is 21 years old.<jupyter_text>Python has a thing called "string format" that allows us to put Python in our string: This can save us from making typo errors. Note curly braces are used for dictionaries and for formatting. We ran of out parenthetical symbols! f" means I'm going to make a quote, but if you see someting in curly braces treat is as python.<jupyter_code>for person in family: 
    print(person['name'] + " is " + str(person['age']) + " years old.")
#The following line is another way of printing the line above, but (eventually) is less error prone:         
    print(f"{person['name']} is {person['age']} years old")
#If we remove f (for formatting) see what happens:         
    print("{person['name']} is {person['age']} years old")
      
# iterate through the list and print the age and state (using f'')<jupyter_output><empty_output><jupyter_text>*Creating strings from variables*    print(f'Inside the curly braces is python: {1 + 1}')    *Nesting dictionaries in lists*You can use lists as the values in dictionaries. You can use lists in lists!You can use dictionaries in lists!  Getting Data from APIs**API** means Application Programmer InterfaceAn API is a set of instructions that describe how computers can interact with each other to request and receive information.Some important questions we will ask that help us discover APIs is below.|Question | In technical terms ||:---------|:--------------------||Where is my data? | What is the domain? ||How do I learn what data is available?| Where is the documentation? ||How do I request specific data?| How do I formulate a URL for a specific purpose? ||How do I interpret the data?| What is the structure and format of the output?| Two questions we always ask when working with an API. 1) Where is the API located? What is the URL? 2) How do I know what's there? What is available to from the API,(and how do I get it?)This second question is best answered through the API Documentation?This will help us figur out the data structure.  **Let's walk through an example in the browser**PlaceKitten!In a browser, go to http://www.placekitten.com|In technical terms | PlaceKitten ||:---------|:--------------------||What is the domain? | http://www.placekitten.com ||Where is the documentation?| The documentation is on the home page. ||How do I formulate a URL for a specific purpose? | You put it in the url like http://www.placekitten/width/height ||What is the structure and format of the output?| It's an image! | If you are not a cat person, you can use https://www.fillmurray.com/ (It works much the same.)Inspecting the "documentation" for PlaceKitten we learn how it works:http://placekitten.com/200/300  gives us an image of a cat 200 x 300 pixels.http://placekitten.com/g/200/300 gives us an image of a cat 200 x 300 pixels greyscale.Now we understand this API!  Now let's try it in Python!  Accessing placekitten in pythonWe're going to use a special library called requests<jupyter_code>from IPython.display import Image  # This line lets you display images. We'll use that in a bit.


# This line lets you use python to download data from the web.
import requests

#Run this line. When it's works, nothing happens.
# Get a 200 by 300 image from placekitten.
response = requests.get("http://placekitten.com/500/500")

#Magically, Python just retreived that image and saved it to the variable 'response'.

  
# Look at the status code
#Hint try typing response. and hitting tab. A pop up will come up that gives you options.
#any varible will work with the tab to give you a context specific menu of options of what you can do with it. 
#If this comes back as '200' than it worked!
response.status_code
# print the content
#This will looks wierd because it's an image!
print(response.content)
#So let's tell Python this is an image instead:
# Use the Image function to display the image
Image(response.content)
#That looks better!  
#Status code and content are usually what you want from the API, 
#but you can also look at the 'header' which gives you some hint of what to expect.
response.headers
#If you want to see how Image works you can add ?? to the function we are curious about:
Image??

#Reading the API docuementation (and trial and error) is the main way we get to understand APIs.
# Use input to request width/height and print an image
#This gives us a series of increasingly larger cat images: 
for x in [200, 250, 300, 400, 450, 500]:
    response = requests.get(f"http://placekitten.com/{x}/{x}")
#diplay is encouraing Python to show us the image    
    display(Image(response.content))
# Let's change the height and width of the images. 
#Adding on to the above for loop, but renaming the variable names:

for width in [200, 250, 300, 400, 450, 500]:
    for height in [200, 250, 300, 400, 450, 500]:
        print (f"http://placekitten.com/{width}/{height}")
#Note about POLITENESS, it can be greedy to ask so much of an API.
#You might get an error if you ask for too much data too fast! 
#Check the documentation for clues about what is considered polite for a given API. 
# Let's add a little time, to our request so we are less demanding on the API host's servers:
#The code below adds 5 seconds between each request. A more polite request!

import time
for width in [200, 250, 300, 400, 450, 500]:
    for height in [200, 250, 300, 400, 450, 500]:
        url = (f"http://placekitten.com/{width}/{height}")
        response = requests.get(url)
        display(Image(response.content))
        time.sleep(5)
#Note the stop button at the top of the Jupyter notebook. 
#Use this when you get tired of loading kittens or anytime you want to stop 
#code that is currently executing in your notebook.<jupyter_output><empty_output><jupyter_text>* What happens when you add "g" to the URL?* Can you write a loop to show several images?  Example 2: Getting World TimesThis example introduces a slightly more complicated API. It also introduces **JSON** which is a very common data format.Our API is at http://worldtimeapi.org/ First we ask, where is the data we can access with this API? Where is the documentation for this API?Conveniently, in this case, it's on this APIs home page!<jupyter_code># Download list of time zones
time_response = requests.get("http://worldtimeapi.org/api/timezone")
print(time_response)
# Use the .json() function to get the response as a list or dict
# What did it return?
timezones = time_response.json()
print(timezones)
# Get the time for your time zone (America/Los_Angeles)
timezone = 'Europe/Zaporozhye'
url = f"http://worldtimeapi.org/api/timezone/{timezone}"
time_response = requests.get(url)
print(time_response.status_code)
print(time_response.json())
#importing the time module so we can control the rate of requests to the API servers
#with the sleep.time function

import time

print(response.status_code)

# What are unique values for time in American time zones?

# Write a function to print the times in all American time zones.

response_tz = requests.get("http://worldtimeapi.org/api/timezone/America")
time_zones = response_tz.json()
print(f"There are {len(time_zones)} in my set")

#for time_zone in time_zones:
#    url = f"http://worldtimeapi.org/api/timezone/{time_zone}"
#    response = requests.get(url)
#    time_data = response.json() 
#    current_dt = time_data['datetime']
#    #print(current_dt) - printed this before adding the next line, just to see what is working so far
#    print(f"{time_zone}: {current_dt}")
#    time.sleep(1)
    
  #  break  use break to test the loop, it will execute the loop once 

#YIKES! This gives a 403 error! Our data we are requesting is broken. We need to make our code a bit less 
# breakable due to data errors by adding if statements 


for time_zone in time_zones:
    url = f"http://worldtimeapi.org/api/timezone/{time_zone}"
    response = requests.get(url)
    status_code = response.status_code
    if status_code == 200: 
        time_data = response.json() 
        current_dt = time_data['datetime']
    #print(current_dt) - printed this before adding the next line, just to see what is working so far
        print(f"{time_zone}: {current_dt[:13]}")
    time.sleep(1)
    

#Challenge 1: truncate to the hour (otherwise the number of zones will = the number of requests to the API you make)
#(see cell below)

#Challenge 2: print only unique time zone values:

#The variable current_dt was set once in our loop above. 
type (current_dt)
#It's a string! Let's slice the string to truncate the datetime. 

current_dt[:13]

#We'll add this above to our loop above. 
#Challenge 2: print only unique time zone values:
#Modding the above cell... 

#importing the time module so we can control the rate of requests to the API servers
#with the sleep.time function

import time

print(response.status_code)

# What are unique values for time in American time zones?

# Write a function to print the times in all American time zones.

response_tz = requests.get("http://worldtimeapi.org/api/timezone/America")
time_zones = response_tz.json()
print(f"There are {len(time_zones)} in my set")

#A NEW VARIABLE TO TRACK WHAT WE'VE SEEN BEFORE
previously_seen_hours = {}

#for time_zone in time_zones:
#    url = f"http://worldtimeapi.org/api/timezone/{time_zone}"
#    response = requests.get(url)
#    time_data = response.json() 
#    current_dt = time_data['datetime']
#    #print(current_dt) - printed this before adding the next line, just to see what is working so far
#    print(f"{time_zone}: {current_dt}")
#    time.sleep(1)
    
  #  break  use break to test the loop, it will execute the loop once 

#YIKES! This gives a 403 error! Our data we are requesting is broken. We need to make our code a bit less 
# breakable due to data errors by adding if statements 


for time_zone in time_zones:
    url = f"http://worldtimeapi.org/api/timezone/{time_zone}"
    response = requests.get(url)
    status_code = response.status_code
    if status_code == 200: 
        time_data = response.json() 
        current_dt = time_data['datetime'][:13]
        if current_dt not in previously_seen_hours:
            print(f"{time_zone}: {current_dt}")
            previously_seen_hours[current_dt] = True
    else: 
        print(f"Failed for {url}  {status_code}")
        time.sleep(10)
    time.sleep(1)
    
    
    

#Notice it takes longer than a second for each line to print. 
#The loop is finding a lot of previously seen time zones and not printing them. 

#BUT also could be because we are getting 403 errors (MISSING DATA!)
#To investigate errors, add some else statements so we can see the status codes 
# The else shows a lot of 403 errors!! Maybe because we are calling the servers too frequently!
#Let's try adding some more time. Or maybe it's time to take a break!

# Challenge 3: print all timezones with the same time. 
That's it for new information this session! 

There are a lot of APIs out there to explore. Check them out!  

https://github.com/public-apis/public-apis<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09641512522507775, 'fraction_numerical': 0.015141594368963824, 'mean_word_length': 3.930992736077482, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '161695', 'license': 'MIT', 'n_tokens_mistral': 3744, 'n_tokens_neox': 3483, 'n_words': 1665, 'path': 'Lecture_2_APIs_ded.ipynb', 'repo_name': 'CommunityDataScienceCollective/lecture_notes-cdsw', 'size': '12218'}