<jupyter_start><jupyter_text>Step One Import the requests library<jupyter_code>import requests<jupyter_output><empty_output><jupyter_text>Step Two Send an HTTP request, get the response, and save in a variable<jupyter_code>response = requests.get("https://www.simplyrecipes.com/?s=asparagus+garlic")
type(response)<jupyter_output><empty_output><jupyter_text>Step Three Check the respomse status code to see if everything went as planned- Status code 200: the request-response cycle was successful- Any other status code if didn't work<jupyter_code>print(response.status_code)<jupyter_output>200<jupyter_text>Step Four Get the content of the responseand convert to utf-8 if necessary<jupyter_code>response.content#.decode('utf-8')<jupyter_output><empty_output><jupyter_text>Here we can see letter __b__ at the first position. It means that this is the representation called __byte string__ and we have to transform it to Unicode:<jupyter_code>response.content.decode('utf-8')<jupyter_output><empty_output><jupyter_text>Here we used the function __decode__ and gave it the __coding scheme__.The coding scheme can vary. There are lots of coding schemes. But __UTF8__ or __UTF16__ are the most common. So generally, if you're going to an English language web page, you can expect that the result is going to come back in UTF8 format. Let's found the subtitle "Welcome to Simply Recipes" from the main page and get number of byte where ir is situated.<jupyter_code>response.content.decode('utf-8').find('Welcome to Simply Recipes')<jupyter_output><empty_output><jupyter_text>JSONThe Python library __json__ deals with converting text to and from JSON<jupyter_code>import json

data_string = '[{"q":[2,3],"r":3.0,"s":"SS"}]'
python_data = json.loads(data_string)
print(type(python_data))
python_data<jupyter_output><class 'list'><jupyter_text>json.loads recursively decodes a string in JSON format into equivalent python objectsdata_string's outermost element is converted into a python listthe first element of that list is converted into a dictionarythe key of that dictionary is converted into a stringthe value of that dictionary is converted into a list of two integer elements<jupyter_code>print(type(data_string),type(python_data))
print(type(python_data[0]),python_data[0])
print(type(python_data[0]['s']),python_data[0]['s'])<jupyter_output><class 'str'> <class 'list'>
<class 'dict'> {'q': [2, 3], 'r': 3.0, 's': 'SS'}
<class 'str'> SS<jupyter_text>__json.loads__ will throw an exception if the format is incorrect<jupyter_code>#Correct
json.loads('"Hello"')<jupyter_output><empty_output><jupyter_text>The next code is wrong. And the reason I get that exception is because here I have a string, but it doesn't contain a JSON object. __To contain a JSON object, it should have a string inside it.__<jupyter_code>#Wrong
json.loads("Hello")<jupyter_output><empty_output><jupyter_text>requests library and JSONThe __Request Library__ has a function that automatically loads a JSON string into Python.<jupyter_code>address = 'KPI, Kyiv, Ukraine'
url = 'https://maps.googleapis.com/maps/api/geocode/json?address=%s'%(address)
response = requests.get(url).json()
print(type(response))<jupyter_output><class 'dict'><jupyter_text>Exception checking!So you should always be ready to face the fact that your code may not work. You may be expecting a JSON object back, but the server instead sends you a malformed JSON object. Be ready for that, too. So always check for exceptions. And that's what we're going to do now. We're going to make sure that we have everything properly checked over here.<jupyter_code>address="KPI,Kyiv,Ukraine"
url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
try:
    response = requests.get(url)
    if not response.status_code == 200:
        print("HTTP error",response.status_code)
    else:
        try:
            response_data = response.json()
        except:
            print("Response not in valid JSON format")
except:
    print("Something wemt wrong with requests.get()")
print(type(response_data))
# Let's see what URL looks like
url<jupyter_output><empty_output><jupyter_text>We get the response data. And we notice that it's of type Dictionary. So let's take a look at what this dictionary looks like.<jupyter_code>response_data<jupyter_output><empty_output><jupyter_text>Problem One Write a function that takes an address as an argument and returns a (latitude,longitude) tuple<jupyter_code>response_data['results']
for thing in response_data['results'][0]:
    print(thing)
response_data['results'][0]['geometry']
response_data['results'][0]['geometry']['location']
def get_lat_lng(address):
    import requests, time
    
    url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
    
    try:
        response = requests.get(url)
        if not response.status_code == 200:
            print('HTTP error',response.status_code)
        else:
            try:
                response_data = response.json()
            except:
                print('Response not valid JSON format')
    except:
        print('Something went wrong with requests.get')
    try:
        time.sleep(1)
        lat = response_data['results'][0]['geometry']['location']['lat']
        lng = response_data['results'][0]['geometry']['location']['lng']
    except:
        print('Try another one')
    return (lat,lng)
get_lat_lng('Sarny,Ukraine')<jupyter_output><empty_output><jupyter_text>Problem Two Extend the function so that it takes a possibly incomplete address as an argument and returns a list of tuples of the form (complete address, latitude, longitude)<jupyter_code>def get_lat_lng_incompl(address):
    #python code goes here
    import requests, time
    
    url="https://maps.googleapis.com/maps/api/geocode/json?address=%s" % (address)
    try:
        response = requests.get(url)
        if not response.status_code == 200:
            print("HTTP error",response.status_code)
        else:
            try:
                response_data = response.json()
            except:
                print("Response not in valid JSON format")
    except:
        print("Something went wrong with requests.get")
    try:
        time.sleep(1)
        propos_adr = []
        for i in range(len(response_data['results'])):
            adr = response_data['results'][i]['address_components'][0]['long_name']
            lat = response_data['results'][i]['geometry']['location']['lat']
            lng = response_data['results'][i]['geometry']['location']['lng']
            propos_adr.append((adr,lat,lng))
    except:
        print("Try another one.")
    return propos_adr    
get_lat_lng_incompl('Lon')<jupyter_output><empty_output><jupyter_text>XMLThe Python library __lxml__ deals with converting an xml string to python objects and vice versa<jupyter_code>data_string = """
<Bookstore>
   <Book ISBN="ISBN-13:978-1599620787" Price="15.23" Weight="1.5">
      <Title>New York Deco</Title>
      <Authors>
         <Author Residence="New York City">
            <First_Name>Richard</First_Name>
            <Last_Name>Berenholtz</Last_Name>
         </Author>
      </Authors>
   </Book>
   <Book ISBN="ISBN-13:978-1579128562" Price="15.80">
      <Remark>
      Five Hundred Buildings of New York and over one million other books are available for Amazon Kindle.
      </Remark>
      <Title>Five Hundred Buildings of New York</Title>
      <Authors>
         <Author Residence="Beijing">
            <First_Name>Bill</First_Name>
            <Last_Name>Harris</Last_Name>
         </Author>
         <Author Residence="New York City">
            <First_Name>Jorg</First_Name>
            <Last_Name>Brockmann</Last_Name>
         </Author>
      </Authors>
   </Book>
</Bookstore>
"""
from lxml import etree
root = etree.XML(data_string)
print(root.tag,type(root.tag))
print(etree.tostring(root,pretty_print=True).decode('utf-8'))<jupyter_output><Bookstore>
   <Book ISBN="ISBN-13:978-1599620787" Price="15.23" Weight="1.5">
      <Title>New York Deco</Title>
      <Authors>
         <Author Residence="New York City">
            <First_Name>Richard</First_Name>
            <Last_Name>Berenholtz</Last_Name>
         </Author>
      </Authors>
   </Book>
   <Book ISBN="ISBN-13:978-1579128562" Price="15.80">
      <Remark>
      Five Hundred Buildings of New York and over one million other books are available for Amazon Kindle.
      </Remark>
      <Title>Five Hundred Buildings of New York</Title>
      <Authors>
         <Author Residence="Beijing">
            <First_Name>Bill</First_Name>
            <Last_Name>Harris</Last_Name>
         </Author>
         <Author Residence="New York City">
            <First_Name>Jorg</First_Name>
            <Last_Name>Brockmann</Last_Name>
         </Author>
      </Authors>
   </Book>
</Bookstore><jupyter_text>Iterating over an XML tree- Use an iterator- The iterator will generate every tree element for a given subtree<jupyter_code>for element in root.iter():
    print(element)<jupyter_output><Element Bookstore at 0x1ef844bc508>
<Element Book at 0x1ef844c9708>
<Element Title at 0x1ef84561608>
<Element Authors at 0x1ef84561848>
<Element Author at 0x1ef844c9708>
<Element First_Name at 0x1ef84561608>
<Element Last_Name at 0x1ef84561848>
<Element Book at 0x1ef84561888>
<Element Remark at 0x1ef84561608>
<Element Title at 0x1ef844c9708>
<Element Authors at 0x1ef84561888>
<Element Author at 0x1ef84561608>
<Element First_Name at 0x1ef844c9708>
<Element Last_Name at 0x1ef84561888>
<Element Author at 0x1ef84561848>
<Element First_Name at 0x1ef84561608>
<Element Last_Name at 0x1ef844c9708><jupyter_text>Or just use the child in subtree construction:<jupyter_code>for child in root:
    print(child)<jupyter_output><Element Book at 0x1ef834d4508>
<Element Book at 0x1ef84561b88><jupyter_text>Accessing the tag<jupyter_code>for child in root:
    print(child.tag)<jupyter_output>Book
Book<jupyter_text>Using the iterator to get specific tags:- in the below example, only the __author__ tags are accessed- for each __author__ tag, the __.find()__ function access the First_Name and Last_Name tags- the __.find()__ function only looks at the children, not other descendants!!!- the __.text__ attribute prints the text in a __leaf node__<jupyter_code>for element in root.iter('Author'):
    print(element.find('First_Name').text,element.find('Last_Name').text)<jupyter_output><NAME>
<NAME>
<NAME><jupyter_text>Problem Find the last names of all authors in the tree 'root' using xpath<jupyter_code>for element in root.findall('Book/Title'):
    print(element.text)
for element in root.findall('Book/Authors/Author'):
    print(element.find('First_Name').text)<jupyter_output>Richard
Bill
Jorg<jupyter_text>Using values of attributes as filters- Example: Find the first name of the author of a book that weights 1.5 oz<jupyter_code>root.find('Book[@Weight="1.5"]/Authors/Author/First_Name').text
root.find('Book[@Price="15.80"]/Authors/Author/Last_Name').text<jupyter_output><empty_output><jupyter_text>Problem Print first and last names of all authors who live in New York City<jupyter_code>books = root.findall('Book')
print(books,type(books),sep='\n')
for i in range(len(books)):
    print(root.findall('Book/Authors/Author[@Residence="New York City"]/First_Name')[i].text,
          root.findall('Book/Authors/Author[@Residence="New York City"]/Last_Name')[i].text)<jupyter_output><NAME>tz
<NAME>
{'fraction_non_alphanumeric': 0.11082947294816886, 'fraction_numerical': 0.02735774844856219, 'mean_word_length': 4.417613636363637, 'pattern_counts': {'":': 3, '<': 180, '<?xml version=': 0, '>': 180, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '249467', 'license': 'MIT', 'n_tokens_mistral': 3775, 'n_tokens_neox': 3554, 'n_words': 992, 'path': 'All iPython Notebooks/_3_BAMM_101_W_3_Request_Web_Scrapping.ipynb', 'repo_name': '2series/Analytics-And-Python', 'size': '11484'}