<jupyter_start><jupyter_text>Week_02: Extracting Data With Regular Expressions Finding Numbers in a HaystackIn this assignment you will read through and parse a file with text and numbers. You will extract all the numbers in the file and compute the sum of the numbers.Data FilesWe provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.Sample data: http://py4e-data.dr-chuck.net/regex_sum_42.txt (There are 90 values with a sum=445833)Actual data: http://py4e-data.dr-chuck.net/regex_sum_308160.txt (There are 99 values and the sum ends with 81)These links open in a new window. Make sure to save the file into the same folder as you will be writing your Python program. Note: Each student will have a distinct data file for the assignment - so only use your own data file for analysis.Data FormatThe file contains much of the text from the introduction of the textbook except that random numbers are inserted throughout the text. Here is a sample of the output you might see:Why should you learn to write programs? 774612 1929 8827Writing programs (or programming) is a very creative 7 and rewarding activity.  You can write programs for many reasons, ranging from making your living to solving8837 a difficult data analysis problem to having fun to helping 128someone else solve a problem.  This book assumes that everyone needs to know how to program ...The sum for the sample text above is 27486. The numbers can appear anywhere in the line. There can be any number of numbers in each line (including none).Handling The DataThe basic outline of this problem is to read the file, look for integers using the re.findall(), looking for a regular expression of '[0-9]+' and then converting the extracted strings to integers and summing up the integers. Optional: Just for FunThere are a number of different ways to approach this problem. While we don't recommend trying to write the most compact code possible, it can sometimes be a fun exercise. Here is a a redacted version of two-line version of this program using list comprehension:Python 2import reprint sum( [ ****** *** * in **********('[0-9]+',**************************.read()) ] )Python 3:import reprint( sum( [ ****** *** * in **********('[0-9]+',**************************.read()) ] ) )Please don't waste a lot of time trying to figure out the shortest solution until you have completed the homework. List comprehension is mentioned in Chapter 10 and the read() method is covered in Chapter 7.<jupyter_code>path = "../../../00_Common_Data_Sets/regex_sum_42.txt"
path2 = "../../../00_Common_Data_Sets/regex_sum_308160.txt"

hand2 = open(path2, "r")

hand1 = open(path, "r")
print( sum( [ int(num) for num in re.findall('[0-9]+',hand1.read()) ] ) )
print( sum( [ int(num) for num in re.findall('[0-9]+',hand2.read()) ] ) )<jupyter_output><empty_output><jupyter_text>Week_03: Networks and Sockets<jupyter_code>Exploring the HyperText Transport Protocol

You are to retrieve the following document using the HTTP protocol in a way that you can examine the HTTP Response headers.

http://data.pr4e.org/intro-short.txt
There are three ways that you might retrieve this web page and look at the response headers:

Preferred: Modify the socket1.py program to retrieve the above URL and print out the headers and data. Make sure to change the code to retrieve the above URL - the values are different for each URL.
Open the URL in a web browser with a developer console or FireBug and manually examine the headers that are returned.
Use the telnet program as shown in lecture to retrieve the headers and content.
Enter the header values in each of the fields below and press "Submit".

Last-Modified:

ETag:

Content-Length:

Cache-Control:

Content-Type:

import socket

mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('data.pr4e.org', 80))
cmd = 'GET http://data.pr4e.org/romeo.txt HTTP/1.0\r\n\r\n'.encode()
mysock.send(cmd)

while True:
    data = mysock.recv(512)
    if len(data) < 1:
        break
    print(data.decode(),end='')

mysock.close()
import socket

mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('data.pr4e.org', 80))
cmd = 'GET http://data.pr4e.org/intro-short.txt HTTP/1.0\r\n\r\n'.encode()
mysock.send(cmd)

while True:
    data = mysock.recv(512)
    if len(data) < 1:
        break
    print(data.decode(),end='')

mysock.close()
Week_04: Programs that Surf the Web<jupyter_output><empty_output><jupyter_text>Scraping Numbers from HTML using BeautifulSoupIn this assignment you will write a Python program similar to http://www.py4e.com/code3/urllink2.py. The program will use urllib to read the HTML from the data files below, and parse the data, extracting numbers and compute the sum of the numbers in the file.We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.Sample data: http://py4e-data.dr-chuck.net/comments_42.html (Sum=2553)Actual data: http://py4e-data.dr-chuck.net/comments_308162.html (Sum ends with 46)You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.Data FormatThe file is a table of names and comment counts. You can ignore most of the data in the file except for lines like the following:Modu90Kenzie88Hubert87You are to find all the  tags in the file and pull out the numbers from the tag and sum the numbers.Look at the sample code provided. It shows how to find all of a certain kind of tag, loop through the tags and extract the various aspects of the tags.... Retrieve all of the anchor tagstags = soup('a')for tag in tags:    Look at the parts of a tag   print 'TAG:',tag   print 'URL:',tag.get('href', None)   print 'Contents:',tag.contents[0]   print 'Attrs:',tag.attrsYou need to adjust this code to look for span tags and pull out the text content of the span tag, convert them to integers and add them up to complete the assignment.Sample Execution$ python3 solution.pyEnter - http://py4e-data.dr-chuck.net/comments_42.htmlCount 50Sum 2...Turning in the AssignmentEnter the sum from the actual data and your Python code below:Sum:  (ends with 46)<jupyter_code># To run this, download the BeautifulSoup zip file
# http://www.py4e.com/code3/bs4.zip

# and unzip it in the same directory as this file

from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl
# http://py4e-data.dr-chuck.net/comments_42.html
# http://py4e-data.dr-chuck.net/comments_308162.html
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
if len(url) < 1:
    url = "http://py4e-data.dr-chuck.net/comments_42.html"
html = urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, "html.parser")
#print(soup)
# Retrieve all of the anchor tags
tags = soup('span')
num_list = []
for tag in tags:
    # Look at the parts of a tag
    #print('TAG:', tag)
    #print('URL:', tag.get('href', None))
    #print('Contents:', tag.contents[0])
    #print('Attrs:', tag.attrs)
    num_list.append(int(tag.contents[0]))
    
total = sum(num_list)
total
from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl
# http://py4e-data.dr-chuck.net/comments_42.html
# http://py4e-data.dr-chuck.net/comments_308162.html
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
if len(url) < 1:
    url = "http://py4e-data.dr-chuck.net/comments_308162.html"
html = urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, "html.parser")

# Retrieve all of the anchor tags
tags = soup('span')
num_list = []
for tag in tags:
    # Look at the parts of a tag
    #print('TAG:', tag)
    #print('URL:', tag.get('href', None))
    #print('Contents:', tag.contents[0])
    #print('Attrs:', tag.attrs)
    num_list.append(int(tag.contents[0]))
    
total = sum(num_list)
print(total)<jupyter_output>Enter - 
3046<jupyter_text>Following Links in PythonIn this assignment you will write a Python program that expands on http://www.py4e.com/code3/urllinks.py. The program will use urllib to read the HTML from the data files below, extract the href= vaues from the anchor tags, scan for a tag that is in a particular position relative to the first name in the list, follow that link and repeat the process a number of times and report the last name you find.We provide two files for this assignment. One is a sample file where we give you the name for your testing and the other is the actual data you need to process for the assignmentSample problem: Start at http://py4e-data.dr-chuck.net/known_by_Fikret.htmlFind the link at position 3 (the first name is 1). Follow that link. Repeat this process 4 times. The answer is the last name that you retrieve.Sequence of names: <NAME> <NAME> AnayahLast name in sequence: AnayahActual problem: Start at: http://py4e-data.dr-chuck.net/known_by_Ines.htmlFind the link at position 18 (the first name is 1). Follow that link. Repeat this process 7 times. The answer is the last name that you retrieve.Hint: The first character of the name of the last page that you will load is: MStrategyThe web pages tweak the height between the links and hide the page after a few seconds to make it difficult for you to do the assignment without writing a Python program. But frankly with a little effort and patience you can overcome these attempts to make it a little harder to complete the assignment without writing a Python program. But that is not the point. The point is to write a clever Python program to solve the program.Sample executionHere is a sample execution of a solution:$ python3 solution.pyEnter URL: http://py4e-data.dr-chuck.net/known_by_Fikret.htmlEnter count: 4Enter position: 3Retrieving: http://py4e-data.dr-chuck.net/known_by_Fikret.htmlRetrieving: http://py4e-data.dr-chuck.net/known_by_Montgomery.htmlRetrieving: http://py4e-data.dr-chuck.net/known_by_Mhairade.htmlRetrieving: http://py4e-data.dr-chuck.net/known_by_Butchi.htmlRetrieving: http://py4e-data.dr-chuck.net/known_by_Anayah.htmlThe answer to the assignment for this execution is "Anayah".Turning in the AssignmentEnter the last name retrieved and your Python code below:Name:  (name starts with M)<jupyter_code># http://py4e-data.dr-chuck.net/known_by_Fikret.html
# http://py4e-data.dr-chuck.net/known_by_Ines.html

from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl
import re

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
if len(url) < 1:
    url = "http://py4e-data.dr-chuck.net/known_by_Fikret.html"
processes = 4
for process in range(processes):
    html = urlopen(url, context=ctx).read()
    soup = BeautifulSoup(html, "html.parser")

    tags = soup("a")
    url = tags[2].get('href', None)
    name = tags[2].contents[0]
    print(url, name)
#name = url.contents
print(name)
from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl
import re

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
if len(url) < 1:
    url = "http://py4e-data.dr-chuck.net/known_by_Ines.html"
processes = 7
position = 17
for process in range(processes):
    html = urlopen(url, context=ctx).read()
    soup = BeautifulSoup(html, "html.parser")

    tags = soup("a")
    url = tags[position].get('href', None)
    name = tags[position].contents[0]
    print(url, name)
#name = url.contents
print(name)<jupyter_output>Enter - 
http://py4e-data.dr-chuck.net/known_by_Lennon.html Lennon
http://py4e-data.dr-chuck.net/known_by_Thiago.html Thiago
http://py4e-data.dr-chuck.net/known_by_Noah.html Noah
http://py4e-data.dr-chuck.net/known_by_Aleeshmah.html Aleeshmah
http://py4e-data.dr-chuck.net/known_by_Hollee.html Hollee
http://py4e-data.dr-chuck.net/known_by_Suzy.html Suzy
http://py4e-data.dr-chuck.net/known_by_Michael.html Michael
Michael<jupyter_text>Extracting Data from XMLIn this assignment you will write a Python program somewhat similar to http://www.py4e.com/code3/geoxml.py. The program will prompt for a URL, read the XML data from that URL using urllib and then parse and extract the comment counts from the XML data, compute the sum of the numbers in the file.We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.Sample data: http://py4e-data.dr-chuck.net/comments_42.xml (Sum=2553)Actual data: http://py4e-data.dr-chuck.net/comments_308164.xml (Sum ends with 83)You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.Data Format and ApproachThe data consists of a number of names and comment counts in XML as follows:  Matthias  97You are to look through all the  tags and find the  values sum the numbers. The closest sample code that shows how to parse XML is geoxml.py. But since the nesting of the elements in our data is different than the data we are parsing in that sample code you will have to make real changes to the code.To make the code a little simpler, you can use an XPath selector string to look through the entire tree of XML for any tag named 'count' with the following line of code:counts = tree.findall('.//count')Take a look at the Python ElementTree documentation and look for the supported XPath syntax for details. You could also work from the top of the XML down to the comments node and then loop through the child nodes of the comments node.Sample Execution$ python3 solution.pyEnter location: http://py4e-data.dr-chuck.net/comments_42.xmlRetrieving http://py4e-data.dr-chuck.net/comments_42.xmlRetrieved 4189 charactersCount: 50Sum: 2...Turning in the Assignment<jupyter_code>import urllib.request, urllib.parse, urllib.error
import xml.etree.ElementTree as ET
import ssl
import sys

serviceurl = 'http://py4e-data.dr-chuck.net/'

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

address = input('Enter location: ')
if len(address) < 1:
    quit()

url = serviceurl + address
print('Retrieving', url)
uh = urllib.request.urlopen(url, context=ctx)

data = uh.read()
    
print('Retrieved', len(data), 'characters') #, data, type(data))
#print(data.decode())
tree = ET.fromstring(data)
comments = tree.findall('comments/comment')

#for comment in comments:
#    print("name :", comment.find("name").text)
#    print("count :", comment.find("count").text)

nums = []

for comment in comments:
    num = int(comment.find("count").text)
    nums.append(num)

print(sum(nums))
#lat = results[0].find('geometry').find('location').find('lat').text
#lng = results[0].find('geometry').find('location').find('lng').text
#location = results[0].find('formatted_address').text

#print('lat', lat, 'lng', lng)
#print(location)
    
    
# Locations: http://py4e-data.dr-chuck.net/comments_42.xml
# http://py4e-data.dr-chuck.net/comments_308164.xml
import urllib.request, urllib.parse, urllib.error
import xml.etree.ElementTree as ET
import ssl

serviceurl = 'http://py4e-data.dr-chuck.net/'

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

address = input('Enter location: ')
if len(address) < 1:
    address = "comments_42.xml"

url = serviceurl + address
print('Retrieving', url)
uh = urllib.request.urlopen(url, context=ctx)

data = uh.read()
    
print('Retrieved', len(data), 'characters') 
tree = ET.fromstring(data)
comments = tree.findall('comments/comment')
nums = []
for comment in comments:
    num = int(comment.find("count").text)
    nums.append(num)

print(sum(nums))<jupyter_output>Enter location: comments_308164.xml
Retrieving http://py4e-data.dr-chuck.net/comments_308164.xml
Retrieved 4191 characters
2683<jupyter_text>Week_6 : Extracting Data from JSON Extracting Data from JSONIn this assignment you will write a Python program somewhat similar to http://www.py4e.com/code3/json2.py. The program will prompt for a URL, read the JSON data from that URL using urllib and then parse and extract the comment counts from the JSON data, compute the sum of the numbers in the file and enter the sum below:We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.Sample data: http://py4e-data.dr-chuck.net/comments_42.json (Sum=2553)Actual data: http://py4e-data.dr-chuck.net/comments_308165.json (Sum ends with 39)You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.Data FormatThe data consists of a number of names and comment counts in JSON as follows:{  comments: [    {      name: "Matthias"      count: 97    },    {      name: "Geomer"      count: 97    }    ...  ]}The closest sample code that shows how to parse JSON and extract a list is json2.py. You might also want to look at geoxml.py to see how to prompt for a URL and retrieve data from a URL.Sample Execution$ python3 solution.pyEnter location: http://py4e-data.dr-chuck.net/comments_42.jsonRetrieving http://py4e-data.dr-chuck.net/comments_42.jsonRetrieved 2733 charactersCount: 50Sum: 2...Turning in the AssignmentEnter the sum from the actual data and your Python code below:Sum:  (ends with 39<jupyter_code># http://www.py4e.com/code3/json2.py 
import json

data = '''
[
  { "id" : "001",
    "x" : "2",
    "name" : "Chuck"
  } ,
  { "id" : "009",
    "x" : "7",
    "name" : "Brent"
  }
]'''

info = json.loads(data)
print('User count:', len(info))

for item in info:
    print('Name', item['name'])
    print('Id', item['id'])
    print('Attribute', item['x'])
import urllib.request, urllib.parse, urllib.error
import json

# Ignore SSL certificate errors
#ctx = ssl.create_default_context()
#ctx.check_hostname = False
#ctx.verify_mode = ssl.CERT_NONE

url = input("Please enter Url: ")
#Sample data: http://py4e-data.dr-chuck.net/comments_42.json (Sum=2553)
#Actual data: http://py4e-data.dr-chuck.net/comments_308165.json (Sum ends with 39)

if len(url) < 1 : url = "http://py4e-data.dr-chuck.net/comments_42.json"
    
portal = urllib.request.urlopen(url) #, context=ctx)

data = portal.read().decode()
print("Data Type : ", type(data), len(data))

comments_dict = json.loads(data)
cment_lst = comments_dict["comments"]
num_lst = []
for comment in cment_lst:
    #print("Name:", comment["name"])
    #print("Count", comment["count"])
    num = int(comment["count"])
    num_lst.append(num)
    
print("Total =", sum(num_lst))
import urllib.request, urllib.parse, urllib.error
import json

url = input("Please enter Url: ")

if len(url) < 1 : url = "http://py4e-data.dr-chuck.net/comments_308165.json"
    
portal = urllib.request.urlopen(url)

data = portal.read().decode()

comments_dict = json.loads(data)
cment_lst = comments_dict["comments"]
num_lst = []
for comment in cment_lst:
    num = int(comment["count"])
    num_lst.append(num)
    
print("Total =", sum(num_lst))<jupyter_output>Please enter Url: 
Data Type :  <class 'str'> 2718
Total = 2439<jupyter_text>Calling a JSON APIIn this assignment you will write a Python program somewhat similar to http://www.py4e.com/code3/geojson.py. The program will prompt for a location, contact a web service and retrieve JSON for the web service and parse that data, and retrieve the first place_id from the JSON. A place ID is a textual identifier that uniquely identifies a place as within Google Maps.API End PointsTo complete this assignment, you should use this API endpoint that has a static subset of the Google Data:http://py4e-data.dr-chuck.net/json?This API uses the same parameter (address) as the Google API. This API also has no rate limit so you can test as often as you like. If you visit the URL with no parameters, you get "No address..." response.To call the API, you need to include a key= parameter and provide the address that you are requesting as the address= parameter that is properly URL encoded using the urllib.parse.urlencode() function as shown in http://www.py4e.com/code3/geojson.pyMake sure to check that your code is using the API endpoint is as shown above. You will get different results from the geojson and json endpoints so make sure you are using the same end point as this autograder is using.Test Data / Sample ExecutionYou can test to see if your program is working with a location of "South Federal University" which will have a place_id of "ChIJNeHD4p-540AR2Q0_ZjwmKJ8".$ python3 solution.pyEnter location: South Federal UniversityRetrieving http://...Retrieved 2412 charactersPlace id ChIJNeHD4p-540AR2Q0_ZjwmKJ8Turn InPlease run your program to find the place_id for this location: UMDMake sure to enter the name and case exactly as above and enter the place_id and your Python code below. Hint: The first seven characters of the place_id are "ChIJpy4 ..."Make sure to retreive the data from the URL specified above and not the normal Google API. Your program should work with the Google API - but the place_id may not match for this assignment.place_id:<jupyter_code>import urllib.request, urllib.parse, urllib.error
import json
import ssl

api_key = False
# If you have a Google Places API key, enter it here
# api_key = 'AIzaSy___IDByT70'
# https://developers.google.com/maps/documentation/geocoding/intro

if api_key is False:
    api_key = 42
    serviceurl = 'http://py4e-data.dr-chuck.net/json?'
else :
    serviceurl = 'https://maps.googleapis.com/maps/api/geocode/json?'

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

while True:
    address = input('Enter location: ')
    if len(address) < 1: break

    parms = dict()
    parms['address'] = address
    if api_key is not False: parms['key'] = api_key
    url = serviceurl + urllib.parse.urlencode(parms)

    print('Retrieving', url)
    uh = urllib.request.urlopen(url, context=ctx)
    data = uh.read().decode()
    print('Retrieved', len(data), 'characters')

    try:
        js = json.loads(data)
    except:
        js = None

    if not js or 'status' not in js or js['status'] != 'OK':
        print('==== Failure To Retrieve ====')
        #print(data)
        continue

    #print(json.dumps(js, indent=4))

    lat = js['results'][0]['geometry']['location']['lat']
    lng = js['results'][0]['geometry']['location']['lng']
    print('lat', lat, 'lng', lng)
    location = js['results'][0]['formatted_address']
    print(location)
    place_id = js['results'][0]['place_id']
    print(place_id)
import urllib.request, urllib.parse, urllib.error
import json
import ssl

api_key = 42
serviceurl = 'http://py4e-data.dr-chuck.net/json?'
address = input('Enter location: ')
if len(address) < 1: address = "UMD"

parms = dict()
parms['address'] = address
parms['key'] = api_key
url = serviceurl + urllib.parse.urlencode(parms)

print('Retrieving', url)
uh = urllib.request.urlopen(url)
data = uh.read().decode()
print('Retrieved', len(data), 'characters')

try:
    js = json.loads(data)
except:
    js = None

if not js or 'status' not in js or js['status'] != 'OK':
    print('==== Failure To Retrieve ====')

place_id = js['results'][0]['place_id']
print(place_id)<jupyter_output>Enter location: 
Retrieving http://py4e-data.dr-chuck.net/json?address=UMD&key=42
Retrieved 2469 characters
ChIJpy4Qmdisr1IRNC8MswLvJqA
{'fraction_non_alphanumeric': 0.10503364066782955, 'fraction_numerical': 0.018523133150593903, 'mean_word_length': 4.354458527907494, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 8, 'xml': 14}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '328284', 'license': 'MIT', 'n_tokens_mistral': 7722, 'n_tokens_neox': 7424, 'n_words': 2797, 'path': '03_Using Python to Access Web Data/Assignments.ipynb', 'repo_name': 'vblacklion/01_Python-for-Everybody_Specialization', 'size': '24098'}