<jupyter_start><jupyter_text>Assignment 5: Web Scraping and Big Data  Part 1: Scraping CraigslistIn this part, we'll be extracting information on apartments from Craigslist search results. You'll be using BeautifulSoup to extract the relevant information from the HTML text.   Getting the HTMLFirst we need to figure out how to submit a query to Craigslist. As with many websites, one way you can do this is simply by constructing the proper URL and sending it to Craigslist. Here's a sample URL that is returned after manually typing in a search to Craigslist: http://philadelphia.craigslist.org/search/apa?bedrooms=1&pets_cat=1&pets_dog=1&is_furnished=1<jupyter_code>import requests
url_base = 'http://philadelphia.craigslist.org/search/apa'
params = {'bedrooms': 1}
rsp = requests.get(url_base, params=params)
# Note that requests automatically created the right URL
print(rsp.url)<jupyter_output>https://philadelphia.craigslist.org/search/apa?bedrooms=1<jupyter_text>1.1 Parse the HTML- Use BeautifulSoup to parse the HTML response.- Use your browser's Web Inspector to identify the HTML element that holds the information on each apartment listing.- Use BeautifulSoup to extract these elements from the HTML.You should have a list of 120 elements, where each element is the listing for a specific apartment on the search page.<jupyter_code>from bs4 import BeautifulSoup
soup = BeautifulSoup(rsp.content, 'html.parser')
rows = soup.select('.result-row')
len(rows)<jupyter_output><empty_output><jupyter_text>1.2 Find the relevant pieces of informationWe will now focus on the first element in the list of 120 apartments. Use the `prettify()` function to print out the HTML for this first element. From this HTML, identify the HTML elements that hold:- The apartment price- The number of bedrooms and square footage (this will be in a single element)- The apartment title- The datetime string of the posting, e.g., '2019-03-23 12:07'For the first apartment, print out each of these pieces of information, using BeautifulSoup to select the proper elements.<jupyter_code>row = rows[0]
print(row.prettify())
price = row.select_one('.result-price').text
housing = row.select_one('.housing').text
title = row.select_one('.result-title.hdrlnk').text
datetime = row.select_one('.result-date')['datetime']
print(price,housing,title,datetime)<jupyter_output>$1420 
                    1br -
                    686ft2 -
                 Free Parking, Trash Pickup, Upgraded Apartments 2019-10-31 03:34<jupyter_text>1.3 Functions to format the results In this section, you'll create two functions that take the price and time results from the last section and format them properly.<jupyter_code>import re
from datetime import datetime
def format_size_and_bedrooms(size_string):
    """
    Extract size and number of bedrooms from the raw
    text, using regular expressions
    """
    split = re.findall("\n(.*?) -", size_string)
    
    # both size and bedrooms are listed
    if len(split) == 2:
        n_brs = split[0].strip().replace('br', '')
        this_size = split[1].strip().replace('ft2', '')
    # only bedrooms is listed
    elif 'br' in split[0]:
        n_brs = split[0].strip().replace('br', '')
        this_size = np.nan
    # only size is listed
    elif 'ft2' in split[0]:
        # It's the size
        this_size = split[0].strip().replace('ft2', '')
        n_brs = np.nan
    
    # return floats
    return float(this_size), float(n_brs)
def format_price(price_string):
    # Format the price string and return a float
    this_price = price_string.strip().replace('$', '')
    # This will involve using the string.strip() function to 
    # remove unwanted characters
    return float(this_price)
def format_time(date_string):
    # Return a Datetime object from the datetime string
    # Format: 2019-10-27 17:20
    return datetime.strptime(date_string,'%Y-%m-%d %H:%M')<jupyter_output><empty_output><jupyter_text>1.4: Putting it all togetherIn this part, you'll complete the code block below using results from previous parts. The code will loop over 4 pages of search results and scrape data for about 500 apartments.The outer for loop will loop over 4 pages of search results. The inner for loop will loop over the 120 apartments listed on each search page.After filling in the missing pieces and executing the code cell, you should have a Data Frame called `results` that holds the data for 480 apartment listings.<jupyter_code>import numpy as np
import pandas as pd
from time import sleep
results = []

# search in batches of 120 for 5 pages
# NOTE: you will get temporarily banned if running more than ~5 pages or so
# the API limits are more leninient during off-peak times, and you can try
# experimenting with more pages
max_pages = 5
results_per_page = 120
search_indices = np.arange(0, max_pages*results_per_page, results_per_page) 
url = 'http://philadelphia.craigslist.org/search/apa'

# loop over each page of search results
for i, s in enumerate(search_indices):
    print('Processing page %s...' % (i+1) )
    
    # get the response
    resp = requests.get(url, params={'bedrooms': 1, 's': s})
    
    # YOUR CODE: get the list of all aparements
    # This is the same code from Part 1.1
    # It should be a list of 120 apartments
    soup2 = BeautifulSoup(resp.content, 'html.parser')
    apts = soup2.select('.result-row')
    print("number of apartments = ", len(apts))

    # loop over each apartment in the list
    page_results = []
    for apt in apts:
        
        # YOUR CODE: the bedrooms/size string
        sizes_brs = apt.select_one('.housing').text

        # YOUR CODE: the title string
        title = apt.select_one('.result-title.hdrlnk').text
        
        # YOUR CODE: the price string
        price = apt.select_one('.result-price').text

        # YOUR CODE: the time string
        dtime = apt.select_one('.result-date')['datetime']
      
        # format using functions from Part 1.3
        sizes, brs = format_size_and_bedrooms(sizes_brs)
        price = format_price(price)
        dtime = format_time(dtime)
        
        # save the result
        page_results.append([dtime, price, sizes, brs, title])
        
    
    # create a dataframe and save
    col_names = ['time', 'price', 'size', 'brs', 'title']
    df = pd.DataFrame(page_results, columns=col_names)
    results.append(df)
    
    print("sleeping for 30 seconds between calls")
    sleep(30)
    
# Finally, concatenate all the results
results = pd.concat(results, axis=0)
results.head()<jupyter_output><empty_output><jupyter_text>1.5: Plotting the distribution of pricesUse matplotlib's `hist()` function to make two histograms for:- Apartment prices- Apartment prices per square foot (price / size)Make sure to add labels to the respective axes and a title describing the plot.<jupyter_code>import matplotlib.pyplot as plt
# Histogram for apartment prices
# Initialize
fig, ax = plt.subplots(figsize=(8,6))

# Plot a quick histogram
ax.hist(results['price'], bins='auto')

# Format
ax.set_xlabel("Prices", fontsize=18)
ax.set_ylabel("Number of Apartments", fontsize=18)
plt.title('Histogram for Apartment Prices', fontsize=15)
# Histogram for apartment price per square foot
results['price_unit'] = results['price'] / results['size']

fig, ax = plt.subplots(figsize=(8,6))
ax.hist(results['price_unit'], bins='auto')

ax.set_xlabel("Price per Square Foot", fontsize=18)
ax.set_ylabel("Number of Apartments", fontsize=18)
plt.title('Histogram for Apartment Prices per Square Foot', fontsize=15)<jupyter_output>C:\Users\USER\.conda\envs\musa-620\lib\site-packages\numpy\lib\histograms.py:391: RuntimeWarning: invalid value encountered in greater_equal
  keep = (a >= first_edge)
C:\Users\USER\.conda\envs\musa-620\lib\site-packages\numpy\lib\histograms.py:392: RuntimeWarning: invalid value encountered in less_equal
  keep &= (a <= last_edge)
C:\Users\USER\.conda\envs\musa-620\lib\site-packages\numpy\lib\histograms.py:824: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
C:\Users\USER\.conda\envs\musa-620\lib\site-packages\numpy\lib\histograms.py:825: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>1.6 Comparing prices for different sizes Use `altair` to explore the relationship between price, size, and number of bedrooms. Make an interactive scatter plot of price (x-axis) vs. size (y-axis), with the points colored by the number of bedrooms. Make sure the plot is interactive (zoom-able and pan-able) and add a tooltip with all of the columns in our scraped data frame.With this sort of plot, you can quickly see the outlier apartments in terms of size and price.<jupyter_code>import altair as alt
alt.renderers.enable('notebook')
# initialize the chart with the data
chart = alt.Chart(results)

# define what kind of marks to use
chart = chart.mark_circle(size=60)

# encode the visual channels
chart = chart.encode(
    x="price",
    y="size",
    color="brs",
    tooltip=["title","price", "size", "brs", "time"],
)

# make the chart interactive
chart.interactive()<jupyter_output><empty_output><jupyter_text>Part 2: Making an animated map with DatashaderIn this part, you will load a "large" data set of your choice and make an animated map of the data using datashader. There are several good options available from Open Data Philly, but you are welcome to choose a different source, as long it meets the requirements below. DeliverableYour final result should be a GIF animating changes in the data set in time. This GIF should be submitted to your repository, and the code used to produce the GIF should be in the notebook. Notes- For best results with datashader, the dataset should be at least 500,000 rows in size. If you are having difficulty finding a data set large enough, please email me.- The data must have a date time column, so that you can animate the data in time.- You can use either dask or pandas to load the data. If you're data set is approaching the size of your machine's memory, you will want to use dask.- Recommendations from Open Data Philly include:     - [311 Requests](https://www.opendataphilly.org/dataset/311-service-and-information-requests)    - [Crime Incidents](https://www.opendataphilly.org/dataset/crime-incidents)    - [L&I Code Violations](https://www.opendataphilly.org/dataset/licenses-and-inspections-violations)    - [Parking Violations](https://www.opendataphilly.org/dataset/parking-violations)    - [Real Estate Transfers](https://www.opendataphilly.org/dataset/real-estate-transfers)- You can animate the data by the hour or by the year, e.g., parking violations by hour of day, or parking violations by year.**Remember: when using datashader your data's coordinates should be in Web Mercator (EPSG coe 3857).**<jupyter_code>import dask.dataframe as dd
import datashader as ds
import datashader.transfer_functions as tf

# Color map imports
from colorcet import fire
# Load a data frame directly with dask
df = dd.read_csv('C:/Users/USER/MUSA620/assignment-5-master/incidents_part1_part2.csv', assume_missing=True, dtype={'psa': 'object'})
df.head()
# Remove rows with missing geometries
df = df.loc[df['point_x'].notnull()]
# Convert lat/lng to Web Mercator coordinates (x, y)
from datashader.utils import lnglat_to_meters
df['x'], df['y'] = lnglat_to_meters(df['point_x'], df['point_y'])
df.head()
# Get the x/y range for Philadelphia
Philly = (( -75.28,  -74.96), (39.86, 40.14))
x_range, y_range = lnglat_to_meters(Philly[0], Philly[1])
print(x_range, y_range)
plot_width  = int(750)
plot_height = int(plot_width//1.2)

def create_image(df, x_range, y_range, w=plot_width, h=plot_height, cmap=fire):
    # create the canvas
    cvs = ds.Canvas(plot_width=w, plot_height=h, x_range=x_range, y_range=y_range)
    # Aggregate the points of incidents
    agg = cvs.points(df, 'x', 'y', agg=ds.count())
    # shade
    img = tf.shade(agg, cmap=cmap, how='eq_hist')
    # return an PIL image
    return tf.set_background(img, "black").to_pil()

def convert_to_12hour(hr24):
    from datetime import datetime
    d = datetime.strptime(str(hr24), "%H")
    return d.strftime("%I %p")

def plot_dropoffs_by_hour(fig, data_all_hours, hour, x_range, y_range):
    # trim to the specific hour
    df_this_hour = data_all_hours.loc[data_all_hours["hour_"] == hour]
    # create the datashaded image
    img = create_image(df_this_hour, x_range, y_range)
    # plot the image on a matplotlib axes
    plt.clf()
    ax = fig.gca()
    ax.imshow(img, extent=[x_range[0], x_range[1], y_range[0], y_range[1]])
    ax.set_axis_off()
    # add a text label for the hour
    ax.text(
        0.05,
        0.9,
        convert_to_12hour(hour),
        color="white",
        fontsize=30,
        ha="left",
        transform=ax.transAxes,
    )
    # add title and my name
    ax.text(
        0.9,
        0.25,
        'Crimes',
        color="#E67E22",
        fontsize=25,
        ha="right",
        transform=ax.transAxes,
    )
    ax.text(
        0.9,
        0.2,
        'in Philadelphia',
        color="#E67E22",
        fontsize=25,
        ha="right",
        transform=ax.transAxes,
    )
    ax.text(
        0.9,
        0.15,
        'By Ann',
        color="white",
        fontsize=15,
        ha="right",
        transform=ax.transAxes,
    )   
    # draw the figure and return the image
    fig.canvas.draw()
    image = np.frombuffer(fig.canvas.tostring_rgb(), dtype="uint8")
    image = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))
    return image
import imageio
# create a figure
fig, ax = plt.subplots(figsize=(10,10), facecolor='black')

# Create an image for each hour
imgs = []
for hour in range(24):
    img = plot_dropoffs_by_hour(fig, df, hour, x_range=x_range, y_range=y_range)
    imgs.append(img)
    
# Combing the images for each hour into a single GIF
imageio.mimsave('crimes_name.gif', imgs, fps=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10340632603406326, 'fraction_numerical': 0.018963789895520253, 'mean_word_length': 3.8507462686567164, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 6, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '82474', 'license': 'Unlicense', 'n_tokens_mistral': 4507, 'n_tokens_neox': 4280, 'n_words': 1457, 'path': 'files/MUSA620_Yian Wang.ipynb', 'repo_name': 'isannwang/isyianwang.github.io', 'size': '13974'}