<jupyter_start><jupyter_text>Surveying my local housing market via Redfin- toc: false- branch: master- badges: true- comments: false- categories: [personal, data science] Here's a somewhat practical example, getting a sense of your local housing market using some mostly-standard python libraries.I'm particularly interested in how overpriced the homes in my area are (relative to the tax-assessed value), but you could imagine adapting this code or doing periodic data extractions to summarize some trends in your own area. Brief technical summaryTo anyone who might try to use this code: - Start from the individual HTTP requests, then adapt those to whatever interests you- To download a list of home listings off redfin, [this example URL returns a CSV file with information for each listing](https://www.redfin.com/stingray/api/gis-csv?al=1&market=dc&min_stories=1&num_homes=350&ord=redfin-recommended-asc&page_number=1&region_id=12839&region_type=6&sf=1,2,3,5,6,7&status=9&uipt=1,2,3,4,5,6,7,8&v=8). I'd actually recommend reverse engineering this URL by going on redfin, searching for a neighborhood, applying any filters, and then copying the address of the "Download all" link. I think they mention this download limits you to 350 homes, but you can see that the `num_homes` param seems changeable- [This redfin package provides a convenient wrapper for some commeon requests to redfin](https://github.com/reteps/redfin). This package stores a lot of the relevant HTTP endpoints, so browse through the repoThe rest of the code involves parallelization and summarizing/cleaning all the redfin requests -- it might be slightly over-engineered and overly-specific to what I'm trying to do<jupyter_code>import requests
import json
import itertools as it
import io

import pandas as pd
from dask import delayed
from dask.distributed import Client, as_completed, wait
from redfin import Redfin<jupyter_output><empty_output><jupyter_text>Define some 'constants' that will help with making requests or just filtering columns we want<jupyter_code>request_headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng.*/*;q=0.8',
    'accept-encoding':'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.8',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Chrome/92.0.4515.130'
}
relevant_columns = [
    'ADDRESS', 'CITY', 'STATE OR PROVINCE', 'ZIP OR POSTAL CODE', "PRICE", "tax_assessed_value", 'overpriced'
]<jupyter_output><empty_output><jupyter_text>Because there are a lot of houses and each request can take a while, I'm going to use `dask` to parallelize many requests.Initially, I used `concurrent.futures`, but I preferred a nice visualizer UI to see the progress of my tasks. Also, the dask parallelization seemed _slightly_ faster?<jupyter_code>daskclient = Client()
redfinclient = Redfin()
daskclient<jupyter_output><empty_output><jupyter_text>Define the functions I'll be using<jupyter_code>def stitch_full_address(row):
    """ Given a house record from redfin, generate a full address for future querying """
    return row['ADDRESS'] + ', ' + row['CITY'] + " " + row['STATE OR PROVINCE']


def process_redfin_response(response, redfinclient):
    """ Given a response from redfin API, return the tax-assessed value
    
    Notes
    -----
    This can get messy because this response is deeply-nested JSON, and 
    there are many chances for us to fail at pulling tax values.
    In all the places where things can go wrong, I do a very sloppy check and 
    then return -1 if something broke
    """
    if response.status_code != 200:
        return -1
    else:
        resp_dict = json.loads(response.text[4:])
        if (
            (resp_dict.get('errorMessage', None) == 'Success') &
            ('exactMatch' in resp_dict['payload'])
        ): 
            # Pull property metadata
            url = resp_dict['payload']['exactMatch']['url']
            
            data = redfinclient.initial_info(url)['payload']
            if data['responseCode'] != 200:
                return -1
            property_id = data['propertyId']
            listing_id = data['listingId']
            info = redfinclient.below_the_fold(property_id)
            # Pull latest tax-assessed value
            if len(info['payload']['publicRecordsInfo']['allTaxInfo']) > 0:
                tax_assessment = (
                    pd.DataFrame(info['payload']['publicRecordsInfo']['allTaxInfo'])
                    .sort_values("rollYear", ascending=False)
                ).iloc[0]
                return tax_assessment.get('taxableLandValue', 0) + tax_assessment.get('taxableImprovementValue', 0)
            else:
                return -1
        else:
            return -1
        

def query_redfin_dask(session, redfinclient, address, headers=None, **kwargs):
    """ For a given address, query redfin and identify tax-assessed value
    
    This is the function we submit to the dask client
    """
    response = session.get(
        'https://redfin.com/stingray/do/location-autocomplete',
        params={
            'location': address,
            'v': 2,
            **kwargs
        },
        headers=headers
    )
    return {address: process_redfin_response(response, redfinclient)}


def compile_results(results):
    """ Aggregate the results from all the redfin requests into a single series
    
    Take a list of dictionaries (from the dask future objects), flatten them into one 
    dictionary, then turn into a pandas series
    """
    compiled = pd.Series(dict(it.chain.from_iterable(a.items() for a in results)), name='tax_assessed_value')
    
    return compiled<jupyter_output><empty_output><jupyter_text>Download a list of all home listings from redfin based on search parameters, do some small cleaning to generate full addresses.There might be good documentation on making this query, but there are definitely a few parameters I don't understand and only kept in here because they showed up when I searched on redfin.com and reverse-engineered the URL<jupyter_code>with requests.Session() as session:
    params = {
        'al': 1,
        'hoa': 150,
        'market': 'dc',
        'max_listing_approx_size': 3000,
        'min_listing_approx_size': 1700,
        'max_num_beds': 4,
        'max_price': 800_000,
        'num_baths': 2,
        'num_beds': 2,
        'num_homes': 450, 
        'page_number': 1,
        'region_id': 2965,
        'region_type': 5,
        'sf': '1,2,3,5,6,7',
        'status': 9,
        'uipt': '1,2,3,4,5,6,7,8',
        'v': 8
    }
    download = session.get(
        'https://www.redfin.com/stingray/api/gis-csv?', params=params,
        headers=request_headers
    )
df = (
    pd.read_csv(io.StringIO(download.content.decode("utf-8")))
    .assign(full_address=lambda x: x.apply(stitch_full_address, axis=1))
)<jupyter_output><empty_output><jupyter_text>For each home listing, download more-detailed information pertaining to the taxable land value and taxable improvement value.There's certainly a lot of other information you could pull from these queries, but I'll leave that as an exercise to the readerThis is technically broken into some different dask steps to scatter the `requests.Session` and `redfin.Client` objects, and then making a bunch of queries (with `dask.delayed` objects) to redfin for each particular address<jupyter_code>with requests.Session() as session:
    session_scattered = daskclient.scatter(session)
    redfinclient_scattered = daskclient.scatter(redfinclient)
    futures = [
        delayed(query_redfin_dask)(session_scattered, redfinclient_scattered, address, headers=request_headers)
        for address in df['full_address'].to_list()
    ]
    futures = daskclient.compute(futures) 
    completed_results = [result for a, result in as_completed(futures, raise_errors=False, with_results=True)]<jupyter_output><empty_output><jupyter_text>Join the tax-assessed values back to the original dataframe, compute how overpriced the property is<jupyter_code>processed_df = (
    df.merge(compile_results(completed_results), left_on='full_address', right_index=True)
    .assign(overpriced=lambda x: x['PRICE'] - x['tax_assessed_value'])
)<jupyter_output><empty_output><jupyter_text>Examine results, some information is maskedIn this particular example, some homes have really low tax-assessed values relative to actual price.I think this is because these homes are so new (built in this year) that there's no taxableImprovementValue yet, only taxableLandValue.As a result, these newly-built properties can appear extremely overpriced<jupyter_code>(
    processed_df
    .query('tax_assessed_value > 0')
    [relevant_columns]
    .sort_values('overpriced')
    .assign(**{
        'ADDRESS': 'SORRY',
        'CITY': 'I AM',
        'STATE OR PROVINCE': "MASKING",
        'ZIP OR POSTAL CODE': "THESE COLUMNS"
    })
)<jupyter_output><empty_output><jupyter_text>Basic statistics to summarize *over our particular housing sample*- Raw price values are skewed because of the filters I imposed when pulling the entier list of homes-for-sale from redfin- Look only at the homes for which I could succesfully identify the tax-assessed value- Look only at the homes that don't seem extremely overpriced. Homes that are extremely overpriced might just be due to the absence of taxableImprovementValue from new builds.- In the end, this is roughly the set of homes I'm interested in, so I can still continue with finding tax assessed values and evaluating how overpriced they are<jupyter_code>(
    processed_df
    .query('tax_assessed_value > 0 & overpriced < 200000')
    [['PRICE', 'tax_assessed_value', 'overpriced']]
    .describe()
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09368907131862493, 'fraction_numerical': 0.012416623909697281, 'mean_word_length': 3.789189189189189, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 40, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '305060', 'license': 'Apache-2.0', 'n_tokens_mistral': 2941, 'n_tokens_neox': 2779, 'n_words': 962, 'path': '_notebooks/2021-08-15-scrape-redfin1.ipynb', 'repo_name': 'ahy3nz/fastpayges', 'size': '9745'}