<jupyter_start><jupyter_text>Team 93: Exploratory Data Analysis To create our gentrification index, we will complete exploratory data anaylsis on LA County Assessor Rolls (2010 - 2020) to see how rail transit stations affect property values.<jupyter_code>#importing libraries + drive

!pip install pingouin
from google.colab import drive
drive.mount('/content/gdrive')
import numpy                 as np
import pandas                as pd
import matplotlib.pyplot     as plt
import seaborn               as sns
import pingouin as pg
import folium  #needed for interactive map
from folium.plugins import HeatMap
%matplotlib inline
sns.set()#sets seaborn theme parameters to their defaults<jupyter_output><empty_output><jupyter_text>LA County Assessor Data<jupyter_code>#loading assessor data 
df = pd.read_csv("gdrive/My Drive/Team 93 Capstone Project/data/Assessor_Parcels_Data_2010 to 2020.csv")
df<jupyter_output><empty_output><jupyter_text>Feature Engineering  To have a consistent list of zip codes across all datasets, we will filter the assessor dataset to 76 Los Angeles city zip codes. Additionally, we'll do feature engineering and create the following: *   A new feature that details the geographical (North, Northwest, West, South, and Central) location.*   A feaure that details the total number of rail transit stations in the location's zip code<jupyter_code>#dictionary of zip codes + their station count for feature engineering

station_zip_codes = {
    '90004' : 1,
    '90006' : 0,
    '90007' : 4,
    '90012' : 2,
    '90013' : 2,
    '90015' : 1,
    '90016' : 3,
    '90017' : 1,
    '90019' : 0,
    '90020' : 1,
    '90024' : 0,
    '90026' : 0,
    '90027' : 2,
    '90028' : 2,
    '90029' : 1,
    '90034' : 1,
    '90035' : 0,
    '90036' : 0,
    '90037' : 0,
    '90038': 0,
    '90039' : 0,
    '90041' : 0,
    '90042' : 1,
    '90045' : 1,
    '90064' : 3,
    '90065' : 1,
    '90068' : 0,
    '90094' : 0,
    '90211' : 0,
    '90212' : 0,
    '90272' : 0,
    '90731' : 0,
    '90732' : 0,
    '91303' : 0,
    '91306' : 0,
    '91307' : 0,
    '91316' : 0,
    '91324' : 0,
    '91325' : 0,
    '91335' : 0,
    '91343' : 0,
    '91344' : 0,
    '91356' : 0,
    '91367' : 0,
    '91401' : 0,
    '91403' : 0,
    '91405' : 0,
    '91406' : 0,
    '91411' : 0,
    '91423' : 0,
    '91601' : 1,
    '91604' : 1,
    '91605' : 0,
    '91606' : 0,
    '91607' : 0,
    '90025' : 0,
    '90046' : 0,
    '90047' : 0,
    '90048' : 2,
    '90049' : 0,
    '90066' : 0,
    '90069' : 0,
    '90210' : 0,
    '90230' : 0,
    '90245' : 0,
    '90291' : 0,
    '90292' : 0,
    '90293' : 0,
    '90501' : 0,
    '90717' : 0,
    '91214' : 0,
    '91304' : 0,
    '91311' : 0,
    '91342' : 0,
    '91364' : 0,
    '91505' : 0
}

#dictionary of each zip code's geographic locaition

area_zip_codes = {
    '90004' : 'West',
    '90006' : 'Central',
    '90007' : 'Central',
    '90012' : 'Central',
    '90013' : 'Central',
    '90015' : 'Central',
    '90016' : 'West',
    '90017' : 'Central',
    '90019' : 'West',
    '90020' : 'West',
    '90024' : 'West',
    '90026' : 'Central',
    '90027' : 'Central',
    '90028' : 'West',
    '90029' : 'Central',
    '90034' : 'West',
    '90035' : 'West',
    '90036' : 'West',
    '90037' : 'Central',
    '90038' : 'West',
    '90039' : 'Central',
    '90041' : 'Central',
    '90042' : 'Central',
    '90045' : 'West',
    '90064' : 'West',
    '90065' : 'Central',
    '90068' : 'West',
    '90094' : 'West',
    '90211' : 'West',
    '90212' : 'West',
    '90272' : 'West',
    '90731' : 'South',
    '90732' : 'South',
    '91303' : 'Northwest',
    '91306' : 'Northwest',
    '91307' : 'Northwest',
    '91316' : 'Northwest',
    '91324' : 'Northwest',
    '91325' : 'Northwest',
    '91335' : 'Northwest',
    '91343' : 'Northwest',
    '91344' : 'Northwest',
    '91356' : 'Northwest',
    '91367' : 'Northwest',
    '91401' : 'Northeast',
    '91403' : 'Northeast',
    '91405' : 'Northeast',
    '91406' : 'Northwest',
    '91411' : 'Northeast',
    '91423' : 'Northeast',
    '91601' : 'Northeast',
    '91604' : 'Northeast',
    '91605' : 'Northeast',
    '91606' : 'Northeast',
    '91607' : 'Northeast',
    '90025' : 'West',
    '90046' : 'West',
    '90047' : 'West',
    '90048' : 'West',
    '90049' : 'West',
    '90066' : 'West',
    '90069' : 'West',
    '90210' : 'West',
    '90230' : 'West',
    '90245' : 'West',
    '90291' : 'West',
    '90292' : 'West',
    '90293' : 'West',
    '90501' : 'South',
    '90717' : 'South',
    '91214' : 'Northeast',
    '91304' : 'Northwest',
    '91311' : 'Northwest',
    '91342' : 'Northeast',
    '91364' : 'Northeast',
    '91505' : 'Northeast'
}
#filter dataframe to only include the list of zipcodes above

assessor_data = df[df['zip_code'].isin(list(station_zip_codes.keys()))]
assessor_data
assessor_data = assessor_data.reset_index()
assessor_data = assessor_data.drop(columns = ["index","Unnamed: 0", "Unnamed: 0.1"])
assessor_data
#changing data types for consistency

assessor_data["year"] = assessor_data["year"].astype("str")
assessor_data["zip_code"] = assessor_data["zip_code"].astype("str")
assessor_data["train_stops"] = assessor_data["train_stops"].astype("category")
assessor_data["net_value"] = assessor_data["net_value"].astype("int")
#adding area column 

def find_area(x):
  if x in area_zip_codes.keys():
    return area_zip_codes[x] 

assessor_data['area']= assessor_data["zip_code"].apply(lambda x: find_area(x))
assessor_data<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis  Univariate Analysis: Net Property Values Across All Zip CodesFirst, we'll examine summary statistics to better understand the spread of the data across the entire dataset. Then, we'll create separate subsets of the data to rule out Simpson's Paradox. Since we are interested in gentrification and displacement of small businesses and local residents, we will creating a subsest of the assessor data that looks only at commerical, industrial, residential, and institutional properties.<jupyter_code>properties = ['Commercial', 'Industrial', 'Residential', 'Institutional']
assessor_data = assessor_data[assessor_data['general_use'].isin(properties)]
assessor_data<jupyter_output><empty_output><jupyter_text>Analysis: Examining by Rail Transist Stops OnlyWe will use the groupby function to see how rail stations along affect property value.<jupyter_code>df_spread1 = assessor_data.groupby("train_stops").describe()
df_spread1<jupyter_output><empty_output><jupyter_text>We'll visualize this spread with a boxplox, removing outliers so that it is easier to read.<jupyter_code>fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)

assessor_data.boxplot(column = "net_value", by=["train_stops"], ax=ax, showfliers=False)
plt.title("Property Net Values by Number of Train Stops in Property's Location Zip Code")
plt.xlabel("Number of Train Stops in Zip Code")
plt.ylabel("Property Net Values")<jupyter_output>/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)<jupyter_text>According to `df_spread1` and the boxplot above, properties in zip codes with 1 or 2 rail trainsit stops have a higher net value than properties in zip codes with no stops. However, this does not seem to be the case for zip codes with 3 or 4 train stops. After examining our station_zip_codes dictionary, there is a very limited amount of zip codes with greater than 2 stops, which could be causing this difference. Since we did not account for outliers in the above box plot, we'll do a strip plot to get an idea of the whole spread.<jupyter_code>fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)

sns.stripplot(x=assessor_data['train_stops'], y=assessor_data['net_value'], ax=ax)<jupyter_output><empty_output><jupyter_text>Zip codes with 1 and 2 train stops have a higher mean and median property value than zip codes with 0, but they seem to have a similar spread even when including outliers.   Analysis: Examining by Rail Transist Stops, Property Type, Year, and LocationTo ensure that this is not a case of Simpson's paradox, we'll examine the summary statistics and complete visualizations that group by rail transit stops + property type and  rail transit stops + year. Finally, we'll account for the geographical location.<jupyter_code>#computing summary statistics 

assessor_data.groupby(["train_stops", "general_use"]).describe()<jupyter_output><empty_output><jupyter_text>Visualizing with a  boxplot that represents the distribution of property values from 2010 to 2020 by train_stop status.<jupyter_code>fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)
sns.boxplot(x="general_use", y='net_value', hue="train_stops", data=assessor_data, showfliers=False)<jupyter_output><empty_output><jupyter_text>From this graph, we see that for commercial and residential properties, the median and mean of property values is higher for zip codes with 2 stops. However, this trend is not consistent for zip codes with 1 train stop. Group by train stops and year So far our anaylsis has not considered how property values have changes over time. We will now graph the mean property value for all types from 2010 to 2020.<jupyter_code>line_graph_time= assessor_data.groupby(by=["year", "train_stops"]).mean().reset_index()
line_graph_time
fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)
sns.lineplot(x=df["year"], y=df["net_value"], hue=df["train_stops"], ax=ax)
plt.title("Average Property Value by Year")
plt.ylabel("Average Net Property Value")<jupyter_output><empty_output><jupyter_text>Our line graph shows that the average property value for zip codes with 2 stops and 1 stop have been higher than those with no stops. The average for 2 and 1 stops also seems to have a steeper slope than the 1 stop line, which could mean that the rate of increase is highe - however, this will need to be verified with further analysis.   Analysis: Accounting for Area Like many US cities, different areas of the city have different economic profiles. For example, West Los Angeles, which includes UCLA/Westwood, Beverly Hills adjacent neighborhoods, and beach neighborhoods, is considerably more wealthy than most other areas of the city. Therefore, when considering the geographical location of a zip code is also important to consider. We have included an area feature in our data to group together zip codes that would likely have similar confounder variables that we are not aware. First - we'll create  box plot of the full assessor data to compare the spread among the different areas. Then, we'll create a facetgrid of line graphs to visualize how avgerage property value has changed over time for each area and train stop status.<jupyter_code>fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)
sns.boxplot(x="area", y='net_value', hue="train_stops", data=assessor_data, showfliers=False)
plt.title("Property Value per Area")<jupyter_output><empty_output><jupyter_text><jupyter_code>line_graph= assessor_data.groupby(by=["year","area", "train_stops"]).mean().reset_index().dropna()
line_graph
g = sns.FacetGrid(line_graph, col='area', hue='train_stops', height=5)
g.map(sns.lineplot, "year","net_value")
g.add_legend()<jupyter_output><empty_output><jupyter_text>Commercial Properties Transit orientated development encourages the develop of multi-family housing and commercial buildings near rail transit stops. As such, we will do a deeeper dive to see how residential and commercial property values near rail stops compare to those not near transit hubs.<jupyter_code>#creating a subset df w/ commercial properties only 
commercial= assessor_data[assessor_data["general_use"]=="Commercial"]
commercial<jupyter_output><empty_output><jupyter_text>Cleaning up some features in the data<jupyter_code>commercial = commercial.reset_index().drop(columns = ["index"])
commercial<jupyter_output><empty_output><jupyter_text>I'll start by describing the data - its spread, count, etc.<jupyter_code>commercial.describe()<jupyter_output><empty_output><jupyter_text>**Univariate analysis** - I'll first plot bar plots of the types of commercial properties in LA, then I'll plot boxplots of commercial properties by train stop status.<jupyter_code>#Exploring types of commercial properties in Los Angeles 
counts = commercial["specific_use"].value_counts().to_frame()
counts.plot(kind="barh")
plt.title("Commercial Property Types in Los Angeles")
plt.show()<jupyter_output><empty_output><jupyter_text>Let's first look at the distribution of commercial values (without outliers) grouping by area and train stops.<jupyter_code>fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)
sns.boxplot(x="area", y='net_value', hue="train_stops", data=commercial, showfliers=False)
plt.title("Average Commercial Property Value per Area");<jupyter_output><empty_output><jupyter_text>let's look over time<jupyter_code>line_graph_commercial = commercial.groupby(by=["year","area", "train_stops"]).mean().reset_index().dropna()
g = sns.FacetGrid(line_graph_commercial, col='area', hue='train_stops', height=5)
g.map(sns.lineplot, "year","net_value")
g.add_legend()<jupyter_output><empty_output><jupyter_text>Residential Properties<jupyter_code>#creating a subset df w/ commercial properties only 
residential= assessor_data[assessor_data["general_use"]=="Residential"].reset_index().drop(columns=['index'])
residential<jupyter_output><empty_output><jupyter_text><jupyter_code>residential.describe()<jupyter_output><empty_output><jupyter_text><jupyter_code>counts = residential["specific_use"].value_counts().to_frame()
counts.plot(kind="barh")
plt.title("Residential Property Types in Los Angeles")
plt.show()
fig_dims = (10, 10)
fig, ax = plt.subplots(figsize=fig_dims)
sns.boxplot(x="area", y='net_value', hue="train_stops", data=residential, showfliers=False)
plt.title("Average Residential Value per Area");
line_graph_residential = residential.groupby(by=["year","area", "train_stops"]).mean().reset_index().dropna()

g = sns.FacetGrid(line_graph_residential, col='area', hue='train_stops', height=5)
g.map(sns.lineplot, "year","net_value")
g.add_legend()<jupyter_output><empty_output><jupyter_text>Hypothesis Testing Finally, we'll finish the Assessor data EDA with hypothesis testing. We will perform a t-test on three scenerios: **Test 1***   Null hypothesis: The mean of properties with 0 train_stops = mean of properties with 1 train stop*   Alt hypothesis: The difference between the means is statistically significant. **Test 2***   Null hypothesis: The mean of properties with 0 train_stops = mean of properties with 2 train stops*   Alt hypothesis: The difference between the means is statistically significant. **Test 3***   Null hypothesis: The mean of properties with 1 train_stops = mean of properties with 2 train stops*   Alt hypothesis: The difference between the means is statistically significant.<jupyter_code>#first scenerio - 0 train stops vs 1+ train stop 
sub1_no_stops= assessor_data[assessor_data["train_stops"].astype(int)==0]
sub2_no_stops= assessor_data[assessor_data["train_stops"].astype(int)>=1]


pg.ttest(sub1_no_stops["net_value"], sub2_no_stops["net_value"])
##second scenerio - 0 train stops vs 1 train stop 
sub3_no_stops= assessor_data[assessor_data["train_stops"].astype(int)==0]
sub4_no_stops= assessor_data[assessor_data["train_stops"].astype(int)==1]
pg.ttest(sub3_no_stops["net_value"], sub4_no_stops["net_value"])
##second scenerio - 0 train stops vs 2 train stop 
sub5_no_stops= assessor_data[assessor_data["train_stops"].astype(int)==0]
sub6_no_stops= assessor_data[assessor_data["train_stops"].astype(int)==1]
pg.ttest(sub5_no_stops["net_value"], sub6_no_stops["net_value"])

##third scenerio - 1 train stops vs 2 train stop 
sub_1_stops= assessor_data[assessor_data["train_stops"].astype(int)==0]
sub_2_stops= assessor_data[assessor_data["train_stops"].astype(int)==1]
pg.ttest(sub_1_stops["net_value"], sub_2_stops["net_value"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12287589718422183, 'fraction_numerical': 0.059566897736335195, 'mean_word_length': 4.993382352941176, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 89, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '283378', 'license': 'MIT', 'n_tokens_mistral': 6230, 'n_tokens_neox': 5503, 'n_words': 1619, 'path': 'Team_93_EDA.ipynb', 'repo_name': 'sorayabailey23/Group-93', 'size': '16301'}