<jupyter_start><jupyter_text>Why do it ?- Because Tech is exciting, but unpredictable.- A back-up income strategy is needed.- For Happy / Early Retirement Case Study:- Buy single family house in expanding job market in __South Carolina__- Be lazy for 10 years and collect rent- Check back to see value increase 4x times after 10 years Factors considered for property investment:- Cost of single family house- School ratings- Job growth- Crime rate- Cost-to-Rent ratio  Data Sources:- Zipcode Csv file from USPS (https://tools.usps.com/go/ZipLookupAction_input)- Rental Data from City Data API (http://www.city-data.com/)- Rental & Cost of Single Family House from Zillow (https://www.zillow.com/howto/api/APIOverview.htm)- Property Data from Quandl API (https://www.quandl.com/tools/api)        - Great Schools API (https://www.greatschools.org/)  - Crime Rate Data retrieved from Crime Rate website (http://www.crimerate.com/)            - Job Growth Data retrieved from Bureau of Labor Statistics (https://www.bls.gov/sae/tables)  Retrieval Methods:- APIs to retrieve data using requests module- Web scraping using BeautifulSoup module- Converting XML data to python dictionaries using XmlDict module- Converting HTML tags into text and selecting data using split function- Converting unstructured scraped data using regular expressions- Python Pandas for aggregation and clean up of data           Data Summary & Metrics- Data retrieved for 407 zip codes in South Carolina state.- After cleanup and aggregation, final data set contains data for 206 zip codes- Normalization of 'Rent', 'School Ratings', 'Job Growth' & 'Crime Rate' to a factor of '1'- Summary metric derived by adding the normalized values  Mapping Function.<jupyter_code>import os
import numpy as np
import pandas as pd
from scipy.interpolate import griddata
import matplotlib.pyplot as plt
import folium
from folium import plugins
from folium.plugins import ImageOverlay
from scipy.ndimage import imread
data = pd.read_csv("../data/final_investment_criteria.csv")
data.head()<jupyter_output><empty_output><jupyter_text>THE REAL CRAP.<jupyter_code>df4 = data[["latitude", "longitude", "School Rating_nom"]]
df5 = data[["latitude", "longitude", "Total Crimes_nom"]]
df6 = data[["latitude", "longitude", "job_growth_per_nom"]]

# Build the base maps.
%matplotlib inline
    
map_osm1 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm2 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm3 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm3a = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm3b = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm3c = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm3d = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm4 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm5 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm6 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)
map_osm7 = folium.Map([33., -80.], tiles='stamentoner', zoom_start=7.4)


# top ten tabs.
df4a = df4.sort_values("School Rating_nom", ascending = False)[0:10]
df5a = df5.sort_values("Total Crimes_nom", ascending = True)[0:10]
df5b = df5.sort_values("Total Crimes_nom", ascending = False)[0:10]
df6a = df6.sort_values("job_growth_per_nom", ascending = False)[0:10]

# add the tabs to the map.
for x in range(len(df4a["School Rating_nom"])):
    folium.Marker([df4a["latitude"].iloc[x], df4a["longitude"].iloc[x]],
                    popup="School Rating: " + str(df4a["School Rating_nom"].iloc[x])).add_to(map_osm3a)

for x in range(len(df5a["Total Crimes_nom"])):
    folium.Marker([df5a["latitude"].iloc[x], df5a["longitude"].iloc[x]],
                    popup="Total Crimes: " + str(df5a["Total Crimes_nom"].iloc[x])).add_to(map_osm3b)
    
for x in range(len(df5b["Total Crimes_nom"])):
    folium.Marker([df5b["latitude"].iloc[x], df5b["longitude"].iloc[x]],
                    popup="Total Crimes: " + str(df5b["Total Crimes_nom"].iloc[x])).add_to(map_osm3c)

for x in range(len(df6a["job_growth_per_nom"])):
    folium.Marker([df6a["latitude"].iloc[x], df6a["longitude"].iloc[x]],
                    popup="Job Growth: " + str(df6a["job_growth_per_nom"].iloc[x])).add_to(map_osm3d)
    
    
# do the contour maps.
x = data["longitude"]
y = data["latitude"]
z1 = data["Cost of Single Family Home"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z1, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay1.png")

x = data["longitude"]
y = data["latitude"]
z2 = data["Rent"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z2, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay2.png")


x = data["longitude"]
y = data["latitude"]
z3 = data["Normalized Ratio_nom"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z3, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay3.png")


x = data["longitude"]
y = data["latitude"]
z4 = data["School Rating_nom"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z4, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay4.png")


x = data["longitude"]
y = data["latitude"]
z5 = data["Total Crimes_nom"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z5, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay5.png")


x = data["longitude"]
y = data["latitude"]
z6 = data["job_growth_per_nom"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z6, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay6.png")


x = data["longitude"]
y = data["latitude"]
z7 = data["summary_metric"]
xi = np.linspace(data["longitude"].min(), data["longitude"].max(), 100)
yi = np.linspace(data["latitude"].min(), data["latitude"].max(), 100)
zi = griddata((x, y), z7, (xi[None,:], yi[:,None]), method = "cubic")
CS = plt.contourf(xi, yi, zi, 15, cmap=plt.cm.jet)
cur_axes = plt.gca()
cur_axes.axes.get_xaxis().set_visible(False)
cur_axes.axes.get_yaxis().set_visible(False)
ax = plt.subplot()
ax.spines["right"].set_visible(False)
ax.spines["top"].set_visible(False)
ax.spines["bottom"].set_visible(False)
ax.spines["left"].set_visible(False)
plt.savefig("../data/overlay7.png")




# Do the overlay of the png contour map.
data1 = plt.imread('../data/overlay1.png')
data2 = plt.imread("../data/overlay2.png")
data3 = plt.imread("../data/overlay3.png")
data4 = plt.imread("../data/overlay4.png")
data5 = plt.imread("../data/overlay5.png")
data6 = plt.imread("../data/overlay6.png")
data7 = plt.imread("../data/overlay7.png")



min_lon = data["longitude"].min()
max_lon = data["longitude"].max()
min_lat = data["latitude"].min()
max_lat = data["latitude"].max()

map_osm1.add_child(plugins.ImageOverlay(data1, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm2.add_child(plugins.ImageOverlay(data2, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm3.add_child(plugins.ImageOverlay(data3, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm3a.add_child(plugins.ImageOverlay(data3, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm3b.add_child(plugins.ImageOverlay(data3, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm3c.add_child(plugins.ImageOverlay(data3, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm3d.add_child(plugins.ImageOverlay(data3, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm4.add_child(plugins.ImageOverlay(data4, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm5.add_child(plugins.ImageOverlay(data5, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm6.add_child(plugins.ImageOverlay(data6, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))

map_osm7.add_child(plugins.ImageOverlay(data7, opacity=0.5, interactive=True,
    cross_origin=False,
    zindex=1, 
    bounds =[[min_lat-0.6, min_lon-.9], [max_lat+0.5, max_lon+0.65]]))


print("done")<jupyter_output>C:\Users\kisho\Anaconda3\envs\pythondata\lib\site-packages\matplotlib\cbook\deprecation.py:106: MatplotlibDeprecationWarning: Adding an axes using the same arguments as a previous axes currently reuses the earlier instance.  In a future version, a new instance will always be created and returned.  Meanwhile, this warning can be suppressed, and the future behavior ensured, by passing a unique label to each axes instance.
  warnings.warn(message, mplDeprecation, stacklevel=1)<jupyter_text>High = Red (GOOD) Low = Blue (BAD)  Cost of Single Family Home.- Cost of single family house increases as we move closer towards the beach<jupyter_code>map_osm1<jupyter_output><empty_output><jupyter_text>Rent.<jupyter_code>map_osm2<jupyter_output><empty_output><jupyter_text>Cost/Rent Ratio.- Rents are higher as we move closer to the coastal area and areas near the capital city Columbia<jupyter_code>map_osm3<jupyter_output><empty_output><jupyter_text>School Rating.<jupyter_code>map_osm4<jupyter_output><empty_output><jupyter_text>Crime Rating.<jupyter_code>map_osm5<jupyter_output><empty_output><jupyter_text>Job Rating.- More jobs in the areas closer to the coast and border of North Carolina- Job growth is pretty consistent towards North-West of South Carolina<jupyter_code>map_osm6<jupyter_output><empty_output><jupyter_text>Summary.- North-East & North-West of South Carolina are the best places to invest.<jupyter_code>map_osm7<jupyter_output><empty_output><jupyter_text>Cost/Rent Ratio v. School Rating--Top Ten.<jupyter_code>map_osm3a<jupyter_output><empty_output><jupyter_text>Cost/Rent Ratio v. Crime--Worst Ten.<jupyter_code>map_osm3b<jupyter_output><empty_output><jupyter_text>Cost/Rent Ratio v. Crime--Best Ten.<jupyter_code>map_osm3c<jupyter_output><empty_output><jupyter_text>Cost/Rent Ratio v. Jobs--Top Ten.<jupyter_code>map_osm3d<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1744047619047619, 'fraction_numerical': 0.028422619047619047, 'mean_word_length': 5.630981746423286, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 5, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '483782', 'license': 'MIT', 'n_tokens_mistral': 5318, 'n_tokens_neox': 5116, 'n_words': 977, 'path': 'src/contour_maps.ipynb', 'repo_name': 'knishina/clean_real_estate', 'size': '13440'}