<jupyter_start><jupyter_text>Finding Zestimate erros by <NAME> and <NAME>  Project GoalsThe main goal of this project is to be able to accurately predict logerror in Zillow home price estimates.This will be accomplished by using data from transactions of single family homes in 2017 and clustering of data to build  models, evaluating the effectiveness of each model, and testing the best model on new data is has never seen.The ability to accurately value a home is essential for both buyers and sellers. The ability of us to predict error in zestimates will allow us to determine the major drivers of error then improve our home value estimation models accordingly. Having the most accurate estimates possible is at the core of our business.   Wrangle1. Pulled 2017 transaction data from sql server.2. Ensure filtered for single family homes (bedroom and bathroom counts > 0, home size > 350 sq. ft.).3. Clean function removes outliers (over $5M homes).4. Drop NA values.5. Add county name column from `fips` data.6. **Feature Engineering** Add columns for age of home, age bins, taxrate, size in acres, bins of acre size, sq_ft bins, structure_dollar_per_sqft,  structure_dollar_sqft_bin, land_dollar_per_sqft, lot_dollar_sqft_bin, calculate bathroom/bedroom ratio7. Update datatypes of binned values8. Remove outliers with invalid zipcodes9. Scale data using linear scaler10. Create clusters for area, prize, size, and tax information (see below).11. Give clusters real-world names.12. Encode categorical data (clusters).13. Split data into train, validate, test.14. Split into X (features) and y (target) dataframes.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns


from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler, RobustScaler
from sklearn.model_selection import train_test_split
import warnings
warnings.filterwarnings('ignore')

#Modeling Tools
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
import statsmodels.api as sm
from statsmodels.formula.api import ols

from datetime import date
from scipy import stats


#Custom functions
from env import host, user, password #Database credentials
import wrangle3
import env
import eval_model
import explore
import model
## Evaluation tools
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
from math import sqrt
#Wrangle data as above
train, X_train, y_train, X_validate, y_validate, X_test, y_test=wrangle3.wrangle()<jupyter_output><empty_output><jupyter_text>Clustering- The elbow method was employed plotting k (2-20) vs. inertia to select an appropriate k value.- Part of wrangling. Will look at clusters in explore below.1. **Area cluster**   - k=6   - Scaled_latitude, scaled_longitude, age2. **Price Cluster**   - k=5   - taxrate, structure_dollar_sqft_bin, lot_dollar_sqft_bin3. **Size Cluster**   - k=7   - scaled_bathroomcnt, sqft_bin, acres_bin, bath_bed_ratio  ---  Explore  Questions- Since we are modeling logerror, most of our questions will center on the interaction of logerror with other features.1. What are the drivers of logerror?2. How does logerror vary within gergraphic clusters?3. How does logerror vary within price clusters?4. How does logerror vary within size clusters? ---  Question 1: **What are the drivers of logerror?**- Which features or categories are most strongly correlated with logerror?<jupyter_code>from IPython.display import Image
Image("logerror.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05529701615110868, 'fraction_numerical': 0.01341363263071448, 'mean_word_length': 5.332755632582322, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '78380', 'license': 'MIT', 'n_tokens_mistral': 1036, 'n_tokens_neox': 975, 'n_words': 460, 'path': 'report_notebook.ipynb', 'repo_name': 'Zillow-Project-JG-DN/clustering-project-jgdn', 'size': '3668'}