<jupyter_start><jupyter_text>Module 1 Final Project Submission* Student name: <NAME>* Student pace: PART TIME* Scheduled project review date/time: FRIDAY, 10/18/2019, 12pm* Instructor name: <NAME>* Blog post URL:  The King County Housing Project  We begin by loading the data, looking at the contents, and start to clean it.<jupyter_code># Load the data set and check the header
import pandas as pd
data = pd.read_csv("kc_house_data.csv")
data.head()
# Get a look at the features, total entries, and types
data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 21597 entries, 0 to 21596
Data columns (total 21 columns):
id               21597 non-null int64
date             21597 non-null object
price            21597 non-null float64
bedrooms         21597 non-null int64
bathrooms        21597 non-null float64
sqft_living      21597 non-null int64
sqft_lot         21597 non-null int64
floors           21597 non-null float64
waterfront       19221 non-null float64
view             21534 non-null float64
condition        21597 non-null int64
grade            21597 non-null int64
sqft_above       21597 non-null int64
sqft_basement    21597 non-null object
yr_built         21597 non-null int64
yr_renovated     17755 non-null float64
zipcode          21597 non-null int64
lat              21597 non-null float64
long             21597 non-null float64
sqft_living15    21597 non-null int64
sqft_lot15       21597 non-null int64
dtypes: float64(8), int64(11), object(2)
memory usage: 3.5+ MB<jupyter_text>Detecting Missing Data<jupyter_code># First I need to look at columns 'yr_renovated' and 'waterfront' due to the NaN data.
# 'yr_renovated  17755' and 'waterfront  19221'
# The column 'view 21534' is also missing some information though it may be minimal.
# I also realized that 'date' and 'sqft_basement' contain objects!
# My first look for missing data is confirmed
data.isna().sum()
missing_1 = 3842/21597
print("'yr_renovated' feature is missing about " + str(round(missing_1*100)) + " percent of the data.")
missing_2 = 2376/21597
print("'waterfront' feature is missing about " + str(round(missing_2*100)) + " percent of the data.")
# While both of these features are missing a lot of data, it doesn't constitute deleting the column.
# Use describe method to further look at the data.
data.describe()
# Many of the columns have max/mins that probably include outliers.<jupyter_output><empty_output><jupyter_text>Which other columns or rows are hiding something?  Let's start with 'bedrooms' and look closer.<jupyter_code>data.bedrooms.unique()
# Just in case this house is a mansion with 33 bedrooms, I wanted to look at the whole row.
data.loc[data['bedrooms'] == 33]
# Looks as if the number was just entered incorrectly.  Should I delete it?  Change it to just '3'?
# I think it's just a typo, so I'll change it to the median.
data.at[15856,'bedrooms'] = 3
data.bedrooms.describe()<jupyter_output><empty_output><jupyter_text>I'm excited to see the variety of 'bathrooms' that houses might have.<jupyter_code># I guess some houses really do have a lot of bathrooms.
data.bathrooms.unique()
data.loc[data['bathrooms'] == 8]
# Two houses with eight bathrooms or similar, meaning it's probably not an error but they are outliers. 
# Let's look at the number of houses with more than 4 standard deviations
data.bathrooms.std()*4 + data.bathrooms.mean()
# Drop all 'bathrooms' rows that are greater than '6'
data = data[data.bathrooms < 6]<jupyter_output><empty_output><jupyter_text>What different kind of 'floors' do the houses have?<jupyter_code>data.floors.unique()  
# Are floor levels continuous?
# Does 2.5 floors make sense?  I think I will just leave these as a continuous variable.<jupyter_output><empty_output><jupyter_text>The 'id' feature is has really big numbers.<jupyter_code>data.id.unique() 
# Probably gonna just drop these since the number don't really mean anything.<jupyter_output><empty_output><jupyter_text>Either you have 'waterfront' or you don't, right?<jupyter_code>data.waterfront.unique()
# I think all the 'nan's should just become 0.  Either you have waterfront or you don't!
# Let's get rid of the Nan's, change them to '0'
data.waterfront.fillna(value=data.waterfront.median(), inplace = True)<jupyter_output><empty_output><jupyter_text>What is this 'view' all about?<jupyter_code>data.view.unique()
# This is going to be tough! Is this continuous?  Are the nan's really 0's.<jupyter_output><empty_output><jupyter_text>How many homes have a 'view' of 4, the max?<jupyter_code>len(data.loc[data['view'] == 4])
# 312 homes feel as if their view is a "4".  Is that 4 stars?  4/4?  Definitely Categorical, should be binned!
# Let's fill them with '0'. Either you have a view or you don't!
data.view.fillna(value=data.view.median(), inplace = True)<jupyter_output><empty_output><jupyter_text>What about that house with 13,540 sq. ft. of 'living space'?<jupyter_code>data.loc[data['sqft_living'] == 13540]
# I guess it's just a big, expensive house.  Lots of bedrooms and bathrooms.  But is it an outlier still?
# What is three standard deviations from the mean?
data.sqft_living.std()*4 + data.sqft_living.mean()
# How many houses fall outside of this?
len(data.loc[data["sqft_living"] > 5700])
data = data[data.sqft_living < 5700]<jupyter_output><empty_output><jupyter_text>What is an average 'sqft_lot' size?<jupyter_code># About a third of an acre?
data.sqft_lot.mean()
# Three standard deviations.
data.sqft_lot.mean()*3
# Number of homes that have more than a 1 acre lot or 43560 sqft.
len(data.loc[data["sqft_lot"] > 45000])
# Time to do some binning!  Maybe 0 - 1 acre, 1 - 3 and 3 - 6 acres?<jupyter_output><empty_output><jupyter_text>Where can I find information on how King County 'grades' their homes?<jupyter_code>data.grade.unique()
# Can I bin these? Or maybe do some sort of Log Transformation to them closer to zero.<jupyter_output><empty_output><jupyter_text>'Basements'? Can I just say YES/NO to make it easier?<jupyter_code>data.sqft_basement.nunique()
# After looking at .unique(), I see that there are lots of different basements, including "0" and 454 are "?"
# Let's just change them to '0'  - either you have a basement or not.
data.sqft_basement = data.sqft_basement.replace(to_replace = '?', value = 0)
data.loc[data['sqft_basement'] == '?']
data.sqft_basement.describe()
# Since 'basement' also came up as an 'object' I'm going to change them to floats, fill Nan's, and then bin them.
data.sqft_basement.fillna(value=data.sqft_basement.median(), inplace = True)
data['sqft_basement'] = data['sqft_basement'].astype(float)
data.sqft_basement.describe()<jupyter_output><empty_output><jupyter_text>How important is 'sqft_above'?  Especially since I know the total?<jupyter_code># 'sqft_above' seems continuous.  Or is this number really just the total sqaure foot minus the basement?
# Also, is this features going to effect the multicollinearity?<jupyter_output><empty_output><jupyter_text>What does the 'condition' feature look like?<jupyter_code>data.condition.unique()
# Should I consider binning these? Most of them are '3'.  Seems like it's a categorical feature.<jupyter_output><empty_output><jupyter_text>Neighboring 'living size' has entries way beyond the 3 standard deviations.<jupyter_code># Let's look at the right skewed histogram.
data.sqft_living15.hist()
# Again, using 4 standard deviations to predict outliers
data.sqft_living15.mean()+data.sqft_living15.std()*4
# There's not that many.
len(data.loc[data["sqft_living15"] > 4600])
# Let's get rid of the outliers
data = data[data.sqft_living15 < 4600]<jupyter_output><empty_output><jupyter_text>Neighboring 'lot size' has a bunch of big values.<jupyter_code># Let's do the same here.
data.sqft_lot15.mean()+data.sqft_lot15.std()*4
len(data.loc[data["sqft_lot15"] > 121120])
# Let's get rid of the outliers
data = data[data.sqft_lot15 < 121120]
# These numbers seem continuous, especially if you speak square footage.
# Also, are these related to zipcode?  Usually neighborhoods have similar sized houses!<jupyter_output><empty_output><jupyter_text>Still have to deal with zipcode, lat, and long<jupyter_code># 'zipcode' is somehow related to 'lat' and 'long'!<jupyter_output><empty_output><jupyter_text>The 'yr_built' feature seems pretty standard. But is it continuous?<jupyter_code>data.yr_built.unique()
# Perhaps I can do something with these numbers to even them out. Log transform?<jupyter_output><empty_output><jupyter_text>The 'year renovated' is certainly going to be more difficult.<jupyter_code>data.yr_renovated.unique()
# Some how I have to deal with these. Most of them have not been renovated.  Some are 'Nan'. 
# Can I bin them into a few groups - recently renovated, renovated a while ago, and not at all?
# Let's fill in the "Nan's" with 0.
data.yr_renovated.fillna(value=data.yr_renovated.median(), inplace = True)<jupyter_output><empty_output><jupyter_text>So I tried to run just a simple model on all the data, before processing - and I realized that I hadn't looked at the feature "date".<jupyter_code># Not sure what to do but I'm especially confused by the fact that is says, "object"
data.date.describe()
# I'll try to change them to something different, perhaps a float.
import datetime as dt
data['date'] = pd.to_datetime(data['date'])
data['date']= data['date'].map(dt.datetime.toordinal)
data.date.describe()
import matplotlib.pyplot as plt
%matplotlib inline
data.date.hist()<jupyter_output><empty_output><jupyter_text>Thought I would check for duplicates.<jupyter_code>duplicates = data[data.duplicated(subset='id')]
print(len(duplicates))

# So I check duplicates of the 'id' and it shows that 177 of the homes are  repeats.
# This is probably because they were sold more than once.  
# First, it's a really small number. 
# Second, all the info is there and the price/date changed. So it's essentially a different house<jupyter_output><empty_output><jupyter_text>Here I just wanted to look at value counts for all features to see if I'm missing anything.  But it's only the top 5 percents, so I might need to look at some of them individually.<jupyter_code>for col in data.columns:
    print(col, '\n', data[col].value_counts(normalize=True).head(), '\n\n')<jupyter_output>id 
 795000620     0.000139
1825069031    0.000093
2019200220    0.000093
7129304540    0.000093
1781500435    0.000093
Name: id, dtype: float64 


date 
 6/23/2014    0.006575
6/25/2014    0.006066
6/26/2014    0.006066
7/8/2014     0.005880
4/27/2015    0.005834
Name: date, dtype: float64 


price 
 350000.0    0.007964
450000.0    0.007964
550000.0    0.007362
500000.0    0.007038
425000.0    0.006945
Name: price, dtype: float64 


bedrooms 
 3    0.454878
4    0.318655
2    0.127796
5    0.074131
6    0.012594
Name: bedrooms, dtype: float64 


bathrooms 
 2.50    0.248970
1.00    0.178312
1.75    0.141131
2.25    0.094782
2.00    0.089364
Name: bathrooms, dtype: float64 


sqft_living 
 1300    0.006390
1400    0.006251
1440    0.006158
1660    0.005973
1010    0.005973
Name: sqft_living, dtype: float64 


sqft_lot 
 5000    0.016576
6000    0.013428
4000    0.011622
7200    0.010187
7500    0.005510
Name: sqft_lot, dtype: float64 


floors 
 1.0    0.494189
2.0    0.381303
1.5    [...]<jupyter_text>Then I tried a histogram plot<jupyter_code>#data.hist(figsize  = [12, 12]);
# First I noticed that not all the graphs are there.<jupyter_output><empty_output><jupyter_text>I also wanted to try a scatter matrix to look for continuous and non-continuous variables<jupyter_code>#pd.plotting.scatter_matrix(data, figsize=(12,12));
# This take a long time but it helped me determine which ones are continuous and with are more categorical.<jupyter_output><empty_output><jupyter_text>Multicollinearity of Features<jupyter_code># Here I used the correlation matrix
#data.corr()
# Here I looked at the same with the correlation set to .75
abs(data.corr()) > 0.75
import seaborn as sns
sns.heatmap(data.corr(), center=0);<jupyter_output><empty_output><jupyter_text>Looking at this heat map I can see that there are a couple of features that may cause multicollinearity. In particular 'sqft_above' and 'grade'.<jupyter_code>import matplotlib as plt
data.plot('lat', 'long')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09627759224566262, 'fraction_numerical': 0.06654720208519997, 'mean_word_length': 4.694805194805195, 'pattern_counts': {'":': 0, '<': 114, '<?xml version=': 0, '>': 115, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '605195', 'license': 'CNRI-Python', 'n_tokens_mistral': 4674, 'n_tokens_neox': 4036, 'n_words': 1402, 'path': 'student_new.ipynb', 'repo_name': 'twhipple/dsc-v2-mod1-final-project-online-ds-pt-090919', 'size': '12296'}