<jupyter_start><jupyter_text>Load packages and read in the dataset<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
import pandas as pd
from scipy import stats
import seaborn as sns
import statsmodels.formula.api as smf

%matplotlib inline  
#Reading in the dataset
df_kingcounty = pd.read_csv("kingcounty.csv")<jupyter_output><empty_output><jupyter_text>Stakeholder requirements & assumptions (=Hypotheses) Requirements (What I think, the seller wants)The seller wants to...   * maximize her sales profit   * sell historical houses from before 1945 only    * know the best time to sell   * understand if there is a difference between the prices of historical and modern houses   * **NOT DONE YET**   * know which features maximize sales profit of historical houses Assumptions on the data = Hypotheses   * The time of selling influences the price.    * There is a difference between the prices of historical and modern house prices.   * **NOT DONE YET**   * There are some features that make some historical houses more expensive than others (e.g no. of rooms, age of house, renovated?, waterview?) Just take some features. Check the correlations   * If T renevates the house, she gets a higher price. compare colums.   Exploring the dataset<jupyter_code>#Show number of rows and colums
df_kingcounty.shape
#Show top 5-rows and all colums
df_kingcounty.head() # First 5 rows
#Show column names
df_kingcounty.columns
#Rename colums to more understandable names 
df_kingcounty.rename(columns={'date': 'sales_date', 
                            'view': 'viewed', 
                            'condition': 'overall_condition', 
                            'grade': 'overall_grade'}, inplace=True)

#Show column names
df_kingcounty.columns
#Look at the data type for each column and the number of null values
df_kingcounty.info()
#Summarize the data
df_kingcounty.describe()
#Conversion of sales dates to month and years
pd.to_datetime(df_kingcounty["sales_date"])
df_kingcounty["sales_year"] = pd.DatetimeIndex(df_kingcounty["sales_date"]).year
df_kingcounty["sales_month"] = pd.DatetimeIndex(df_kingcounty["sales_date"]).month

df_kingcounty.head()
#Remove outliers and funny values
#Find NAN values 
df_kingcounty.isnull().sum().sort_values()<jupyter_output><empty_output><jupyter_text>Exploring the dataset with regard to the assumptions  Variables needed to investigate seller's questions**Dependent variable*** price for historical houses built before 1945* price for modern houses built in 1945 or later**Explanatory variable(s)*** sales date (year and month)**Further variable(s) needed for calculation of main variables*** year built<jupyter_code>#Create a dataset that contains only historical houses built before 1945
df_hist = df_kingcounty[df_kingcounty['yr_built'] < 1945]
df_hist.head()
#Create a dataset that contains only modern houses built 1945 or after
df_modern = df_kingcounty[df_kingcounty['yr_built'] >= 1945]
df_modern.head()
#Make histograms to investigate the number of houses in general
df_hist['yr_built'].hist()
df_modern['yr_built'].hist()

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(5, 3))
axes[0].plot('yr_built')
axes[1].plot('yr_built');

#CONCLUSIONS from histograms
#There are more modern houses in the data set than historical houses. 
#After World War II and before the financial crisis many new houses were built. 
#The distribution of both data sets appears to be non-nornmal -> Take log. 

#BAD
#No axis description
#No title
#Some strange diagrams below
#Make histograms to investigate the prices of historical and modern houses
df_hist['price'].hist();
df_modern['price'].hist();
#CONCLUSIONS from histograms
#The price distribution of both data sets appears to be non-nornmal -> Take log. 

#BAD
#I am unable to join histograms. 
#Before takint the log I check on outliers in the price using boxplots
df_hist['price'].plot(kind='box');
df_modern['price'].plot(kind='box');
#CONCLUSION
#Strong outliers are present in both data sets. -> delete outliers
#Delete outliers (NOT VERY SCIENTIFIC USING A GRAPHICAL METHOD)
df_hist_cleaned = df_hist[df_hist['price'] < 4.000000e+06]
df_modern_cleaned = df_modern[df_modern['price'] < 4.000000e+06]

#Check if it worked by using 'describe' & 'boxplots' again
#Taking the logarithms of the outlier-cleaned data to normalize the distributions
df_hist_cleaned['lnprice'] = np.log(df_hist_cleaned['price'])
df_modern_cleaned['lnprice'] = np.log(df_modern_cleaned['price'])
#Taking a look at the price histograms again after taking the logarithm to check on the distribution
df_hist_cleaned['lnprice'].hist();
df_modern_cleaned['lnprice'].hist();
#CONCLUSION
#Distribution of prices looks more normal in both data sets after taking the logarithm
#Create scatterplots to see in which month historical and modern houses reached highest prices &
#to check whether the prices of historical and modern houses differ

f, axes = plt.subplots(1,2, figsize=(15,5), squeeze=False)
sns.scatterplot(data=df_hist_cleaned, x="sales_month", y="lnprice", hue="yr_built", ax=axes[0][0])
sns.scatterplot(data=df_modern_cleaned, x="sales_month", y="lnprice", hue="yr_built", ax=axes[0][1]);
#CONCLUSION
# With the bare eye no "higest sales price" is visible in either data set
# The same applies to differences between historical and modern house prices. 
#Repeat the analyis using median values
df_hist_cleaned_median = df_hist_cleaned.groupby('sales_month')['lnprice'].median()
df_modern_cleaned_median = df_modern_cleaned.groupby('sales_month')['lnprice'].median()
#Create scatterplots with median price per month
f, axes = plt.subplots(1,2, figsize=(15,5), squeeze=False)
sns.scatterplot(data=df_hist_cleaned, x="sales_month", y="lnprice", hue="yr_built", ax=axes[0][0])
sns.scatterplot(data=df_modern_cleaned, x="sales_month", y="lnprice", hue="yr_built", ax=axes[0][1]);
#CONCLUSION
#Same conclusion as above - no pattern visible with naked eye
#Regarding historical houses, price differences between sales month appear to be rather "small" and driven by outliers.
# -> Remove further outliers and repeat analysis. Also look at numbers, not just graphs

#Better do a t-test to check whether the price distribution of historical an modern houses differ
#Looking at the median per sales month for historical and modern houses
print(df_hist_cleaned_median, df_modern_cleaned_median)
#CONCLUSION of looking at median per sales month
#Historical houses reach higehst prices in median in April & in Juli. 
#Modern houses reach highest prices in median in April & June. 
#Median price differences between sales month are generally small
#Median price differences between historical and modern houses are roughly about 1 Mio. $.<jupyter_output><empty_output><jupyter_text>Use of multiple linear regression to explore the data<jupyter_code>smf.ols(formula='lnprice ~ sales_month', data=df_hist_cleaned).fit().summary()
#CONCLUSION
#The impact of the sales month is insignficiant for historical houses. 
smf.ols(formula='lnprice ~ sales_month', data=df_modern_cleaned).fit().summary()
#CONCLUSION
#The impact of the sales month is significant for modern houses.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08931044032124065, 'fraction_numerical': 0.010108003323179175, 'mean_word_length': 5.794920037629351, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '292194', 'license': 'MIT', 'n_tokens_mistral': 2258, 'n_tokens_neox': 2065, 'n_words': 790, 'path': 'Kingcounty_EDA_DRAFT_clear.ipynb', 'repo_name': 'EcholotFisher/Kingcounty_EDA', 'size': '7222'}