<jupyter_start><jupyter_text>Use Machine Learning and Deep Learning to predict and analyze US Housing Price Team: <NAME>, <NAME>, <NAME> As an important source of economic development, real estate has become a pillar industry for the United States. There are several contributions for the housing price, for instance, type of house, size, location, tax and cost etc.. In this project, we analyze 4 datasets based on Linear Regression and Deep Learning Regression to analyze the factors and their relationships with housing prices, then applying Random Forest to do the prediction. We will use King Country Housing Price Database to do sample analyze and compare all results from 4 datasets to make conclusion.  KC house data Data Preparation  Load data from csv<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
# loading data king country as data
data = pd.read_csv('kc_house_data.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Now we have loaded dataset. Before analysing it, we need to make sure there is no incorrect, incomplete, improperly formatted data in dataset.<jupyter_code>data.info()
data.isnull().sum()
data.duplicated().sum()<jupyter_output><empty_output><jupyter_text>In this part we check whether there are duplicated rows or null value in the dataset. If there are abnormal value, we will drop them.Results show that there is no duplication or null value. We can move forward to explore datasets. If it has null value, we can use following code to drop null value.<jupyter_code>data=data.dropna(axis=0, how='any')<jupyter_output><empty_output><jupyter_text>Data Exploration For data exploration, first we use the describe() method to show the summary statistics of the numeric variables. And the we can use savefig to get attribute data's distribution and get their distibution chart.<jupyter_code>data.describe()
data.hist(bins=50, figsize=(20,15))
plt.savefig("attribute_histogram_plots")
plt.show()<jupyter_output><empty_output><jupyter_text>Some of the distribution are a little bit right skewed, but this is not abnormal.  Mutiple Regression To explore the house price, we use heapmap matrix to get correlation of each atttributes and their relationships.<jupyter_code># Function plot_correlation_map for visualization 
def plot_correlation_map( df ):
    corr = df.corr()
    #set size of the figure
    map , ax = plt.subplots( figsize =( 20 , 20 ) )
    cmap = sns.diverging_palette( 20 , 10 , as_cmap = True )
    map = sns.heatmap(
        corr,
        cmap = cmap,
        square=True, 
        ax=ax, 
        annot = True, 
        annot_kws = { 'fontsize' : 8 }
    )
plot_correlation_map(data)<jupyter_output><empty_output><jupyter_text>Then we try to pick top relative attributes from heatmap for regression<jupyter_code># choose key variables and show the corresponed heatmap
corr = data[['bathrooms', 'bedrooms', 'sqft_living', 'sqft_lot', 'floors', 'grade', 'price']]
##set heatmapsize
plt.figure(figsize=(10,8))
##set title
plt.title('Correlation of variables')
##set max corr=1,add note in blank and data type is float
sns.heatmap(corr.astype(float).corr(),vmax=1.0,  annot=True)
plt.show()<jupyter_output><empty_output><jupyter_text>From this heaptmap we can find that bathroom number, living room sqft and grade are more relative to house price. So we pick these 4 attributes to do regression.<jupyter_code># Setup X, y
X = data[["bedrooms","bathrooms","sqft_living","grade"]]
y = data["price"]<jupyter_output><empty_output><jupyter_text>Then we will apply KNeighbors Regression,Linear Regression, Lasso Regression, Ridge Regression, Decision Tree Regression and Support Vector Regression to our dataset. K-nearest neighbors (kNN) is a non-parametric method used for classification and regression.Linear regression is the simplest and most widely used statistical technique for predictive modeling. It basically gives us an equation, where we have our features as independent variables, on which our target variable is dependent upon. We expect linear regrssion will perform well with our dataset.Ridge Regression is a technique for analyzing multiple regression data that suffer from multicollinearity. When multicollinearity occurs, least squares estimates are unbiased, but their variances are large so they may be far from the true value.Lasso Refression is similar to Ridge regression, and it is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the statistical model it produces. Decision tree builds regression  models in the form of a tree structure. It brakes down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The Support Vector Regression (SVR) are very specific class of algorithms, characterized by usage of kernels, absence of local minima, sparseness of the solution and capacity control obtained by acting on the margin, or on number of support vectors.<jupyter_code># load state model with ordinary least squares model, fit with X, y
model = sm.OLS(y, X)
results = model.fit()
results.summary()<jupyter_output><empty_output><jupyter_text>We split dataset into train data and test data. The training set contains a known output and the model learns on this data in order to be generalized to other data later on.<jupyter_code>##Get train set and test set
X_train,X_test, y_train, y_true= train_test_split(X, y, test_size=0.2) 
# Set up knn, linear regression, ridge, lasso, decision tree and svr model
knn = KNeighborsRegressor(n_neighbors=3)
linear = LinearRegression()
ridge = Ridge()
lasso = Lasso()
decision = DecisionTreeRegressor()
svr = SVR()
# fit our model with train data
knn.fit(X_train,y_train)
linear.fit(X_train,y_train)
ridge.fit(X_train,y_train)
lasso.fit(X_train,y_train)
decision.fit(X_train,y_train)
svr.fit(X_train,y_train)
# get prediction
y_pre_knn = knn.predict(X_test)
y_pre_linear = linear.predict(X_test)
y_pre_ridge = ridge.predict(X_test)
y_pre_lasso = lasso.predict(X_test)
y_pre_decision = decision.predict(X_test)
y_pre_svr = svr.predict(X_test)<jupyter_output><empty_output><jupyter_text>Now we can get the accuracy score of each model.<jupyter_code># display score of our prediction
knn_score = r2_score(y_true,y_pre_knn)
linear_score=r2_score(y_true,y_pre_linear)
ridge_score=r2_score(y_true,y_pre_ridge)
lasso_score=r2_score(y_true,y_pre_lasso)
decision_score=r2_score(y_true,y_pre_decision)
svr_score=r2_score(y_true,y_pre_svr)
display(knn_score,linear_score,ridge_score,lasso_score,decision_score,svr_score)<jupyter_output><empty_output><jupyter_text>From the result, linear regression, lasso regression and ridge regression perform better as we expect.For other data sets, we can choose those three regressions to do analyze.  Deep Learning Regression The term deep learning regression refers to training neural networks. Neural networks are used for the purpose of clustering through unsupervised learning, classification through supervised learning, or regression. First, we will create a baseline neural network model for the regression problem.<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# load dataset
dataframe = pd.read_csv('kc_house_Sample.csv')
dataframe=dataframe.dropna(axis=0, how='any')
# split into input (X) and output (Y) variables
X = dataframe[["bedrooms", "bathrooms","grade","sqft_living"]]
Y = dataframe["price"]<jupyter_output><empty_output><jupyter_text>Baseline ModelThe efficient ADAM optimization algorithm is used and a mean squared error loss function is optimized. This will be the same metric that we will use to evaluate the performance of the model.<jupyter_code># define base model
def baseline_model():
    # create 3-layer model(more layer will make the model more accurate)
    model = Sequential()
    model.add(Dense(4, input_dim=4, kernel_initializer='normal', activation='relu'))
    model.add(Dense(6, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model(after test, cosine_proximity is the one fit our model most)
    model.compile(loss='cosine_proximity', optimizer='adam')
    return model<jupyter_output><empty_output><jupyter_text>We can create Keras models and evaluate them with scikit-learn by using handy wrapper objects provided by the Keras library. We also initialize the random number generator with a constant random seed, to ensure we compare models consistently.<jupyter_code># fix random seed for reproducibility
seed = 7
np.random.seed(seed)
# evaluate model with standardized dataset
estimator = KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=5, verbose=0)<jupyter_output><empty_output><jupyter_text>We will use 10-fold cross validation to evaluate the model. The result reports the mean squared error including the average and standard deviation (average variance) across all 10 folds of the cross validation evaluation.<jupyter_code>kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, Y, cv=kfold)
print("Results: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))<jupyter_output>Results: -1.00 (0.00) cosine_proximity<jupyter_text>Modeling The Standardized DatasetContinuing on from the above baseline model, we can re-evaluate the same model using a standardized version of the input dataset. In order to avoid data leakage, we can use scikit-learn’s Pipeline framework to perform the standardization during the model evaluation process, within each fold of the cross validation.<jupyter_code># evaluate model with standardized dataset
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Standardized: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))<jupyter_output>Standardized: -0.80 (0.60) cosine_proximity<jupyter_text>Random ForestRandom forests, also called random decision forests are an ensemble learning method for regression that operate by constructing a multitude of decision trees at training time and outputting the class that is the mean prediction (regression) of the individual trees. we start with import<jupyter_code>from pprint import pprint
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import RandomizedSearchCV<jupyter_output><empty_output><jupyter_text>Data preperation Random forest regression need values to be SparseDataFrame, so we use panda get_dummies to get our partial data with most related variables. In theory, we need labels to be what we wanna to predict, and features be other data we use to predict our labels. So, we applay column "price" to labels, and other variables to features.<jupyter_code>features = pd.get_dummies(X)
labels = data["price"]
features_list = list(features.columns)
features = np.array(features)
labels = np.array(labels)<jupyter_output><empty_output><jupyter_text>With all initial setup done, we can do train test split, with test size and train size at ratio 1:3<jupyter_code># split train and test data
train_features, test_features, train_labels, test_labels = train_test_split(features, labels, test_size = 0.25, random_state = 42)
train_features.shape<jupyter_output><empty_output><jupyter_text>Random Forest Regression setup After, we already splited train and test data. We want to setup our random forest model, we want to have the same result everytime if we reproduce our random forest regression, so we apply random_state = 42. For every random forest model, we need to do hyperparameter first in order to find the best set  of parameters. Lets find the default set parameters for our model.<jupyter_code># Set up our randomforest with random_state = 42
rf = RandomForestRegressor(random_state = 42)
# Look at parameters used by our current forest
print('Parameters currently in use:\n')
pprint(rf.get_params())<jupyter_output>Parameters currently in use:

{'bootstrap': True,
 'criterion': 'mse',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 10,
 'n_jobs': 1,
 'oob_score': False,
 'random_state': 42,
 'verbose': 0,
 'warm_start': False}<jupyter_text>Hyperparameter Tuning As we saw the default parameters, we want to do further hyperparameter to find best variables to improve our random forest model. First we initialize our random parameter grid<jupyter_code># Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 100, stop = 1000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]
random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}
pprint(random_grid)<jupyter_output>{'bootstrap': [True, False],
 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, None],
 'max_features': ['auto', 'sqrt'],
 'min_samples_leaf': [1, 2, 4],
 'min_samples_split': [2, 5, 10],
 'n_estimators': [100, 200, 300, 400, 500, 600, 700, 800, 900, 1000]}<jupyter_text>We should use our random forest model to estimate the best variables we already setup. So, Let's fit our model with test and train data.<jupyter_code># fit our randomforest model with features and labels
rf.fit(features, labels)<jupyter_output><empty_output><jupyter_text>After that we need a method to estimate the hyperparameter. There is a randmonized Search CV that randomized search on hyper parameters. RandomizedSearchCV implements a “fit” and a “score” method. The parameters of the estimator used to apply those methods are optimized by cross-validated search over parameter settings.The randomizedSerch cross validation is a good way to find hyperparameter for our random forest model.<jupyter_code># Use the random grid to search for best hyperparameters
# First create the base model to tune
# rf = RandomForestRegressor()
# Random search of parameters, using 3 fold cross validation, 
# search across 100 different combinations, and use all available cores
rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, n_iter = 100, cv = 3, verbose=2, random_state=42, n_jobs = -1)
# Fit the random search model
rf_random.fit(train_features, train_labels)<jupyter_output>Fitting 3 folds for each of 100 candidates, totalling 300 fits<jupyter_text>There we found the best parameters<jupyter_code>rf_random.best_params_<jupyter_output><empty_output><jupyter_text>Get result from Random Forest Regression<jupyter_code>best_random = rf_random.best_estimator_
best_random.fit(train_features, train_labels)
predictions = best_random.predict(test_features)<jupyter_output><empty_output><jupyter_text>we can find the score of our best random hyperparameter<jupyter_code>rf_random.best_score_<jupyter_output><empty_output><jupyter_text>We need to know how well our random forest model with best randomized parameter fit our dataset<jupyter_code>best_random.score(test_features, test_labels)<jupyter_output><empty_output><jupyter_text>Get the predictionLet us see our prediction related to our test_features<jupyter_code>np.set_printoptions(threshold=np.inf)  
predictions<jupyter_output><empty_output><jupyter_text>Results of other datasets For San Louis, we can get the accuracy score of KNeighbors Regression,Linear Regression, Lasso Regression, Ridge Regression, Decision Tree Regression and Support Vector Regression models:<jupyter_code># display score of our prediction
knn_score = r2_score(y_true,y_pre_knn)
linear_score=r2_score(y_true,y_pre_linear)
ridge_score=r2_score(y_true,y_pre_ridge)
lasso_score=r2_score(y_true,y_pre_lasso)
decision_score=r2_score(y_true,y_pre_decision)
svr_score=r2_score(y_true,y_pre_svr)
display(knn_score,linear_score,ridge_score,lasso_score,decision_score,svr_score)<jupyter_output><empty_output><jupyter_text>The Results of deep learning regression are as folowing:<jupyter_code># set a 10-fold cross validation
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, Y, cv=kfold)
print("Results: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))# display score of our prediction
knn_score = r2_score(y_true,y_pre_knn)
linear_score=r2_score(y_true,y_pre_linear)
ridge_score=r2_score(y_true,y_pre_ridge)
lasso_score=r2_score(y_true,y_pre_lasso)
decision_score=r2_score(y_true,y_pre_decision)
svr_score=r2_score(y_true,y_pre_svr)
display(knn_score,linear_score,ridge_score,lasso_score,decision_score,svr_score)
# evaluate model with standardized dataset
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Standardized: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))<jupyter_output>Standardized: 1.00 (0.00) cosine_proximity<jupyter_text>The ramdom forest score and predictions are as following:<jupyter_code>rf_random.best_score_
best_random.score(test_features, test_labels)
np.set_printoptions(threshold=np.inf)  
predictions<jupyter_output><empty_output><jupyter_text>For Iowa dataset, the accuracy score of KNeighbors Regression,Linear Regression, Lasso Regression, Ridge Regression, Decision Tree Regression and Support Vector Regression models:<jupyter_code># display score of our prediction
knn_score = r2_score(y_true,y_pre_knn)
linear_score=r2_score(y_true,y_pre_linear)
ridge_score=r2_score(y_true,y_pre_ridge)
lasso_score=r2_score(y_true,y_pre_lasso)
decision_score=r2_score(y_true,y_pre_decision)
svr_score=r2_score(y_true,y_pre_svr)
display(knn_score,linear_score,ridge_score,lasso_score,decision_score,svr_score)<jupyter_output><empty_output><jupyter_text>The Results of deep learning regression are as folowing:<jupyter_code>kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, y, cv=kfold)
print("Results: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))
# evaluate model with standardized dataset
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, y, cv=kfold)
print("Standardized: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))<jupyter_output>Standardized: 1.00 (0.00) cosine_proximity<jupyter_text>The ramdom forest score and predictions are as following:<jupyter_code>rf_random.best_score_
best_random.score(test_features, test_labels)
np.set_printoptions(threshold=np.inf)  
predictions<jupyter_output><empty_output><jupyter_text>For California dataset, the accuracy score of KNeighbors Regression,Linear Regression, Lasso Regression, Ridge Regression, Decision Tree Regression and Support Vector Regression models:<jupyter_code># display score of our prediction
knn_score = r2_score(y_true,y_pre_knn)
linear_score=r2_score(y_true,y_pre_linear)
ridge_score=r2_score(y_true,y_pre_ridge)
lasso_score=r2_score(y_true,y_pre_lasso)
decision_score=r2_score(y_true,y_pre_decision)
svr_score=r2_score(y_true,y_pre_svr)
display(knn_score,linear_score,ridge_score,lasso_score,decision_score,svr_score)<jupyter_output><empty_output><jupyter_text>The Results of deep learning regression are as folowing:<jupyter_code>kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, Y, cv=kfold)
print("Results: %.2f (%.2f) cosine_proximity" % (results.mean(), results.std()))<jupyter_output>Results: nan (nan) cosine_proximity<jupyter_text>The ramdom forest score and predictions are as following:<jupyter_code>rf_random.best_score_
best_random.score(test_features, test_labels)
np.set_printoptions(threshold=np.inf)  
predictions<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08414807783578547, 'fraction_numerical': 0.012766967252017087, 'mean_word_length': 6.915477084898573, 'pattern_counts': {'":': 0, '<': 140, '<?xml version=': 0, '>': 140, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '36', 'dir': 'jupyter-structured-clean-dedup', 'id': '66595', 'license': 'MIT', 'n_tokens_mistral': 6816, 'n_tokens_neox': 6458, 'n_words': 2095, 'path': 'Final Project Blog-4.27.ipynb', 'repo_name': 'Fymoo/INFO7390', 'size': '21082'}