<jupyter_start><jupyter_text>Microsoft Capstone Project Mari Bisnis Dataset Subtheme: **Business** By: <NAME> - DAI007 - Data & AI<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
# Loading the dataset
df = pd.read_csv('/content/drive/My Drive/MariBisnis/MariBisnis.csv')
df.head()<jupyter_output><empty_output><jupyter_text>- Id - Unique ID for each home sold- date - Date of the home sale- price - Price of each home sold- bedrooms - Number of bedrooms- bathrooms - Number of bathrooms, where .5 accounts for a room with a toilet but no shower- sqft_living - Square footage of the apartments interior living space- sqft_lot - Square footage of the land space- floors - Number of floors- waterfront - A dummy variable for whether the apartment was overlooking the waterfront or not- view - An index from 0 to 4 of how good the view of the property wascondition - An index from 1 to 5 on the condition of the apartment,- grade - An index from 1 to 13, where 1-3 falls short of building construction and design, 7 has an average level of construction and design, and 11-13 have a high quality level of construction and design.- sqft_above - The square footage of the interior housing space that is above ground level- sqft_basement - The square footage of the interior housing space that is below ground level- yr_built - The year the house was initially built- yr_renovated - The year of the house’s last renovation- zipcode - What zipcode area the house is in- lat - Lattitude- long - Longitude- sqft_living15 - The square footage of interior housing living space for the nearest 15 neighbors- sqft_lot15 - The square footage of the land lots of the nearest 15 neighbors<jupyter_code>df.isnull().sum()
df.shape
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 21613 entries, 0 to 21612
Data columns (total 21 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   id             21613 non-null  int64  
 1   date           21613 non-null  object 
 2   price          21613 non-null  float64
 3   bedrooms       21613 non-null  int64  
 4   bathrooms      21613 non-null  float64
 5   sqft_living    21613 non-null  int64  
 6   sqft_lot       21613 non-null  int64  
 7   floors         21613 non-null  float64
 8   waterfront     21613 non-null  int64  
 9   view           21613 non-null  int64  
 10  condition      21613 non-null  int64  
 11  grade          21613 non-null  int64  
 12  sqft_above     21613 non-null  int64  
 13  sqft_basement  21613 non-null  int64  
 14  yr_built       21613 non-null  int64  
 15  yr_renovated   21613 non-null  int64  
 16  zipcode        21613 non-null  int64  
 17  lat            21613 non-null  float64
 18  long  [...]<jupyter_text>Only one categorical variable (object type) which is the date column. The rest are numerical, this is sufficient for predictions using machine learning algorithms. But first, we must do feature engineering on these columns.<jupyter_code>numerical = ['id', 'price','bedrooms',  'bathrooms',  'sqft_living','sqft_lot',
           'floors', 'waterfront','view', 'condition',  'grade','sqft_above','sqft_basement','yr_built','yr_renovated','zipcode','lat','long','sqft_living15','sqft_lot15']
# Since only categorical = ['date'], we will just omit the date feature column as it holds no significance in prediction goals.
df_original = df
df = df[numerical]<jupyter_output><empty_output><jupyter_text>Outlier Detection<jupyter_code>from sklearn.decomposition import PCA
from matplotlib import rcParams
from sklearn.preprocessing import QuantileTransformer
import scipy.stats as stats
from sklearn.cluster import DBSCAN
# Detecting outliers of price with Z-Score
out=[]
def Zscore_outlier(df):
    m = np.mean(df)
    sd = np.std(df)
    for i in df: 
        z = (i-m)/sd
        if np.abs(z) > 3: 
            out.append(i)
    print("Outliers:",out)
Zscore_outlier(df_original['price'])
sns.boxplot(df['price'])
plt.title("Box Plot before outlier removing")
plt.show()
def drop_outliers(df, field_name):
    iqr = 1.5 * (np.percentile(df[field_name], 75) - np.percentile(df[field_name], 25))
    df.drop(df[df[field_name] > (iqr + np.percentile(df[field_name], 75))].index, inplace=True)
    df.drop(df[df[field_name] < (np.percentile(df[field_name], 25) - iqr)].index, inplace=True)
drop_outliers(df, 'price')
sns.boxplot(df['price'])
plt.title("Box Plot after outlier removing")
plt.show()
df.shape # After removing
df_original.shape # Before removing<jupyter_output><empty_output><jupyter_text>Feature Analysis & Engineering<jupyter_code>features = df.columns
traindf = df.dropna()
traindf = pd.DataFrame(traindf,columns=features)
traindf
plt.subplots(figsize=(9,7))
corr=traindf.drop(['id'],1).corr()
sns.heatmap(corr,square=True)
# Okay, let's assume a good house would have a price of >200000
y = traindf['price']>=200000
X = traindf.copy()
del X['price']
X.head()
len(X.columns)
feature_name = list(X.columns)
# no of maximum features I need to select 
num_feats=10 # So from 19 -> 10 only
def cor_selector(X, y,num_feats):
    cor_list = []
    feature_name = X.columns.tolist()
    # calculate the correlation with y for each feature
    for i in X.columns.tolist():
        cor = np.corrcoef(X[i], y)[0, 1]
        cor_list.append(cor)
    # replace NaN with 0
    cor_list = [0 if np.isnan(i) else i for i in cor_list]
    # feature name
    cor_feature = X.iloc[:,np.argsort(np.abs(cor_list))[-num_feats:]].columns.tolist()
    # feature selection? 0 for not select, 1 for select
    cor_support = [True if i in cor_feature else False for i in feature_name]
    return cor_support, cor_feature

# Pearson Correlation:
cor_support, cor_feature = cor_selector(X, y,num_feats)
print(str(len(cor_feature)), 'selected features')
# Chi Squared:
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
from sklearn.preprocessing import MinMaxScaler
X_norm = MinMaxScaler().fit_transform(X)
chi_selector = SelectKBest(chi2, k=num_feats)
chi_selector.fit(X_norm, y)
chi_support = chi_selector.get_support()
chi_feature = X.loc[:,chi_support].columns.tolist()
print(str(len(chi_feature)), 'selected features')
# Recursive Feature Elimination
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
rfe_selector = RFE(estimator=LogisticRegression(), n_features_to_select=num_feats, step=10, verbose=5)
rfe_selector.fit(X_norm, y)
rfe_support = rfe_selector.get_support()
rfe_feature = X.loc[:,rfe_support].columns.tolist()
print(str(len(rfe_feature)), 'selected features')
# Embedded Method
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier

embeded_rf_selector = SelectFromModel(RandomForestClassifier(n_estimators=100), max_features=num_feats)
embeded_rf_selector.fit(X, y)

embeded_rf_support = embeded_rf_selector.get_support()
embeded_rf_feature = X.loc[:,embeded_rf_support].columns.tolist()
print(str(len(embeded_rf_feature)), 'selected features')
embeded_rf_support = embeded_rf_selector.get_support()
embeded_rf_feature = X.loc[:,embeded_rf_support].columns.tolist()
print(str(len(embeded_rf_feature)), 'selected features')
pd.set_option('display.max_rows', None)
# put all selection together
feature_selection_df = pd.DataFrame({'Feature':feature_name, 'Pearson':cor_support, 'Chi-2':chi_support, 'RFE':rfe_support,'Random Forest':embeded_rf_support})
# count the selected times for each feature
feature_selection_df['Total'] = np.sum(feature_selection_df, axis=1)
# display the top 100
feature_selection_df = feature_selection_df.sort_values(['Total','Feature'] , ascending=False)
feature_selection_df.index = range(1, len(feature_selection_df)+1)
feature_selection_df.head(num_feats)<jupyter_output><empty_output><jupyter_text>So we will be taking these features for regression ML models:<jupyter_code>features_selected = ['sqft_living15','sqft_living','sqft_above','lat','grade','sqft_basement','bathrooms','view','floors','bedrooms']
print("The distinct values of grade: ", df['grade'].unique())
print("The distinct values of view: ", df['view'].unique())<jupyter_output>The distinct values of grade:  [ 7  6  8  9  5 10 11  4 12  3  1]
The distinct values of view:  [0 3 4 2 1]<jupyter_text>Linear Regression<jupyter_code>X = df[features_selected]
y = df['price']
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
from sklearn import metrics
from sklearn.model_selection import cross_val_score

def cross_val(model):
    pred = cross_val_score(model, X, y, cv=10)
    return pred.mean()

def print_evaluate(true, predicted):  
    mae = metrics.mean_absolute_error(true, predicted)
    mse = metrics.mean_squared_error(true, predicted)
    rmse = np.sqrt(metrics.mean_squared_error(true, predicted))
    r2_square = metrics.r2_score(true, predicted)
    print('MAE:', mae)
    print('MSE:', mse)
    print('RMSE:', rmse)
    print('R2 Square', r2_square)
    print('__________________________________')
    
def evaluate(true, predicted):
    mae = metrics.mean_absolute_error(true, predicted)
    mse = metrics.mean_squared_error(true, predicted)
    rmse = np.sqrt(metrics.mean_squared_error(true, predicted))
    r2_square = metrics.r2_score(true, predicted)
    return mae, mse, rmse, r2_square
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('std_scalar', StandardScaler())
])

X_train = pipeline.fit_transform(X_train)
X_test = pipeline.transform(X_test)
# Importing Linear Regression from sklearn
from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression(normalize=True)
lin_reg.fit(X_train,y_train)
# print the intercept
print(lin_reg.intercept_)
coeff_df = pd.DataFrame(lin_reg.coef_, X.columns, columns=['Coefficient'])
coeff_df
pred = lin_reg.predict(X_test)
test_pred = lin_reg.predict(X_test)
train_pred = lin_reg.predict(X_train)

print('Test set evaluation:\n_____________________________________')
print_evaluate(y_test, test_pred)
print('Train set evaluation:\n_____________________________________')
print_evaluate(y_train, train_pred)
comparison_linreg = y_test.to_frame()
comparison_linreg.columns = ['Actual value']
comparison_linreg['Predicted value'] = test_pred
comparison_linreg.head(10)
# Making a combined df for all the results

results_df = pd.DataFrame(data=[["Linear Regression", *evaluate(y_test, test_pred) , cross_val(LinearRegression())]], 
                          columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
results_df<jupyter_output><empty_output><jupyter_text>Random Forest Regression<jupyter_code>from sklearn import metrics
from sklearn import preprocessing
from sklearn.neighbors import KNeighborsRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
model = RandomForestRegressor(n_jobs=-1)

# Try different numbers of n_estimators - this will take a minute or so
estimators = np.arange(10, 450, 10)
scores = []
for n in estimators:
    model.set_params(n_estimators=n)
    model.fit(X_train, y_train)
    scores.append(model.score(X_test, y_test))
plt.figure(figsize=(7, 5))
plt.title("Effect of Estimators")
plt.xlabel("no. estimator")
plt.ylabel("score")
plt.plot(estimators, scores)
results = list(zip(estimators,scores))
predictions = model.predict(X_test)
'Mean Absolute Error:', metrics.mean_absolute_error(y_test, predictions)

'Mean Squared Error:', metrics.mean_squared_error(y_test, predictions)

'Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, predictions))


model = RandomForestRegressor(n_estimators=200, 
                               bootstrap = True)
# Fit on training data
model.fit(X_train, y_train)

# Actual class predictions
rf_predictions = model.predict(X_test)

comparison_fy21= y_test.to_frame()
comparison_fy21.columns = ['Actual Price']
comparison_fy21['Predicted Price'] = rf_predictions
comparison_fy21.head(10)

train_predictions = model.predict(X_train)
# Print RF evaluations: 
print_evaluate(y_test, predictions)
print('Test set evaluation:\n_____________________________________')
print_evaluate(y_test, predictions) # x test
print('Train set evaluation:\n_____________________________________')
print_evaluate(y_train, train_predictions) # x train
results_df_2 = pd.DataFrame(data=[["Random Forest Regressor", *evaluate(y_test, predictions), 0]], 
                            columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
results_df = results_df.append(results_df_2, ignore_index=True)
results_df<jupyter_output><empty_output><jupyter_text>Robust RegressionRobust regression is a form of regression analysis designed to overcome some limitations of traditional parametric and non-parametric methods. Robust regression methods are designed to be not overly affected by violations of assumptions by the underlying data-generating process.One instance in which robust estimation should be considered is when there is a strong suspicion of heteroscedasticity.A common situation in which robust estimation is used occurs when the data contain outliers. In the presence of outliers that do not come from the same data-generating process as the rest of the data, least squares estimation is inefficient and can be biased. Because the least squares predictions are dragged towards the outliers, and because the variance of the estimates is artificially inflated, the result is that outliers can be masked. (In many situations, including some areas of geostatistics and medical statistics, it is precisely the outliers that are of interest.)  Using RANSAC:Random sample consensus (RANSAC) is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers, when outliers are to be accorded no influence on the values of the estimates. Therefore, it also can be interpreted as an outlier detection method.A basic assumption is that the data consists of "inliers", i.e., data whose distribution can be explained by some set of model parameters, though may be subject to noise, and "outliers" which are data that do not fit the model. The outliers can come, for example, from extreme values of the noise or from erroneous measurements or incorrect hypotheses about the interpretation of data. RANSAC also assumes that, given a (usually small) set of inliers, there exists a procedure which can estimate the parameters of a model that optimally explains or fits this data.<jupyter_code>from sklearn.linear_model import RANSACRegressor

model = RANSACRegressor(base_estimator=LinearRegression(), max_trials=100)
model.fit(X_train, y_train)

test_pred = model.predict(X_test)
train_pred = model.predict(X_train)

print('Test set evaluation:\n_____________________________________')
print_evaluate(y_test, test_pred)
print('====================================')
print('Train set evaluation:\n_____________________________________')
print_evaluate(y_train, train_pred)<jupyter_output>Test set evaluation:
_____________________________________
MAE: 101740.43529127826
MSE: 18334693128.912533
RMSE: 135405.66136211785
R2 Square 0.5649223749779188
__________________________________
====================================
Train set evaluation:
_____________________________________
MAE: 102385.41724872409
MSE: 18545756301.458145
RMSE: 136182.8047201927
R2 Square 0.5780777999372472
__________________________________<jupyter_text>To conclude, if we excluded feature engineering with RANSAC and Robust Linear Regression, it takes approximately 1000 trials to get to R squared of 54%, whereas with feature engineering it only took us 100 trials.<jupyter_code>comparison_robust_regression= y_test.to_frame()
comparison_robust_regression.columns = ['Actual Price']
comparison_robust_regression['Predicted Price'] = test_pred
comparison_robust_regression.head(10)
results_df_2 = pd.DataFrame(data=[["Robust Regression", *evaluate(y_test, test_pred) , cross_val(RANSACRegressor())]], 
                            columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', "Cross Validation"])
results_df = results_df.append(results_df_2, ignore_index=True)
results_df<jupyter_output><empty_output><jupyter_text>Stochastic Gradient DescentGradient Descent is a very generic optimization algorithm capable of finding optimal solutions to a wide range of problems. The general idea of Gradient Sescent is to tweak parameters iteratively in order to minimize a cost function. Gradient Descent measures the local gradient of the error function with regards to the parameters vector, and it goes in the direction of descending gradient. Once the gradient is zero, you have reached a minimum.<jupyter_code>from sklearn.linear_model import SGDRegressor

sgd_reg = SGDRegressor(n_iter_no_change=300, penalty=None, eta0=0.0001, max_iter=100000)
sgd_reg.fit(X_train, y_train)

test_pred = sgd_reg.predict(X_test)
train_pred = sgd_reg.predict(X_train)

print('Test set evaluation:\n_____________________________________')
print_evaluate(y_test, test_pred)
print('====================================')
print('Train set evaluation:\n_____________________________________')
print_evaluate(y_train, train_pred)
comparison_stochastic= y_test.to_frame()
comparison_stochastic.columns = ['Actual Price']
comparison_stochastic['Predicted Price'] = test_pred
comparison_stochastic.head(10)
results_df_2 = pd.DataFrame(data=[["Stochastic Gradient Descent", *evaluate(y_test, test_pred), 0]], 
                            columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
results_df = results_df.append(results_df_2, ignore_index=True)
results_df<jupyter_output><empty_output><jupyter_text>Elastic Net RegressionA linear regression model trained with L1 and L2 prior as regularizer.This combination allows for learning a sparse model where few of the weights are non-zero like Lasso, while still maintaining the regularization properties of Ridge.Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.A practical advantage of trading-off between Lasso and Ridge is it allows Elastic-Net to inherit some of Ridge’s stability under rotation.<jupyter_code>from sklearn.linear_model import ElasticNet

model = ElasticNet(alpha=0.1, l1_ratio=0.9, selection='random', random_state=42)
model.fit(X_train, y_train)

test_pred = model.predict(X_test)
train_pred = model.predict(X_train)

print('Test set evaluation:\n_____________________________________')
print_evaluate(y_test, test_pred)
print('====================================')
print('Train set evaluation:\n_____________________________________')
print_evaluate(y_train, train_pred)
comparison_stochastic= y_test.to_frame()
comparison_stochastic.columns = ['Actual Price']
comparison_stochastic['Predicted Price'] = test_pred
comparison_stochastic.head(10)
results_df_2 = pd.DataFrame(data=[["Elastic Net Regression", *evaluate(y_test, test_pred), 0]], 
                            columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
results_df = results_df.append(results_df_2, ignore_index=True)
results_df<jupyter_output><empty_output><jupyter_text>Artificial Neural NetworkDue to the low R squared values from the regressions, we will try using Artificial Neural Network approaches in hopes of increasing its accuracy parameter.<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Dense, Activation, Dropout
from tensorflow.keras.optimizers import Adam

X_train = np.array(X_train)
X_test = np.array(X_test)
y_train = np.array(y_train)
y_test = np.array(y_test)

model = Sequential()

model.add(Dense(X_train.shape[1], activation='relu'))
model.add(Dense(32, activation='relu'))
# model.add(Dropout(0.2))

model.add(Dense(64, activation='relu'))
# model.add(Dropout(0.2))

model.add(Dense(128, activation='relu'))
# model.add(Dropout(0.2))

model.add(Dense(512, activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(1))

model.compile(optimizer=Adam(0.00001), loss='mse')

r = model.fit(X_train, y_train,
              validation_data=(X_test,y_test),
              batch_size=1,
              epochs=100)<jupyter_output>Epoch 1/100
14326/14326 [==============================] - 39s 3ms/step - loss: 270321991680.0000 - val_loss: 252546727936.0000
Epoch 2/100
14326/14326 [==============================] - 39s 3ms/step - loss: 181579005952.0000 - val_loss: 63668543488.0000
Epoch 3/100
14326/14326 [==============================] - 38s 3ms/step - loss: 37629505536.0000 - val_loss: 33862787072.0000
Epoch 4/100
14326/14326 [==============================] - 39s 3ms/step - loss: 31403929600.0000 - val_loss: 31133843456.0000
Epoch 5/100
14326/14326 [==============================] - 39s 3ms/step - loss: 29079986176.0000 - val_loss: 28863307776.0000
Epoch 6/100
14326/14326 [==============================] - 34s 2ms/step - loss: 27237251072.0000 - val_loss: 27040190464.0000
Epoch 7/100
14326/14326 [==============================] - 34s 2ms/step - loss: 25672257536.0000 - val_loss: 25490880512.0000
Epoch 8/100
14326/14326 [==============================] - 39s 3ms/step - loss: 24433666048.0000 - val_loss: 242344[...]<jupyter_text><jupyter_code>loss_df = pd.DataFrame(r.history)
plt.figure(figsize=(10,5))
sns.lineplot(data = loss_df["loss"])
sns.lineplot(data = loss_df["val_loss"])
test_pred = model.predict(X_test)
train_pred = model.predict(X_train)

print('Test set evaluation:\n_____________________________________')
print_evaluate(y_test, test_pred)

print('Train set evaluation:\n_____________________________________')
print_evaluate(y_train, train_pred)
plt.figure(figsize=(12, 10))

ax1 = sns.distplot(df['price'], hist=False, color="r", label="Actual Value") # hist=True
sns.distplot(test_pred, hist=False, color="b", label="Predicted Values" , ax=ax1)


plt.title('Actual vs Predicted Values for House')
plt.xlabel('Price')
plt.ylabel('Proportion of House')

plt.show()
plt.close()
results_df_2 = pd.DataFrame(data=[["Artficial Neural Network", *evaluate(y_test, test_pred), 0]], 
                            columns=['Model', 'MAE', 'MSE', 'RMSE', 'R2 Square', 'Cross Validation'])
results_df = results_df.append(results_df_2, ignore_index=True)
results_df
# Convert the results_df to an excel
results_df.to_excel("results_df_regression.xlsx")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11423542354235423, 'fraction_numerical': 0.04026402640264026, 'mean_word_length': 5.591067285382831, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '52864', 'license': 'MIT', 'n_tokens_mistral': 7780, 'n_tokens_neox': 6932, 'n_words': 1993, 'path': '[2] Mari Bisnis/MariBisnis_Microsoft_Capstone_Trial.ipynb', 'repo_name': 'ikanberjalandidarat/MicrosoftCapstoneKMMI', 'size': '22736'}