<jupyter_start><jupyter_text>Housing Prices ProblemThe goal of this notebook is to use regression to predict house prices based on several variables. Data set avalable here: https://www.kaggle.com/harlfoxem/housesalespredictionkc_house_data.csv<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn import metrics
from sklearn.ensemble import RandomForestRegressor
import warnings
warnings.filterwarnings("ignore")
%matplotlib inline
# suppress scientific notation
pd.set_option('display.float_format', lambda x: '%.3f' % x)
# read the data
df = pd.read_csv('input/kc_house_data.csv', parse_dates = ["date"])
df.head()<jupyter_output><empty_output><jupyter_text>Columns "id" and "date" don't have any meaning in relation to the sale price, let's drop them.<jupyter_code>df = df.drop(['id', 'date'],axis=1)
X = df.drop("price", axis=1)
y = df.price
df.describe()<jupyter_output><empty_output><jupyter_text>From inspection of data we can conclude that:1. Average price for a house  is 540 088\\$, maximum price is 7 700 000\\$2. Average number of bedrooms is 3.3 and bathrooms - 2.2, there is one house with 33 bedrooms3. Average lot size is 15 107 sqft4. And average living space is 2 079 sqftTime to do some data inspection to get a feeling of data set.<jupyter_code># Check if data contains null values
#df.isnull().sum()<jupyter_output><empty_output><jupyter_text>The data is clean, there are no null values. The numbers below show, that there are 9 824 houses with 3 bedrooms, 6882 houses with 4 bedrooms, one house with 11 and one with 33 bedrooms.<jupyter_code>X["bedrooms"].value_counts()
X['bedrooms'].value_counts().sort_index(inplace=True)
sns.countplot(X.bedrooms);<jupyter_output><empty_output><jupyter_text>There are 163 houses, which have a view to waterfront:<jupyter_code>X['waterfront'].value_counts()
X['grade'].value_counts()
X['grade'].value_counts().sort_index(inplace=True)
sns.countplot(X.grade);<jupyter_output><empty_output><jupyter_text>Plot correlation heatmap to see which features correlate with each other<jupyter_code>_, ax = plt.subplots(figsize = (12, 10))
    
colormap = sns.diverging_palette(220, 10, as_cmap = True)

sns.heatmap(df.corr(), annot=True, cmap = colormap);<jupyter_output><empty_output><jupyter_text>Based on the analysis of the above correlation matrix, we can see that __price__ strongly correlates with __sqft_living__, __bathrooms__ and __grade__.For now I'll use linear regression with just one variable to determine the price.<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y)

X_train = np.array(X_train["sqft_living"]).reshape(-1, 1)
y_train = np.array(y_train).reshape(-1, 1)
X_test = np.array(X_test["sqft_living"]).reshape(-1, 1)
y_test = np.array(y_test).reshape(-1, 1)
lr = LinearRegression()
lr.fit(X_train, y_train)
pred = lr.predict(X_test)
rmse = np.sqrt(metrics.mean_squared_error(y_test, pred))
print ("Linear regression with one variable")
print ("Mean squared error:", round(rmse, 2))
print('R-squared (training) ', round(lr.score(X_train, y_train), 3))
print('R-squared (testing) ', round(lr.score(X_test, y_test), 3))
_, ax = plt.subplots(figsize= (10, 6))
plt.scatter(X_test, y_test, color= 'darkblue', label = 'Original data', s=4)
plt.plot(X_test, lr.predict(X_test), color='red', label= 'Regression line (predicted)')
plt.xlabel('Living space (sqft)')
plt.ylabel('Price')
plt.legend();<jupyter_output><empty_output><jupyter_text>The results of regression with one variable are quite poor, let's try using several variablesVariables that are highly correlated with price are, as we have found out earlier: __bathrooms__ and __grade__.<jupyter_code>features = ["sqft_living", "bathrooms", "grade"]
X_train, X_test, y_train, y_test = train_test_split(X, y)
lr_multi = LinearRegression()
lr_multi.fit(X_train[features], y_train)
pred_multi = lr_multi.predict(X_test[features])
rmse_multi = np.sqrt(metrics.mean_squared_error(y_test, pred_multi))
print ("Linear regression with 3 variables")
print ("Mean squared error:", round(rmse_multi, 2))
print('R-squared (training) ', round(lr_multi.score(X_train[features], y_train), 3))
print('R-squared (testing) ', round(lr_multi.score(X_test[features], y_test), 3))<jupyter_output>Linear regression with 3 variables
Mean squared error: 247104.38
R-squared (training)  0.532
R-squared (testing)  0.553<jupyter_text>The results with 3 features are somewhat better. Let's add more varibles<jupyter_code>corr_coefficients = df.corr().price.sort_values(ascending=False)
corr_coefficients
# first coefficient (index 0) is the price, we need to exclude it
features = corr_coefficients.index[1:12].values
X_train, X_test, y_train, y_test = train_test_split(X, y)
lr_multi = LinearRegression()
lr_multi.fit(X_train[features], y_train)
pred_multi = lr_multi.predict(X_test[features])
rmse_multi = np.sqrt(metrics.mean_squared_error(y_test, pred_multi))
print ("Linear regression with 11 variables")
print ("Mean squared error:", round(rmse_multi, 2))
print('R-squared (training) ', round(lr_multi.score(X_train[features], y_train), 3))
print('R-squared (testing) ', round(lr_multi.score(X_test[features], y_test), 3))<jupyter_output>Linear regression with 11 variables
Mean squared error: 211428.62
R-squared (training)  0.656
R-squared (testing)  0.663<jupyter_text>Try another model: RandomForest<jupyter_code>rand_for = RandomForestRegressor()
rand_for.fit(X_train[features], y_train)
pred_for = rand_for.predict(X_test[features])
rmse_for = np.sqrt(metrics.mean_squared_error(y_test, pred_for))
print ("Random forest with 11 variables")
print ("Mean squared error:", round(rmse_for, 2))
print('R-squared (training) ', round(rand_for.score(X_train[features], y_train), 3))
print('R-squared (testing) ', round(rand_for.score(X_test[features], y_test), 3))<jupyter_output>Random forest with 11 variables
Mean squared error: 183939.31
R-squared (training)  0.964
R-squared (testing)  0.745
{'fraction_non_alphanumeric': 0.12179487179487179, 'fraction_numerical': 0.024326101249178174, 'mean_word_length': 7.113333333333333, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '213949', 'license': 'MIT', 'n_tokens_mistral': 2258, 'n_tokens_neox': 2051, 'n_words': 557, 'path': 'Housing Prices Regression/Housing_Prices.ipynb', 'repo_name': 'o-mrost/Machine-Learning', 'size': '6084'}