<jupyter_start><jupyter_text>Data and Model Exploration<jupyter_code># import necessary libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from datetime import datetime, date
from scipy import stats
from sklearn.metrics import mean_squared_error
from math import sqrt


# magic word for producing visualizations in notebook
%matplotlib inline
# load, understand, explore the dataset
houses = pd.read_json("street_group_data_science_bedrooms_test.json", lines=True)
houses
houses.property_type.value_counts(normalize=True)
houses.bedrooms.value_counts().sort_index()<jupyter_output><empty_output><jupyter_text>Check correlations between different features, if there is high collinearity between two features, it suggests that one of these features would be redundant<jupyter_code>corr = houses.corr()
plt.figure(figsize=(20,10))

sns.heatmap(corr, xticklabels=corr.columns.values, yticklabels=corr.columns.values);
from sklearn import preprocessing<jupyter_output><empty_output><jupyter_text>Apply Label encoding for the feature `property_type` to convert it into numerical category<jupyter_code>le = preprocessing.LabelEncoder()
le.fit(houses.property_type)
le.classes_
houses['property_type'] = le.transform(houses['property_type'])
houses
# more exploration - following some intuition

houses['lat'] = np.radians(houses['latitude'])
houses['lon'] = np.radians(houses['longitude'])
houses
houses[['lat','lon']].to_numpy()
houses[houses['number_habitable_rooms']!=houses['number_heated_rooms']]
import statsmodels.api as sm
from scipy import stats
X = houses.drop(['lat', 'lon', 'bedrooms'], axis=1)
y = houses['bedrooms']<jupyter_output><empty_output><jupyter_text>I chose to explore the linear regression first since we have to predict number of rooms which is a continuous variable.<jupyter_code>X2 = sm.add_constant(X)
est = sm.OLS(y, X2)
est2 = est.fit()
print(est2.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:               bedrooms   R-squared:                       0.651
Model:                            OLS   Adj. R-squared:                  0.651
Method:                 Least Squares   F-statistic:                 2.330e+05
Date:                Sun, 21 Nov 2021   Prob (F-statistic):               0.00
Time:                        19:56:28   Log-Likelihood:            -8.7115e+05
No. Observations:             1000000   AIC:                         1.742e+06
Df Residuals:                  999991   BIC:                         1.742e+06
Df Model:                           8                                         
Covariance Type:            nonrobust                                         
==========================================================================================
                             coef    std[...]<jupyter_text>From the heatmap it is evident that the features `number_habitable_rooms` & `number_heated_rooms` and `estimated_min_price` & `estimated_max_price` are highly correlated to each other. This explains the note which says there is an indication of strong multicollinearity<jupyter_code>rms2 = sqrt(mean_squared_error(y, np.floor(est2.predict(X2))))
rms2<jupyter_output><empty_output><jupyter_text>I explored the p-values and the coefficients of each variable but looking at the p-values I thought if exploring stepwise regression would be useful since it chooses the most significant variables first.<jupyter_code>est2.params.sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>I explored stepwise regression, but it did not prove to be helpful.<jupyter_code>import statsmodels.formula.api as smf


def forward_selected(data, response):
    """Linear model designed by forward selection.

    Parameters:
    -----------
    data : pandas DataFrame with all possible predictors and response

    response: string, name of response column in data

    Returns:
    --------
    model: an "optimal" fitted statsmodels linear model
           with an intercept
           selected by forward selection
           evaluated by adjusted R-squared
    """
    remaining = set(data.columns)
    remaining.remove(response)
    selected = []
    current_score, best_new_score = 0.0, 0.0
    while remaining and current_score == best_new_score:
        scores_with_candidates = []
        for candidate in remaining:
            formula = "{} ~ {} + 1".format(response,
                                           ' + '.join(selected + [candidate]))
            score = smf.ols(formula, data).fit().rsquared_adj
            scores_with_candidates.append((score, candidate))
        scores_with_candidates.sort()
        best_new_score, best_candidate = scores_with_candidates.pop()
        if current_score < best_new_score:
            remaining.remove(best_candidate)
            selected.append(best_candidate)
            current_score = best_new_score
    formula = "{} ~ {} + 1".format(response,
                                   ' + '.join(selected))
    model = smf.ols(formula, data).fit()
    return model
model = forward_selected(houses.drop(['lat', 'lon',
                                      'estimated_min_price', 'number_habitable_rooms'], axis=1), 'bedrooms')
print(model.model.formula)
print(model.rsquared_adj)
y<jupyter_output><empty_output><jupyter_text>I explored Ridge and Lasso regression as well but the R2 score did not improve at all. It was also observed that normalising the data was not of much help.<jupyter_code>from sklearn.linear_model import Ridge
clf = Ridge(alpha=0, normalize=True)
clf.fit(X, y)
clf.score(X, y)
from sklearn import linear_model
clf = linear_model.Lasso(alpha=0.0)
clf.fit(X, y)
clf.score(X, y)<jupyter_output><empty_output><jupyter_text>Finally when I explored Polynomial regression with degrees 2 and 3, I chose the model with degree=2 since it proved to be robust than other models with the R2 score (`=0.692`) the highest among all the explored model and Root Mean Squared error (`=0.827`) the least.<jupyter_code>from sklearn.preprocessing import PolynomialFeatures


poly = PolynomialFeatures(2)
degree2 = poly.fit_transform(X)
poly4 = PolynomialFeatures(3)
degree3 = poly4.fit_transform(X)

poly3 = PolynomialFeatures(4)
degree4 = poly3.fit_transform(X)
degree2.shape
X.shape
X3 = sm.add_constant(degree2)
est3 = sm.OLS(y, X3)
est4 = est3.fit()
print(est4.summary())
est4.rsquared.round(3)
np.floor(est4.predict(X3))
from sklearn.metrics import mean_squared_error
from math import sqrt

rms = sqrt(mean_squared_error(y, np.floor(est4.predict(X3))))
rms
degree3.shape
degree4.shape
X5 = sm.add_constant(degree3)
est5 = sm.OLS(y, X5)
est6 = est5.fit()
print(est6.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:               bedrooms   R-squared:                       0.607
Model:                            OLS   Adj. R-squared:                  0.607
Method:                 Least Squares   F-statistic:                 3.288e+04
Date:                Sun, 21 Nov 2021   Prob (F-statistic):               0.00
Time:                        19:58:14   Log-Likelihood:            -9.3012e+05
No. Observations:             1000000   AIC:                         1.860e+06
Df Residuals:                  999952   BIC:                         1.861e+06
Df Model:                          47                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>Test Data Creation<jupyter_code>houses2 = pd.read_json("street_group_data_science_bedrooms_test.json", lines=True)
del houses2['bedrooms']
houses2
houses2.to_json('test_data.json')
pd.read_json("test_data.json")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.128192350815996, 'fraction_numerical': 0.024791329263734896, 'mean_word_length': 2.8933074684772064, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '661482', 'license': 'MIT', 'n_tokens_mistral': 2460, 'n_tokens_neox': 2248, 'n_words': 594, 'path': 'Data and Model Exploration.ipynb', 'repo_name': 'isammitr/predict-number-of-rooms', 'size': '8027'}