<jupyter_start><jupyter_text>I will begin with some data exploration<jupyter_code>train = pd.read_csv('../input/kc_house_data.csv')
train.info()<jupyter_output><empty_output><jupyter_text>Except date, all features are numeric. Let's have some insight on data.<jupyter_code>train.head()<jupyter_output><empty_output><jupyter_text>From first sight, data seems OK. Let's dive deeper to understand it.'<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
train.hist(bins=30, figsize=(14,12))
plt.show()<jupyter_output><empty_output><jupyter_text>Notes:* There is some sparse data. So we have to look for possible outliers* Create dummy variables for sqft_basement and yr_renovated because 0 value means that value is absent<jupyter_code>train.describe()<jupyter_output><empty_output><jupyter_text>Let's now look for correlations with price feature<jupyter_code>correlations = train.corr()

def heatmap_gen(corr):
    sns.set(style='white')
    # Generate a mask for the upper triangle
    mask = np.zeros_like(corr, dtype=np.bool)
    mask[np.triu_indices_from(mask)] = True

    # Set up the matplotlib figure
    plt.figure(figsize=(9,8))
    cmap = sns.diverging_palette(220, 10, as_cmap=True)
    # Draw the heatmap with the mask and correct aspect ratio
    sns.heatmap(corr * 100, annot=True, fmt='.0f', mask=mask, cmap=cmap, center=0,
            square=True, linewidths=.5, cbar_kws={"shrink": .5})
    

heatmap_gen(correlations)<jupyter_output><empty_output><jupyter_text>It is clear that there is some linear correlation between price and (grade and sqft_living)<jupyter_code>sns.pairplot(train[['sqft_living', 'grade', 'bathrooms', 'price']])
plt.show()
sns.boxplot(x='grade', y='price', data=train)
plt.show()
train.plot.scatter(x='grade', y='price', s=train.sqft_living * 0.05,alpha=0.2, c='waterfront',colormap='plasma',edgecolors='grey', figsize=(10,10))
plt.show()<jupyter_output><empty_output><jupyter_text>According to last graph, we can roughly say that price increases with increasing grade and sqft_living. Also, for the same grade and size, houses in front of water seems to have higher prices.  Now let's verify data consistency<jupyter_code>train.isnull().any()
for feature in ['bedrooms', 'bathrooms', 'floors', 'view', 'condition', 'grade', 'yr_built', 'yr_renovated', 'waterfront']:
    print(feature, train[feature].sort_values().unique())<jupyter_output><empty_output><jupyter_text>I'm going to verify some suspecious values like bedrooms=33 or bathrooms=0<jupyter_code>train[train.bathrooms == 0]<jupyter_output><empty_output><jupyter_text>It looks like it is not an outlier and I'm not sure if this is a mistake so I'm going to leave these rows<jupyter_code>sns.violinplot(x='bedrooms', data=train)
plt.show()
train[train.bedrooms > 30]<jupyter_output><empty_output><jupyter_text>I'm going to delete this row because I think it is a big outlier<jupyter_code>train_cp = train.copy()
train_cp = train_cp[train_cp.bedrooms < 30]<jupyter_output><empty_output><jupyter_text>Now I'm going to verify data consistency. For example, is yr_renovated always bigger than yr_built<jupyter_code>train_cp['diff_renov_built'] = train_cp.yr_renovated - train_cp.yr_built
train_cp[(train_cp.diff_renov_built < 0) & (train_cp.yr_renovated !=0)]<jupyter_output><empty_output><jupyter_text>Now let's create dummy variables for renovated and basement<jupyter_code>train_cp['isrenovated'] = (train_cp.yr_renovated != 0).astype(int)
train_cp['hasbasement'] = (train_cp.sqft_basement != 0).astype(int)<jupyter_output><empty_output><jupyter_text>Now let's procede to feature engineering.  I will extract year sold from date feature<jupyter_code>train_cp['yr_sold'] = train_cp['date'].str[:4].astype(int)
train_cp['age'] = train_cp.yr_sold - train_cp.yr_built
train_cp['yr_after_renov'] = train_cp.yr_sold - train_cp.yr_renovated - train_cp.yr_built * (1 - train_cp.isrenovated)<jupyter_output><empty_output><jupyter_text>Now I want to see if there are some clusters in house structures.<jupyter_code>train_cp.plot.scatter(x='bedrooms', y='bathrooms', alpha=0.1, s=train_cp.price*0.0001, figsize=(10,10))
plt.show()<jupyter_output><empty_output><jupyter_text>It is not clear if there is interessant clusters or if it is possible to extract some. If you have some idea It would be nice of you to share it with me :)<jupyter_code>train_cp.plot.scatter(x='long', y='lat', c='price', colormap='jet', alpha=0.3, edgecolor='grey', figsize=(10,10))
plt.show()<jupyter_output><empty_output><jupyter_text>From this graph, I see that area within lat > 47.5 contains more expensive houses. So, I'm going to create a feature to distinguish this area<jupyter_code>train_cp['lat>47.5'] = (train_cp.lat >= 47.5).astype(int)
train_cp.head()<jupyter_output><empty_output><jupyter_text>Now I'm going to drop non relevant features<jupyter_code>train_cp = train_cp.drop(['date', 'yr_built', 'yr_renovated', 'zipcode', 'lat', 'long', 'yr_sold'], axis=1)
train_cp = train_cp.set_index('id')<jupyter_output><empty_output><jupyter_text>Now it's time to train some models.<jupyter_code>from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Now I'm going to split data and prepare pipelines<jupyter_code>y = train_cp.price
X = train_cp.drop('price', axis=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import ElasticNet
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.preprocessing import StandardScaler
pipelines = {
    'elsnet' : make_pipeline(StandardScaler(), ElasticNet(random_state=123)),
    'rf' : make_pipeline(StandardScaler(), RandomForestRegressor(random_state=123)),
    'gb' : make_pipeline(StandardScaler(), GradientBoostingRegressor(random_state=123)),
}<jupyter_output><empty_output><jupyter_text>Prepare hyperparameters<jupyter_code>elsnet_hyper = {
'elasticnet__alpha': [0.05, 0.1, 0.5, 1, 5, 10],
'elasticnet__l1_ratio' : [0.1, 0.3, 0.5, 0.7, 0.9]
}

rf_hyper = {
'randomforestregressor__n_estimators' : [100, 200],
'randomforestregressor__max_features': ['auto', 'sqrt', 0.33],
}

gb_hyper = {
'gradientboostingregressor__n_estimators': [100, 200],
'gradientboostingregressor__learning_rate' : [0.05, 0.1, 0.2],
'gradientboostingregressor__max_depth': [1, 3, 5]
}

hyper={
    'elsnet': elsnet_hyper,
    'rf': rf_hyper,
    'gb': gb_hyper
}
from sklearn.model_selection import GridSearchCV
fitted_models = {}
for name , pipeline in pipelines.items():
    model = GridSearchCV(pipeline , hyper[name], cv=10)
    model.fit(X_train , y_train)
    fitted_models[name] = model
    print(name)
for name, model in fitted_models.items():
    print(name, model.best_score_)
from sklearn.metrics import r2_score
from sklearn.metrics import mean_absolute_error
for name , model in fitted_models.items():
    pred = model.predict(X_test)
    print( name )
    print( '------------' )
    print( 'R^2:', r2_score(y_test , pred ))
    print( 'MAE:', mean_absolute_error(y_test , pred))
    print()
fitted_models['rf'].best_estimator_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12823725981620718, 'fraction_numerical': 0.01921470342522974, 'mean_word_length': 6.87609649122807, 'pattern_counts': {'":': 1, '<': 83, '<?xml version=': 0, '>': 85, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '548766', 'license': 'MIT', 'n_tokens_mistral': 2689, 'n_tokens_neox': 2538, 'n_words': 570, 'path': 'downloaded_kernels/house_sales/kernel_103.ipynb', 'repo_name': 'josepablocam/common-code-extraction', 'size': '7182'}