<jupyter_start><jupyter_text>Attributes1. CRIM : per capita crime rate by town2. ZN : proportion of residential land zoned for lots over 25,000 sq.ft3. INDUS : proportion of non-retail business acres per town4. CHAS : Charles River dummy variable(=1 if tract bounds river;0 otherwise)5. NOX : nitric oxides concentration(parts per 10 million)6. RM : average number of rooms per dwelling7. AGE : proportion of owner-occupied units built prior to 19408. DIS : weighted distances to five Boston employment centers9. RAD : index of accessibility to radial highways10. TAX : full-value property-tax rate per ten thousand dollars11. PTRATIO : pupil-teacher ratio by town12. B: 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town13. LSTAT : percent of lower status of the population14. MEDV : Median value of owner-occupied homes in $1000s  Loading the dataset<jupyter_code># Load libraries
import numpy 
from numpy import arange
from matplotlib import pyplot
from pandas import read_csv
from pandas import set_option
from pandas.plotting import scatter_matrix
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Lasso
from sklearn.linear_model import ElasticNet
from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.svm import SVR
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.ensemble import ExtraTreesRegressor
from sklearn.ensemble import AdaBoostRegressor
from sklearn.metrics import mean_squared_error
# Load dataset
filename = 'housing.csv'
names = ['CRIM','ZN','INDUS','CHAS','NOX','RM','AGE','DIS','RAD','TAX','PTRATIO','B','LSTAT','MEDV']
dataset = read_csv(filename,delim_whitespace=True,names=names)<jupyter_output><empty_output><jupyter_text>Analyze data<jupyter_code>## Descriptive statistics

#shape
print(dataset.shape)
# types
print(dataset.dtypes)
# head
print(dataset.head(20))
# descriptions
set_option('precision',1)
print(dataset.describe())
# correlations
set_option('precision',2)
print(dataset.corr(method='pearson'))<jupyter_output>CRIM    ZN  INDUS      CHAS  ...  PTRATIO     B  LSTAT  MEDV
CRIM     1.00 -0.20   0.41 -5.59e-02  ...     0.29 -0.39   0.46 -0.39
ZN      -0.20  1.00  -0.53 -4.27e-02  ...    -0.39  0.18  -0.41  0.36
INDUS    0.41 -0.53   1.00  6.29e-02  ...     0.38 -0.36   0.60 -0.48
CHAS    -0.06 -0.04   0.06  1.00e+00  ...    -0.12  0.05  -0.05  0.18
NOX      0.42 -0.52   0.76  9.12e-02  ...     0.19 -0.38   0.59 -0.43
RM      -0.22  0.31  -0.39  9.13e-02  ...    -0.36  0.13  -0.61  0.70
AGE      0.35 -0.57   0.64  8.65e-02  ...     0.26 -0.27   0.60 -0.38
DIS     -0.38  0.66  -0.71 -9.92e-02  ...    -0.23  0.29  -0.50  0.25
RAD      0.63 -0.31   0.60 -7.37e-03  ...     0.46 -0.44   0.49 -0.38
TAX      0.58 -0.31   0.72 -3.56e-02  ...     0.46 -0.44   0.54 -0.47
PTRATIO  0.29 -0.39   0.38 -1.22e-01  ...     1.00 -0.18   0.37 -0.51
B       -0.39  0.18  -0.36  4.88e-02  ...    -0.18  1.00  -0.37  0.33
LSTAT    0.46 -0.41   0.60 -5.39e-02  ...     0.37 -0.37   1.00 -0.74
MEDV    -0.39  0.36 [...]<jupyter_text>Interesting part is many of the attributes have a strong correlation(e.g >0.70 or <-0.70)1. Examples* **NOX** and **INDUS** with 0.76* **DIS** and **INDUS** with -0.71* **TAX** and **INDUS** with 0.72 * **AGE** and **NOX** with 0.73* **DIS** and **NOX** with -0.78  Data Visualizations<jupyter_code># Unimodal Data Visualizations

# histograms
dataset.hist(sharex=False,sharey=False,xlabelsize=1,ylabelsize=1)
pyplot.show()
# density
dataset.plot(kind='density',subplots=True,layout=(4,4),sharex=False,legend=False,fontsize=1)
pyplot.show()
# box and whisker plots
dataset.plot(kind='box',subplots=True,layout=(4,4),sharex=False,sharey=False,fontsize=8)
pyplot.show()
## Multimodal Data Visualizations

# scatter plot matrix
scatter_matrix(dataset)
pyplot.show()
## correlation matrix
fig = pyplot.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(dataset.corr(),vmin=-1,vmax=1,interpolation='none')
fig.colorbar(cax)
ticks = numpy.arange(0,14,1)
ax.set_xticks(ticks)
ax.set_yticks(ticks)
ax.set_xticklabels(names)
ax.set_yticklabels(names)
pyplot.show()<jupyter_output><empty_output><jupyter_text>Validation Dataset<jupyter_code>## split-out validation dataset

array = dataset.values
X = array[:,0:13]
Y = array[:,13]
validation_size = 0.20
seed = 7
X_train,X_validation,Y_train,Y_validation = train_test_split(X,Y,test_size=validation_size,random_state=seed)<jupyter_output><empty_output><jupyter_text>Evaluate Algorithms<jupyter_code>## Test options and evaluation metric
num_folds = 10
seed = 7
scoring = 'neg_mean_squared_error'
# Spot-check algorithms
models = []
models.append(('LR',LinearRegression()))
models.append(('LASSO',Lasso()))
models.append(('EN',ElasticNet()))
models.append(('KNN',KNeighborsRegressor()))
models.append(('CART',DecisionTreeRegressor()))
models.append(('SVR',SVR()))
# evaluate each model in turn
results = []
names = []
for name,model in models:
    kfold = KFold(n_splits=num_folds,shuffle=True,random_state=seed)
    cv_results = cross_val_score(model,X_train,Y_train,cv=kfold,scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name,cv_results.mean(),cv_results.std())
    print(msg)
## comparing algorithms
fig = pyplot.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
pyplot.boxplot(results)
ax.set_xticklabels(names)
pyplot.show()<jupyter_output><empty_output><jupyter_text>Evaluate Algorithms  : Standardization<jupyter_code># standardize the dataset
pipelines = []
pipelines.append(('ScaledLR',Pipeline([('Scaler',StandardScaler()),('LR',LinearRegression())])))
pipelines.append(('ScaledLASSO',Pipeline([('Scaler',StandardScaler()),('LASSO',Lasso())])))
pipelines.append(('ScaledEN',Pipeline([('Scaler',StandardScaler()),('EN',ElasticNet())])))
pipelines.append(('ScaledKNN',Pipeline([('Scaler',StandardScaler()),('KNN',KNeighborsRegressor())])))
pipelines.append(('ScaledCART',Pipeline([('Scaler',StandardScaler()),('CART',DecisionTreeRegressor())])))
pipelines.append(('ScaledSVR',Pipeline([('Scaler',StandardScaler()),('SVR',SVR())])))

results = []
names = []
for name,model in pipelines:
    kfold = KFold(n_splits=num_folds,shuffle=True,random_state=seed)
    cv_results = cross_val_score(model,X_train,Y_train,cv=kfold,scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" % (name,cv_results.mean(),cv_results.std())
    print(msg)



# compare algorithms 
fig = pyplot.figure()
fig.suptitle('Scaled Algorithm Comparison')
ax = fig.add_subplot(111)
pyplot.boxplot(results)
ax.set_xticklabels(names)
pyplot.show()<jupyter_output><empty_output><jupyter_text>Improve results with tuning* We see that KNN achieves good results on a scaled version of the dataset.* But it can do better,the default value for the number of neighbors in KNN is 7.* we can use a grid search to try a set of different numbers of neighbors and see if we can improve the score.<jupyter_code>## KNN Algorithm tuning
scaler = StandardScaler().fit(X_train)
rescaledX = scaler.transform(X_train)
k_values = numpy.array([1,3,5,7,9,11,13,15,17,19,21])
param_grid = dict(n_neighbors=k_values)
model = KNeighborsRegressor()
kfold = KFold(n_splits=num_folds,shuffle=True,random_state=seed)
grid = GridSearchCV(estimator=model,param_grid=param_grid,scoring=scoring,cv=kfold)
grid_result = grid.fit(rescaledX,Y_train)
print("Best: %f using %s" %(grid_result.best_score_,grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

for mean,stdev,param in zip(means,stds,params):
    print("%f (%f) with: %r" %(mean,stdev,param))<jupyter_output>Best: -19.497829 using {'n_neighbors': 1}
-19.497829 (15.769847) with: {'n_neighbors': 1}
-19.977984 (13.803973) with: {'n_neighbors': 3}
-21.270967 (14.833544) with: {'n_neighbors': 5}
-21.577292 (14.952592) with: {'n_neighbors': 7}
-21.001075 (14.701297) with: {'n_neighbors': 9}
-21.490306 (14.866957) with: {'n_neighbors': 11}
-21.268533 (14.454969) with: {'n_neighbors': 13}
-21.968092 (14.209894) with: {'n_neighbors': 15}
-22.739880 (14.492752) with: {'n_neighbors': 17}
-23.506901 (14.903224) with: {'n_neighbors': 19}
-24.240303 (15.156565) with: {'n_neighbors': 21}<jupyter_text>Ensemble Methods* **Boosting Methods**: AdaBoost(AB) and Gradient Boosting(GBM)* **Bagging Methods**: Random Forests(RF) and Extra Trees(ET)<jupyter_code># ensembles 
ensembles = []
ensembles.append(('ScaledAB',Pipeline([('Scaler',StandardScaler()),('AB',AdaBoostRegressor())])))
ensembles.append(('ScaledGBM',Pipeline([('Scaler',StandardScaler()),('GBM',GradientBoostingRegressor())])))
ensembles.append(('ScaledRF',Pipeline([('Scaler',StandardScaler()),('RF',RandomForestRegressor())])))
ensembles.append(('ScaledET',Pipeline([('Scaler',StandardScaler()),('ET',ExtraTreesRegressor())])))
results = []
names = []
for name,model in ensembles:
    kfold = KFold(n_splits=num_folds,shuffle=True,random_state=seed)
    cv_results = cross_val_score(model,X_train,Y_train,cv=kfold,scoring=scoring)
    results.append(cv_results)
    names.append(name)
    msg = "%s: %f (%f)" %(name,cv_results.mean(),cv_results.std())
    print(msg)


## compare algorithms
fig = pyplot.figure()
fig.suptitle('Scaled Ensemble Algorithm Comparison')
ax = fig.add_subplot(111)
pyplot.boxplot(results)
ax.set_xticklabels(names)
pyplot.show()<jupyter_output><empty_output><jupyter_text>Tune Ensemble Methods* The default number of boosting stages to perform (n_estimators) is 100.* Often the larger the number of boosting stages,the better the performance but the longer the training time.<jupyter_code>## Tune Scaled GBM
scaler = StandardScaler().fit(X_train)
rescaledX = scaler.transform(X_train)
param_grid = dict(n_estimators=numpy.array([50,100,150,200,250,300,350,400]))
model = GradientBoostingRegressor(random_state=seed)
kfold = KFold(n_splits=num_folds,shuffle=True,random_state=seed)
grid = GridSearchCV(estimator=model,param_grid=param_grid,scoring=scoring,cv=kfold)
grid_result = grid.fit(rescaledX,Y_train)

print("Best: %f using %s" %(grid_result.best_score_,grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

for mean,stdev,param in zip(means,stds,params):
    print("%f (%f) with: %r" %(mean,stdev,param))<jupyter_output>Best: -10.560638 using {'n_estimators': 400}
-11.629999 (8.833221) with: {'n_estimators': 50}
-11.073455 (8.772155) with: {'n_estimators': 100}
-10.838924 (8.734377) with: {'n_estimators': 150}
-10.785568 (8.657244) with: {'n_estimators': 200}
-10.654778 (8.540552) with: {'n_estimators': 250}
-10.649385 (8.497232) with: {'n_estimators': 300}
-10.589337 (8.413596) with: {'n_estimators': 350}
-10.560638 (8.353747) with: {'n_estimators': 400}<jupyter_text>Finalize model<jupyter_code># prepare the model
scaler = StandardScaler().fit(X_train)
rescaledX = scaler.transform(X_train)
model = GradientBoostingRegressor(random_state=seed,n_estimators=400)
model.fit(rescaledX,Y_train)

# transform the validation dataset
rescaledValidationX = scaler.transform(X_validation)
predictions = model.predict(rescaledValidationX)
print(mean_squared_error(Y_validation,predictions))
# close to our estimate of -9.3<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1503362560653784, 'fraction_numerical': 0.072954796969439, 'mean_word_length': 5.6978335233751425, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '474868', 'license': 'MIT', 'n_tokens_mistral': 4926, 'n_tokens_neox': 4384, 'n_words': 897, 'path': 'Boston House Price Prediction.ipynb', 'repo_name': 'kuluruvineeth/Boston-House-Price-Prediction', 'size': '11747'}