<jupyter_start><jupyter_text>Machine Learning and Statistics Project - Boston House Price Dataset![Numpy](https://cdn-images-1.medium.com/max/1200/1*L0yNfRdiH8DfYuiJTKPxww.png) The Boston Housing dataset contains US census data published in 1978 of houses in various parts of Boston. The dataset contains information on 505 samples with 12 measurements for each sample. The samples are contained in the rows of the dataset and the measurements in the columns. The following is analysis of the dataset using descriptive and inferential statistics:<jupyter_code>import numpy as np  # import numnerical python with shorthand  
import matplotlib.pyplot as plt  # shorthand import for matplotlib library
import seaborn as sns # import seaborn for advanced graphics
import pandas as pd # import pandas for data structures and operations for manipulating numerical tables and time series
import sklearn # for Machine Learning in Python
import scipy.stats as stats # statisitcal function
import statsmodels.api as sm # conducting statistical tests, and statistical data exploration
%matplotlib inline 
from matplotlib import rcParams # improved plots from matplotlib
from sklearn.model_selection import train_test_split #Split arrays or matrices into random train and test subsets
from sklearn import svm # support vector classification
import sklearn.preprocessing as pre # For encoding categorical variables.
import sklearn.model_selection as mod # For splitting into training and test sets.
import sklearn.decomposition as dec
# make the plots bigger
plt.rcParams['figure.figsize'] = (20.0, 10.0)<jupyter_output><empty_output><jupyter_text>The Boston House Price Dataset is loaded into the jupyter notebook from the scikit learn library<jupyter_code># Load the Boston House Price dataset from scikit-learn library
from sklearn.datasets import load_boston
boston_dataset = load_boston()
#Print the values of the Boston dataset
print(boston_dataset.keys())<jupyter_output>dict_keys(['data', 'target', 'feature_names', 'DESCR', 'filename'])<jupyter_text>data: contains the information for various housestarget: prices of the housefeature_names: names of the featuresDESCR: describes the dataset Describe the shape of the dataset rows, columns<jupyter_code># Give the description of the dataset
print(boston_dataset.data.shape)
print(boston_dataset.target.shape)<jupyter_output>(506,)<jupyter_text>Print the first five rows of the dataset<jupyter_code>#Print the header of the Boston dataset
boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
boston.head()<jupyter_output><empty_output><jupyter_text>Give a description of what is contained in each column<jupyter_code>print(boston_dataset.DESCR)<jupyter_output>.. _boston_dataset:

Boston house prices dataset
---------------------------

**Data Set Characteristics:**  

    :Number of Instances: 506 

    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.

    :Attribute Information (in order):
        - CRIM     per capita crime rate by town
        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
        - INDUS    proportion of non-retail business acres per town
        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
        - NOX      nitric oxides concentration (parts per 10 million)
        - RM       average number of rooms per dwelling
        - AGE      proportion of owner-occupied units built prior to 1940
        - DIS      weighted distances to five Boston employment centres
        - RAD      index of accessibility to radial highways
        - TAX      full-value property-tax rate per $10,000
        - PTRATIO  pu[...]<jupyter_text>Append the MEDV value from target to the dataset<jupyter_code>boston['MEDV'] = boston_dataset.target
# Add MEDV value to the dataset
print(boston.head())
# Dataframe with price<jupyter_output>CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \
0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   
1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   
2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   
3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   
4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   

   PTRATIO       B  LSTAT  MEDV  
0     15.3  396.90   4.98  24.0  
1     17.8  396.90   9.14  21.6  
2     17.8  392.83   4.03  34.7  
3     18.7  394.63   2.94  33.4  
4     18.7  396.90   5.33  36.2<jupyter_text>Descriptive StatisticsCalculate the minimum, maximum, mean, median, and standard deviation of 'MEDV', which is stored in target. Median value of owner-occupied homes in $1000's<jupyter_code>minimum_price = np.min(boston.MEDV) # Minimum price of the data
maximum_price = np.max(boston.MEDV) # maximum price of the data
mean_price = np.mean(boston.MEDV) # mean price of the data
median_price = np.median(boston.MEDV) # median price of the data
std_price = np.std(boston.MEDV) # standard deviation of the house prices


# Show the calculated statistics
print ("The lowest house price is:", minimum_price)
print ("The highest house price is:", maximum_price)
print ("The average house price is:", mean_price)
print ("The median house price is:", median_price)
print ("The standard deviation of houses prices is:", std_price)
boston.describe().T<jupyter_output><empty_output><jupyter_text>Histograms of variables Plot histograms of the variables within the dataset as a visual representation<jupyter_code>boston.hist(bins=50, figsize=(20,15))
plt.show()<jupyter_output><empty_output><jupyter_text>Plot the house prices to show distribution of prices<jupyter_code># Histogram of house prices in 1000's
plt.hist(boston.MEDV)
plt.show<jupyter_output><empty_output><jupyter_text>Distribution plot of house prices<jupyter_code>sns.set(rc={'figure.figsize':(12,8)})
sns.distplot(boston['MEDV'], bins=10)
plt.show()<jupyter_output><empty_output><jupyter_text>Correlation HeatmapHeatmap showing the correlation matrix meausing the linear relationships between the variables of the dataset<jupyter_code>correlation_matrix = boston.corr().round(2)
# annot = True to print the values inside the square
sns.heatmap(data=correlation_matrix, annot=True)<jupyter_output><empty_output><jupyter_text>ObservationsThe correlation coefficient runs on a scale of -1 to 1, 1 having stong correlation and -1 showing no correlation. MEDV shows a strong cprrelation to the RM value indicating a higher house price for a dwelling with a larger number of rooms. It shows negative correlation to LSTAT, lower status withing the population.  Linear Regression LSAT - Percentage of lower status of the populationLSAT was selected for the linear regression module as it showed the stongest negative correlation with MEDV (Median value of owner-occupied homes in $1000s) in the correlation matrix. It shows that the house prices are negatively correlated linearly with the status within the population.More "lower class" areas have lower prices.<jupyter_code>plt.figure(figsize=(20, 5))

features = ['LSTAT']
target = boston['MEDV']

for i, col in enumerate(features):
    plt.subplot(1, len(features) , i+1)
    x = boston[col]
    y = target
    plt.scatter(x, y, marker='o')
    plt.title(col)
    plt.xlabel(col)
    plt.ylabel('MEDV')<jupyter_output><empty_output><jupyter_text>Positive Correlation  RM - Average number of rooms per dwellingThere is strong positive correlation with with MEDV (Median value of owner-occupied homes in $1000s) in the correlation matrix. There is a higher MEDV for a higher RM as more rooms would imply more space, thereby costing more, taking all other factors constant.<jupyter_code>plt.figure(figsize=(20, 5))

features = ['RM']
target = boston['MEDV']

for i, col in enumerate(features):
    plt.subplot(1, len(features) , i+1)
    x = boston[col]
    y = target
    plt.scatter(x, y, marker='o')
    plt.title(col)
    plt.xlabel(col)
    plt.ylabel('MEDV')<jupyter_output><empty_output><jupyter_text>Pupil-teacher ratio by townThere is a lower price in areas where there is a high student-to-teacher ratio due to a lower demand for houses in such areas. A high student teacher ratio results in less attention dedicated to each student and potentially impact their performance.<jupyter_code>plt.figure(figsize=(20, 5))

features = ['PTRATIO']
target = boston['MEDV']

for i, col in enumerate(features):
    plt.subplot(1, len(features) , i+1)
    x = boston[col]
    y = target
    plt.scatter(x, y, marker='o')
    plt.title(col)
    plt.xlabel(col)
    plt.ylabel('MEDV')<jupyter_output><empty_output><jupyter_text>=======================================================================================================================  Charles River AnalysisThe following section uses inferential statistics are used to analyse the difference in median house prices along the Charles river and houses which are not. The following tools have been used in the analysis:- Boxplots- Histogram- Anova - Paired T TestThe CHAS variable in the dataset is the Charles River dummy variable. The variable is equal to one is the house is bound to the Charles River and zero if it is not. This section analyzes the data based on the CHAS value and its impact on the respective MEDV value.<jupyter_code># Split the Chas data into along the river and not along the river
Along_The_River=boston.loc[boston['CHAS'] != 0]
Not_Along_The_River=boston.loc[boston['CHAS'] != 1]
chas = boston.CHAS
chas<jupyter_output><empty_output><jupyter_text>HistogramDisplay a histogram to show how many houses in the dataset track the Charles River and how many do not.<jupyter_code>plt.hist(boston.CHAS)
plt.show<jupyter_output><empty_output><jupyter_text>The histogram dispalys the number of houses houses which do not tract the river (0) and the number of house which do tract the river (1). A significantly higher number of houses do not tract the river. For futher analysis the median price for each category will be compared.  BoxplotsVisual comparison of the median values for the houses which tract the river verus those that do not.<jupyter_code>boston.boxplot(column='MEDV', by='CHAS')
plt.show()<jupyter_output><empty_output><jupyter_text>The boxplot shows the median price for the houses which tract the river was higher in Boston at the time the dataset was generated.  AnovaAnalysis of variance (ANOVA) is a collection of statistical models and their associated estimation procedures (such as the "variation" among and between groups) used to analyze the differences among group means in a sample. (https://en.wikipedia.org/wiki/Analysis_of_variance)<jupyter_code>atrm = np.mean(Along_The_River)                  # mean of Along_The_River
atrs = np.std(Along_The_River)                   # std of Along_The_River

natrm = np.mean(Not_Along_The_River)             # mean of Not_Along_The_River
natrs = np.std(Not_Along_The_River)              # std of Not_Along_The_River

print (atrm['MEDV'])
print (atrs['MEDV'])
print (natrs['MEDV'])
print (natrs['MEDV'])
# x values for plotting.
x = np.linspace(1.25, 2.25, 1000)

# The probability density functions (PDFs) for the two populations.
pdfA = stats.norm.pdf(x, atrm['MEDV'], atrs['MEDV'])
pdfB = stats.norm.pdf(x, natrs['MEDV'], natrs['MEDV'])

# Plot the population PDFs as shaded regions.
plt.fill_between(x, pdfA, color='g', alpha=0.25, label="PDF Along_The_River") # Fill the area between two horizontal curves.
plt.fill_between(x, pdfB, color='b', alpha=0.25, label="PDF Not_Along_The_River") # Fill the area between two horizontal curves.

# Plot histograms of the two samples.
plt.hist(Along_The_River['MEDV'], density=True, color='g', alpha=0.25, label="Along_The_River")
plt.hist(Not_Along_The_River['MEDV'], density=True, color='b', alpha=0.25, label="Not_Along_The_River")

# Display a legend.
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>The probability density function (PDF) is calculated to find probabilities associated with a continuous random variable. Therefore it is not an analytical tool which can be applied to the Charles River Data.  Paired T Testhttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.htmlWhen performing an independent sample t-test we assume that there is a given difference between the means of two populations, usually a difference of zero.The t-statistic looks at the two samples to see how different they are. The hypothesised difference (usually zero) is a indication of the probability of seeing a t-statistic at least this extreme.If the t-statistic is too extreme then we cannot accept the null hypothesis. If there is a large p-value, (> 0.05 or 0.1), then we accept the null hypothesis. If the p-value is smaller than the threshold, e.g. 1%, 5% or 10%, then we reject the null hypothesis of equal averages.![Null](NullHypothesis.png) T Test of median prices along the river and those that are not along the river:<jupyter_code>import scipy.stats as ss
ss.ttest_ind(Along_The_River['MEDV'], Not_Along_The_River['MEDV'])<jupyter_output><empty_output><jupyter_text>As the p value is 0.0000739 the null hypothesis is rejected.  T test of median prices and if they bound the Charles River<jupyter_code>ss.ttest_ind(boston['MEDV'], boston['CHAS'])<jupyter_output><empty_output><jupyter_text>Houses along the Charles River have a higher median price than those that are not along the river. =================================================================================================================  Neural Network![neural network](https://upload.wikimedia.org/wikipedia/commons/thumb/f/fe/Kernel_Machine.svg/220px-Kernel_Machine.svg.png)  Train the datasetTraining Dataset: The sample of data used to fit the model.The actual dataset that we use to train the model (weights and biases in the case of Neural Network). The model sees and learns from this data. (https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7)<jupyter_code>from sklearn.linear_model import LinearRegression # used to train the model on both the training and test sets.<jupyter_output><empty_output><jupyter_text>Prepare the data for training the model<jupyter_code>X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
Y = boston['MEDV']<jupyter_output><empty_output><jupyter_text>Split the data into training and testing sets. 80% of the samples are used to train the model and the remaining 20% are used to test<jupyter_code>from sklearn.model_selection import train_test_split # used to split the data

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=5)
print(X_train.shape)
print(X_test.shape)
print(Y_train.shape)
print(Y_test.shape)<jupyter_output>(404, 2)
(102, 2)
(404,)
(102,)<jupyter_text>Scikit-learn’s LinearRegression is used to train the model on both the training and test sets.<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

lin_model = LinearRegression()
lin_model.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>Model EvaluationThe model is evaluated using RMSE and R2-score.<jupyter_code>from sklearn.metrics import r2_score
# model evaluation for training set
y_train_predict = lin_model.predict(X_train)
rmse = (np.sqrt(mean_squared_error(Y_train, y_train_predict)))
r2 = r2_score(Y_train, y_train_predict)

print("The model performance for training set")
print("--------------------------------------")
print('RMSE is {}'.format(rmse))
print('R2 score is {}'.format(r2))
print("\n")

# model evaluation for testing set
y_test_predict = lin_model.predict(X_test)
rmse = (np.sqrt(mean_squared_error(Y_test, y_test_predict)))
r2 = r2_score(Y_test, y_test_predict)

print("The model performance for testing set")
print("--------------------------------------")
print('RMSE is {}'.format(rmse))
print('R2 score is {}'.format(r2))<jupyter_output>The model performance for training set
--------------------------------------
RMSE is 5.637129335071195
R2 score is 0.6300745149331701


The model performance for testing set
--------------------------------------
RMSE is 5.137400784702912
R2 score is 0.6628996975186952<jupyter_text>Scatter plot Scatter plot to visualize the difference between actual vs predicted valuesusing the X_test values<jupyter_code>lm = LinearRegression()
lm.fit(X_train, Y_train)

Y_pred = lm.predict(X_test)

plt.scatter(Y_test, Y_pred)
plt.xlabel("Actual Price: $Y_i$")
plt.ylabel("Predicted price: $\hat{Y}_i$")
plt.title("Actual price vs Predicted prices: $Y_i$ vs $\hat{Y}_i$")<jupyter_output><empty_output><jupyter_text>![Keras](https://miro.medium.com/max/1000/1*LkKz4wtZNBo5i-Vc8DWhTA.png)Create a neural network to predict the median house price based on the other variables in the dataset. The features, 'RM', 'LSTAT', and 'PTRATIO', give us quantitative information about each data point. The target variable, 'MEDV', will be the variable we seek to predict.  ![NeuralNetwork](https://upload.wikimedia.org/wikipedia/commons/thumb/4/46/Colored_neural_network.svg/280px-Colored_neural_network.svg.png)An artificial neural network is an interconnected group of nodes, inspired by a simplification of neurons in a brain. Here, each circular node represents an artificial neuron and an arrow represents a connection from the output of one artificial neuron to the input of another.(https://en.wikipedia.org/wiki/Artificial_neural_network)<jupyter_code># Inputs from the dataset
inputs = pd.DataFrame(boston, columns=['RM','LSTAT','PTRATIO'])
inputs<jupyter_output><empty_output><jupyter_text>Pre-processing refers to the transformations applied to our data before feeding it to the algorithm. Data Preprocessing is a technique that is used to convert the raw data into a clean data set. In other words, whenever the data is gathered from different sources it is collected in raw format which is not feasible for the analysis. (https://www.geeksforgeeks.org/data-preprocessing-machine-learning-python/)<jupyter_code># https://scikit-learn.org/stable/modules/preprocessing.html preprocessing
xscale = pd.DataFrame(pre.scale(inputs), columns=inputs.columns)
xscale
output = boston['MEDV'] #seek to output a house price value
output<jupyter_output><empty_output><jupyter_text>Build The ModelUsing keras to create a neural network that can predict the median house price based on the other variables in the dataset.To build the model I will use the following subset of the dataset:        - RM       average number of rooms per dwelling        - PTRATIO  pupil-teacher ratio by town        - LSTAT    % lower status of the population        - MEDV     Median value of owner-occupied homes in $1000's<jupyter_code>from __future__ import absolute_import, division, print_function, unicode_literals

import tensorflow as tf

from tensorflow import keras as kr
# Create a new neural network.
m = kr.models.Sequential()
# Add neurons.
# m.add(kr.layers.Dense(1, input_dim=1, activation="linear"))

# Add a hidden layer with x neurons and an input layer with 3.
m.add(kr.layers.Dense(units=10, input_dim=3, activation="linear"))
m.add(kr.layers.Dense(10, activation="linear"))
m.add(kr.layers.Dense(1, activation='linear'))
# Compile the model.
m.compile(loss="mean_squared_error", optimizer="adam")<jupyter_output><empty_output><jupyter_text>The number of epochs is a hyperparameter that defines the number times that the learning algorithm will work through the entire training dataset. (https://machinelearningmastery.com/difference-between-a-batch-and-an-epoch/)  Split<jupyter_code># Split the inputs and outputs into training and test sets.
inputs_train, inputs_test, output_train, output_test = mod.train_test_split(inputs, output, test_size=0.2)
inputs_test.iloc[0]
m.predict(inputs_test.as_matrix()[0:1])<jupyter_output>C:\Users\Frank\Anaconda4\lib\site-packages\ipykernel_launcher.py:1: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  """Entry point for launching an IPython kernel.<jupyter_text>Train<jupyter_code># Train the neural network.
m.fit(inputs_train, output_train, epochs=500, batch_size=20)
# Train the model.
# m.fit(inputs, output, epochs=15, batch_size=128)
# https://stackoverflow.com/questions/37232782/nan-loss-when-training-regression-network<jupyter_output><empty_output><jupyter_text>Predict<jupyter_code>p = m.predict(inputs_test.as_matrix())
p
#Plot of actual v predicted price of first 100 data points
plt.figure(figsize=(12, 12))
plt.plot(p[:100], 'r-', label ='Predicted MEDV')
plt.plot(output[:100], 'b', label = 'Actual MEDV')
plt.title("Predicted Prices vs Actual Prices")
plt.ylabel("Price(1000's)")
plt.xlabel("Zone")
plt.legend(["Prediction", "Actual"], loc="upper left")
plt.show()<jupyter_output><empty_output><jupyter_text>Preprocessing the DataPut data into format for data analysis help the nural network make better predictions. Principal Components Analysis (PCA) is a dimensionality reduction algorithm that can be used to significantly speed up your unsupervised feature learning algorithm. More importantly, understanding PCA will enable us to later implement whitening, which is an important pre-processing step for many algorithms.(http://ufldl.stanford.edu/tutorial/unsupervised/PCAWhitening/).  WhiteningWhitening of data is a way to preprocess the data. The idea behind whitening is to remove the underlying correlation in the data. It is a process done usually after the data is projected onto the eigen vectors as a result of PCA.(https://www.quora.com/What-is-whitening-of-data-in-Neural-Networks)<jupyter_code># https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing
xscale = pd.DataFrame(pre.scale(inputs), columns=inputs.columns)
xscale
scaler = pre.StandardScaler()
scaler.fit(inputs)
scaler.mean_, scaler.scale_
xscale = pd.DataFrame(scaler.transform(inputs), columns=inputs.columns)
xscale
inputs.corr()
pca = dec.PCA(n_components=3, whiten=True)
pca.fit(inputs)
inputs_white = pd.DataFrame(pca.transform(inputs), columns=inputs.columns)
inputs_white
inputs_white.corr().round()
inputs_white.mean().round()
inputs_white.std().round()
# Add neurons.
# m.add(kr.layers.Dense(1, input_dim=1, activation="linear"))

# Add a hidden layer with x neurons and an input layer with 3.
m.add(kr.layers.Dense(units=10, input_dim=3, activation="linear"))
m.add(kr.layers.Dense(10, activation="linear"))
m.add(kr.layers.Dense(1, activation='linear'))
# Compile the model.
m.compile(loss="mean_squared_error", optimizer="adam")
# Train the neural network.
m.fit(inputs_train_white, output_train, epochs=500, batch_size=20)
w = m.predict(inputs_white.as_matrix())
w
#Plot of actual v whitened predicted price of first 100 data points
plt.figure(figsize=(12, 12))
plt.plot(w[:100], 'r-', label ='WhitePredicted MEDV')
plt.plot(output[:100], 'b', label = 'Actual MEDV')
plt.title("White Predicted Prices vs Actual Prices")
plt.ylabel("Price(1000's)")
plt.xlabel("Zone")
plt.legend(["WhitePrediction", "Actual"], loc="upper left")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10802415875754962, 'fraction_numerical': 0.026143226919758414, 'mean_word_length': 4.991470664254329, 'pattern_counts': {'":': 0, '<': 126, '<?xml version=': 0, '>': 127, 'https://': 14, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '228637', 'license': 'Apache-2.0', 'n_tokens_mistral': 7541, 'n_tokens_neox': 6966, 'n_words': 2406, 'path': 'Machine_Learning.ipynb', 'repo_name': 'GraceAKelly/Machine_learning', 'size': '23180'}