<jupyter_start><jupyter_text>Regression with Multiple Features TO DO- Nothing for the moment. **ACKNOWLEDGEMENT****The dataset used in this notebook is from <NAME>'s course on Machine Learning on Coursera.**<jupyter_code># Share functions used in multiple notebooks
%run Shared-Functions.ipynb
# Import our usual libraries
import numpy as np
import pandas as pd
import math
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>The Business Problem: Predicting Housing PricesWhat's the market value of a house? One way to determine the market price is to collect up the prices of houses based on a few characteristics such as size in square feet and number of bedrooms. Then think like a machine and "learn" what the price should be based on this data.The data we have is our familiar dataset of housing prices in Portland, Oregon. Let's refresh our memories of what this dataset looks like.  Load the Data<jupyter_code>import os
# OS-independent way to navigate the file system
# Data directory is one directory up in relation to directory of this notebook
data_dir_root = os.path.normpath(os.getcwd() + os.sep + os.pardir)
# Where the file is
file_url = data_dir_root + os.sep + "Data" + os.sep + "portland-house-prices.txt"
# Load the data into a dataframe
data2 = pd.read_csv(file_url, header=None, names=['Size', 'Bedrooms', 'Price'])
data2.head()
data2.describe()
data2.Size.min()<jupyter_output><empty_output><jupyter_text>We have information about 47 homes in this dataset. Each house is described by two characteristics or "features".  Step 1: Visualize the Data<jupyter_code>data2.plot.scatter(x='Size', y='Price', figsize=(8,5))
data2.plot.scatter(x='Bedrooms', y='Price', figsize=(8,5))
data2.plot.scatter(x='Bedrooms', y='Size', figsize=(8,5))
# 3D scatter plot
from mpl_toolkits.mplot3d import Axes3D
threeD = plt.figure(figsize=(10,8)).gca(projection='3d')
threeD.scatter(data2['Bedrooms'], data2['Size'], data2['Price'])
threeD.set_xlabel('Bedrooms')
threeD.set_ylabel('Size')
threeD.set_zlabel('Price')
plt.show()<jupyter_output><empty_output><jupyter_text>Rescale/Normalize the DataNotice that the size of a house in square feet is about 1,000 times the number of bedrooms it has. Similarly, the price of a house is about 100 times the size of the house. This is quite common in many datasets, but when it happens, the iterative method of gradient descent (which we'll use again) becomes inefficient. This is simply a matter of making the computations efficient. To do so, we'll do something called *feature normalization*.<jupyter_code># Normalize the dataset
data2Norm = (data2 - data2.mean()) / data2.std()
data2Norm.head()
# Let's keep track of the mean and standard deviation of the house sizes, number of bedrooms, and prices in the dataset.
# We'll need these values when we make predictions

# We can get them easily for size, bedrooms and price by using data2.mean()[0], ..., [2]
data2.mean()[0], data2.mean()[1], data2.mean()[2], data2.std()[0], data2.std()[1], data2.std()[2]<jupyter_output><empty_output><jupyter_text>Step 2: Define the Task You Want to Accomplish  Step 2a: Identify the InputsIn this case we have 2 inputs -- the size and number of bedrooms of the house.<jupyter_code># Number of columns in the dataset
cols = data2Norm.shape[1]
# Inputs are our first two columns
X = data2Norm.iloc[:, 0:cols-1]
# Add an initial column of 1s to X to keep the notation simple
# X.insert(0,'x0',1)
X.head()<jupyter_output><empty_output><jupyter_text>Step 2b: Identify the OutputThe output is the house price. Let's set it up as a variable y.<jupyter_code># The output
y = data2Norm.iloc[:, cols-1:cols]
y.head()<jupyter_output><empty_output><jupyter_text>Step 3: Define the Model  Step 3a: Define the FeaturesIn this case our features are exactly the same as our inputs. We have 2 features: the size of a house in square feet and the number of bedrooms a house has. These are the features encoded in the variables $x_{1}$ and $x_{2}$.Later we'll see models where the features and the inputs are not one and the same. In these cases the features are constructed by combining inputs in various ways.  Step 3b: Transform the Inputs Into an OutputWe're going to stick to a simple model -- exactly the same one as before but now with an additional feature.$$y = h_{W}(X) = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}$$It's still linear but now has more than 1 variable, hence the $x_{2}$.  Step 3c: Clarify the Parameters of the Model$w_{0}$, $w_{1}$, and $w_{2}$ are the *parameters* of the model. These parameters can each take on an infinite number of values. In other words they are continuous variables. $w_{0}$ is called the *bias value*.With this model, we know exactly how to transform an input into an output -- that is, once the values of the parameters are given.Let's pick a value of X from the dataset, fix a specific value for $w_{0}$ and $w_{1}$, and see what we get for the value of y.Specifically, let$\begin{bmatrix}w_{0} \\w_{1} \\w_{2}\end{bmatrix} = \begin{bmatrix}-10 \\1 \\0\end{bmatrix}$This means $w_{0}$ is -10, $w_{1}$ is 1, and $w_{2}$ is 0.Let's try out X * W for the first few rows of X.<jupyter_code># X * W for the first 5 rows of X
df_addOnes(X.iloc[0:5]) * np.matrix('-10;1;0')
# Outputs generated by our model for the first 5 inputs with the specific theta values below
W_init = [-1, 1.4, 0.5]
learning_rate = 0.001 # the learning rate
num_iters = 10 # number of iterations
W_opt, final_penalty, running_w, running_penalty = gradientDescent(X, y, W_init, num_iters, learning_rate)
y[0:5]
# Compare these outputs to the actual values of y in the dataset
# Here's the difference between 
(df_addOnes(X.iloc[0:5]) * np.matrix(W_opt))<jupyter_output><empty_output><jupyter_text>That's quite a bit off from the actual values; so we know that the values for $W$ in $W_{opt}$ must be quite far from the optimal values for $W$ -- the values that will minimize the cost of getting it wrong.  Step 4: Define the Penalty for Getting it WrongOur cost function is exactly the same as it was before for the single variable case. The cost of getting it wrong is defined as a function $J(W)$:$$J(W) = \frac{1}{2m} \sum_{i=1}^{m} (h_{w}x^{(i)}) - y^{(i)})^2$$The only difference from what we had before is in $h_{W}(x)$ which as a $w_{2}$ and $x_{2}$ now added as we saw above$$h_{W}(X) = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}$$<jupyter_code># Compute the cost for a given set of W values over the entire dataset
# Get X and y in to matrix form
penalty(X, y, W_init, squaredPenalty)<jupyter_output><empty_output><jupyter_text>We don't know yet if this is high or low -- we'll have to try out a whole bunch of $W$ values. Or better yet, we can use pick an iterative method and implement it.  Step 5: Find the Parameter Values that Minimize the Penalty Once again, the method that will "learn" the optimal values for $W$ is gradient descent. We don't have to do a thing to the function we wrote before for gradient descent. Let's use it to find the minimum cost and the values of $W$ that result in that minimum cost.<jupyter_code># Set hyper-parameters
num_iters = 1000 # number of iterations
learning_rate = 0.001 # the learning rate
learning_rates = [0.001, 0.001, 0.003, 0.01, 0.03, 0.1]
# Run gradient descent and capture the progression of cost values and the ultimate optimal W values
%time W_opt, final_penalty, running_w, running_penalty = gradientDescent(X, y, W_init, num_iters, learning_rate)
# Get the optimal theta values and the last few cost values
W_opt, final_penalty, running_w[-5:], running_penalty[-5:]
gdResults = [gradientDescent(X, y, W_init, num_iters, learning_rates[i]) for i in range(len(learning_rates))]
penalty_list = [gdResults[i][3] for i in range(len(gdResults))]
penalty_list[0]
# How the penalty changes as the number of iterations increase
fig, ax = plt.subplots(figsize=(8,5))
ax.plot(np.arange(num_iters), running_penalty, 'g')
ax.set_xlabel('Number of Iterations')
ax.set_ylabel('Penalty')
ax.set_title('Penalty vs. Iterations Over the Dataset')
# How the cost of the transformation varies with the learning rate
fig, ax = plt.subplots(figsize=(8,5))
[ax.plot(np.arange(num_iters), penalty_list[i], 'g', label=learning_rates[i]) for i in range(len(learning_rates))]
ax.set_xlabel('Iterations')
ax.set_ylabel('Penalty')
ax.legend()
ax.set_title('Penalty vs. Learning Rate (aka what learning is)')<jupyter_output><empty_output><jupyter_text>Step 6: Use the Model and Optimal Parameter Values to Make PredictionsIt looks like a learning rate ($\alpha$ value) greater than 0.003 is good enough to get our iterative gradient descent to plunge down to arrive at the lowest cost value and stay there. Let's make some predictions...What is our prediction for a house that is 5,000 square feet in size with 4 bedrooms? Let's plug these in to our model and use the optimal $\theta$ values we've calculated above.<jupyter_code># Change size and num_bedrooms to make distinct predictions
size = 5000
num_bedrooms = 5

# Remember we've run the model using rescaled house sizes and number of bedrooms
# So we should scale the inputs down and then scale the prediction up when we're done
size_scaled = (size - data2.mean()[0])/data2.std()[0]
beds_scaled = (num_bedrooms - data2.mean()[1])/data2.std()[1]

# This is our model -- we're just using it here to make a calculation
pred_price = (W_opt[0] * 1) + (W_opt[1] * size_scaled) + (W_opt[2] * beds_scaled)

# Now we re-re-scale the price to convert it back to the right order of magnitude
pred_price_real = (pred_price * data2.std()[2]) + data2.mean()[2]

# Format and print the result
print("Predicted Price: ", '${:20,.0f}'.format(math.ceil(pred_price_real)))<jupyter_output>Predicted Price:  $             783,447
{'fraction_non_alphanumeric': 0.10552661150270712, 'fraction_numerical': 0.022065583818571866, 'mean_word_length': 5.058168316831683, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '297483', 'license': 'MIT', 'n_tokens_mistral': 3219, 'n_tokens_neox': 3034, 'n_words': 1299, 'path': 'Notebooks/Regression-With-Multiple-Features.ipynb', 'repo_name': 'baiaooo/MLCourse', 'size': '9792'}