<jupyter_start><jupyter_text>Exercise 1: Linear RegressionIn this exercise, we will implement linear regression and get to see it work on data.In our example, we will build a model for the profitability of a food truck in a city. Our sample data contains the data set for our regression problem. The first column in the data set is the population of a city, the second column contains the profit of a food truck in that city. A negative value for profit indicates a loss.<jupyter_code># Import python modules
from six.moves import range
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# This needs to be set after importing matplotlib.pyplot, but in a different cell
plt.rcParams['figure.figsize'] = (10, 6.8)<jupyter_output><empty_output><jupyter_text>Examining the data<jupyter_code>data = np.genfromtxt('data/ex1data1.txt', delimiter=',')<jupyter_output><empty_output><jupyter_text>For example, the first row of our data looks like the following.<jupyter_code>data[0]<jupyter_output><empty_output><jupyter_text>We assume that our data is scaled by 10000. Therefore, our first city has a population of 61101 and the food truck's annual profit is 175920$.Let's visualize our data.<jupyter_code>X = data[0:, 0]
y = data[0:, 1]
m = y.size
# plt.figure(figsize=(10,6))
plt.plot(X, y,'rx',markersize=10)
# plt.grid(True) #Always plot.grid true!
plt.ylabel('Profit in $10,000s')
plt.xlabel('Population of City in 10,000s');<jupyter_output><empty_output><jupyter_text>Gradient descent algorithmLet's program our gradient descent algorithm. First we initilaize some variables.<jupyter_code># Add a column of ones to X, X is then a matrix with m rows and 2 columns
X = np.hstack((np.ones((m, 1)), np.reshape(data[0:, 0], (m,1))))
y = np.reshape(data[0:, 1], (m,1))

# Initialize fitting parameters
theta = np.zeros((2, 1)); 

# Some gradient descent settings
iterations = 1500;
alpha = 0.01;
X.shape<jupyter_output><empty_output><jupyter_text>The objective of our linear regression (implemented using gradient descent) is to fit our parameters $ \theta = (\theta_1, \theta_2) $ in such a way, that a so called cost function $ J $ is minimized. $$ J(\theta) = \frac{1}{2m} \sum_{i=1}^{m}(h_\theta(x^{(i)}) - y^{(i)})^2  $$ with$$ h_\theta(x) = \theta^T x $$Remember: $ X $ is a matrix, whose $i$th row is $ (1, x_(i)) $.Therefore, we can implement our cost function as follows:<jupyter_code>def h(theta, x):
    return np.transpose(np.dot(np.transpose(theta), np.transpose(x)))
def computeCost(X, y, theta):
    
    m = y.size
    
    J = 0;
    # J = 1/(2*m) * (h(theta, X)-y)' * (h(theta, X)-y);
    J = 1.0/(2*m) * np.asscalar(np.dot(np.transpose(h(theta, X) - y), h(theta, X) - y))
    return J<jupyter_output><empty_output><jupyter_text>Let's test our cost function:<jupyter_code>computeCost(X, y, theta)
computeCost(X, y, np.reshape([-1, 2], (2,1)))
computeCost(X, y, [[-1], [2]])<jupyter_output><empty_output><jupyter_text>Now, let's implement the gradient descent algorithm.<jupyter_code>def rowSum(m):
    return (np.sum(m, axis=0)[np.newaxis, :])
    
def gradientDescent(X, y, theta, alpha, num_iters):

    # Initialize some useful values
    m = y.size 
    # Record the development of our cost function for debugging purposes
    J_history = np.zeros(num_iters)

    for iter in range(num_iters):

        # theta = theta - 1/m * alpha * X' * (h(theta, X) - y ) ;
        theta = theta - 1.0/m * alpha * np.dot(np.transpose(X), h(theta, X) - y)
        #print theta

        # Save the cost J in every iteration    
        J_history[iter] = computeCost(X, y, theta)
    
    return (theta, J_history)
(theta, J_history) = gradientDescent(X, y, [[0], [0]], alpha, iterations)
J_history.size<jupyter_output><empty_output><jupyter_text>Hence, our fitting values are:<jupyter_code>theta<jupyter_output><empty_output><jupyter_text>We can now implement our prediction function:<jupyter_code>def predictProfit(theta, population):
    theta0 = theta[0,0]
    theta1 = theta[1,0]
    return theta0 + population*theta1<jupyter_output><empty_output><jupyter_text>Let's evaluate our result: in a city with a population of 170000 people, our food truck should create a profit of:<jupyter_code>predictProfit(theta, 17)<jupyter_output><empty_output><jupyter_text>Some more examples:<jupyter_code>predictProfit(theta, 3.5)
predictProfit(theta, 7)<jupyter_output><empty_output><jupyter_text>Let's visualize our linear regression result together with our data:<jupyter_code>x1 = X[:, 1]
x2 = np.linspace(0, 25)
g = predictProfit(theta, x2)
# plt.figure(figsize=(10,6))
p1 = plt.plot(x1, y, 'rx', markersize=10)
p2 = plt.plot(x2, g, 'b-')
plt.xlabel('Population in 10000s')
plt.ylabel('Profit in 10000$')
plt.show()<jupyter_output><empty_output><jupyter_text>Some ObservationsLet's examine how well our gradient descent algorithm was converging by plotting the history of our cost function during the execution of gradient descent.<jupyter_code>iteration_axis = np.linspace(1, 1500, 1500)
plt.plot(iteration_axis, J_history, '-')
plt.xlabel('Iterations')
plt.ylabel('J')
plt.show;
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

# Grid over which we will calculate J
theta0_vals = np.linspace(-10, 10, 100)
theta1_vals = np.linspace(-1, 4, 100)

# Because of the way meshgrids work in the surface and contour plots, 
# we need to flip the axes, see: 
# https://eli.thegreenplace.net/2014/meshgrids-and-disambiguating-rows-and-columns-from-cartesian-coordinates/
# We can do so by specifiying the indexing during the creation of the meshgrid - or -
# by transposing J_vals after computing it.
(U, V) = np.meshgrid(theta0_vals, theta1_vals, indexing='ij' )

# Compute J_vals grid
J_vals = np.zeros( (theta0_vals.size, theta1_vals.size) )
for i in range(theta0_vals.size):
    for j in range(theta1_vals.size):
        J_vals[i, j] = computeCost(X, y, np.array([[theta0_vals[i]], [theta1_vals[j]]]))

# We created the meshgrids with correct indexing, therfore we do not need to transpose J_vals
# J_vals = np.transpose(J_vals)
        
ax.view_init(30, 240)
surf = ax.plot_surface(U, V, J_vals, cmap=plt.get_cmap('viridis'))
plt.contour(U, V, J_vals, np.logspace(-2, 3, 20))
plt.plot(theta[0,0], theta[1,0], 'rx', markersize=10);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1312460468058191, 'fraction_numerical': 0.029411764705882353, 'mean_word_length': 5.0584291187739465, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '194115', 'license': 'MIT', 'n_tokens_mistral': 2404, 'n_tokens_neox': 2238, 'n_words': 651, 'path': 'exercise1/exercise1-python.ipynb', 'repo_name': 'mabauer/coursera-machine-learning', 'size': '6324'}