<jupyter_start><jupyter_text>ML Exercise 1 - Linear Regression To help you start with Python and NumPy, there is a great tutorial online,created at Stanford. It can be downloaded as a notebook at https://github.com/kuleshov/cs228-material/tree/master/tutorials/python. Note that this tutorial is written in Python 2.7. PLEASE DO NOT HESITATE to include your remarks/comments (in colors of your choice) in the notebook. That will be considered as a short report.<jupyter_code># Change here using your first and last names
fn1 = "bonnie"
ln1 = "parker"
fn2 = "clyde"
ln2 = "barrow"

filename = "_".join(map(lambda s: s.strip().lower(), 
                        ["tp1", ln1, fn1, "and", ln2, fn2])) + ".ipynb"
print(filename)<jupyter_output>tp1_parker_bonnie_and_barrow_clyde.ipynb<jupyter_text>Part 1: Linear regression with one variable You will implement linear regression with one variable to predict profits for a food truck. Suppose you are the CEO of a restaurant franchise and are considering different cities for opening a new outlet. The chain already has trucks in various cities and you have data for profits and populations from the cities.  PLEASE READ CAREFULLY this. When you open a notebook (ipython) with google colab, you create a new working session. If you want that your "created session" can see your data in your google drive, you need to "mount" the drive and include the correct path to your data. Do not worry, the following code will do that. When call drive.mount, you need to authorize: click the link, copy the authorization code,...You can run some basic commands like "pwd, ls" using "!pwd, or !ls"<jupyter_code>###### METHOD 1: 

from google.colab import drive

# This will prompt for authorization.
drive.mount('/content/drive')

import os
os.chdir('/content/drive/My Drive/Option_AI_2nd/ML')

# TODO: change the correct path, you may need to create a "data" folder in your drive
!wget https://www.dropbox.com/s/us61lvxcjnn1n3j/ex1data1.txt?dl=0 \
   -O /content/drive/My Drive/Option_AI_2nd/ML/data/ex1data1.txt
###### METHOD 2: EASIER
!wget https://www.dropbox.com/s/us61lvxcjnn1n3j/ex1data1.txt?dl=0 \
   -O ex1data1.txt
# Check your current folder if method 1
!pwd<jupyter_output>/content/drive/My Drive/Option_AI_2nd/ML<jupyter_text>Importe some libraries and examine the data.<jupyter_code>##### If you use METHOD 1
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import os
path = os.getcwd() + 'ex1data1.txt' # TO DO: CHANGE THE PATH
data = pd.read_csv(path, header=None, names=['Population', 'Profit'])
data.head()
##### If you use METHOD 2
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

data = pd.read_csv('ex1data1.txt', header=None, names=['Population', 'Profit'])
data.head()
data.describe()<jupyter_output><empty_output><jupyter_text>Plot it to get a better idea of what the data looks like.<jupyter_code>data.plot(kind='scatter', x='Population', y='Profit', figsize=(15,10))

#data.T.plot.scatter(x = 'Population', y = 'Profit')<jupyter_output><empty_output><jupyter_text>Goal: fit the linear regression parameters $\mathbf{w}$ to your dataset using gradient descent.  You suppose that your relation between your data $x$ and target $t$ can be modeled by a linear mode $y(x,\mathbf{w})= w_0+w_1x$ \\The objective of linear regression is to minimize the cost function:$E(\mathbf{w})=\displaystyle \frac{1}{2N} \sum_{n=1}^{N} \{y(x_n,\mathbf{w})-t_n\}^2 $ \\$N$ is the number of samples in the data set. Question: in this dataset, $N=?$  Your model parameters are the $w_j$ values which you will adjust to minimize cost $E(\mathbf{w})$. One solution is the batch gradient descent algorithm. In batch gradient descent,each iteration performs the update:$\mathbf{w}=\displaystyle \mathbf{w} - \eta \frac{1}{N} \sum_{n=1}^{N} \{y(x_n,\mathbf{w})-t_n\}x_n$With each step of gradient descent, your parameters $w_j$ come closer to the optimal values that will achieve the lowest cost $E(\mathbf{w})$. Question: how many parameters $w$ do you have to find for this dataset.   TODO: Create a function to compute the cost of a given solution (characterized by the parameters w).  Functions like power, sum of numpy may be useful. Important notes for implementation in NumpyFor array, * means element-wise multiplication, and the dot() function is used for matrix multiplication.For matrix, * means matrix multiplication, and the multiply() function is used for element-wise multiplication. Maybe in this part (Linear Regression), you should learn to use np.array. In part 2 (Logistic Regression), you learn to use np.matrix<jupyter_code>def computeCost(X, t, w):
    #X : input dataset of shape (N x D+1), N: number of examples, D: the number of features. 
    
    #t : target of shape (N, ).
    
    #w : parameters of shape (D+1, ).
    
    # Note that if you use np.array, * is elementwise multiplication, unlike Matlab,
    # you may need to use it 
    # then using np.sum...

    # you can use np.dot
    # WRITE YOU CODE HERE

    # In this dataset, X is Population (each data point has only one variable, one feature)
    # t is profit 
    # but for coding this function, you should consider that X is a np.array... 
    # (array in numpy may have more than one dimension) 
    ...
    return ...<jupyter_output><empty_output><jupyter_text>Do some data preparation.TODO: get X (training data) (first column) and t (target variable) (last column) from "data".<jupyter_code># set X (training data) and t (target variable)
cols = data.shape[1]
X = data.iloc[:,0:1]
t = data.iloc....   ##### COMPLETE YOUR CODE HERE#####<jupyter_output><empty_output><jupyter_text>Take a look to make sure X (training set) and t (target variable) look correct.<jupyter_code>X.head()
t.head()<jupyter_output><empty_output><jupyter_text>Let's add a column of ones to the X so we can use a vectorized solution to computing the cost and gradients (use X.insert).Question: why do you need to do this? (hint: related to $w_0$)<jupyter_code>X.insert(0,'Ones',1)
X.head()<jupyter_output><empty_output><jupyter_text>The cost function is expecting numpy array (or matrices) so we need to convert X and t before we can use w (use np.array or np.matrix ...). We also need to initialize w (np.zeros). Question: explain the shape of w?<jupyter_code>X = np.array(X.values)
t = np.array....  ##### COMPLETE YOUR CODE HERE#####
w = np.zeros...   ##### COMPLETE YOUR CODE HERE#####<jupyter_output><empty_output><jupyter_text>Let's take a quick look at the shape of our X,t,w.<jupyter_code>X.shape, w.shape, t.shape<jupyter_output><empty_output><jupyter_text>Now let's compute the cost for our initial solution (0 values for w).<jupyter_code>computeCost(X, t, w)<jupyter_output><empty_output><jupyter_text>Expected result 32.072733877455676. So far so good. Exercise: Define a function to perform gradient descent on the parameters w using the update rules defined in the text.Functions like multiply, sum of numpy may be useful.<jupyter_code>def gradientDescent(X, t, w, eta, iters):
    # Initialize some useful values
    N=...                    # number of training examples
    cost = np.zeros(iters);  

    # make a copy of theta, to avoid changing the original array, since numpy arrays
    # are passed by reference to functions
    w = w.copy()
    
    for i in range(iters):
        ##### WRITE YOUR CODE HERE#####
        w = w - ...##### COMPLETE YOUR CODE HERE#####

        cost[i] = computeCost(X, t, w) # you should stock the cost function for each epoch
                                    # this is useful for checking whether the cost reduces...
    return w, cost<jupyter_output><empty_output><jupyter_text>Initialize some additional variables - the learning rate eta, and the number of iterations to perform.<jupyter_code>eta = 0.01
iters = 1000<jupyter_output><empty_output><jupyter_text>Now let's run the gradient descent algorithm to fit our parameters eta to the training set.<jupyter_code>w, cost = gradientDescent(X, t, w, eta, iters)
w<jupyter_output><empty_output><jupyter_text>Expected result: matrix([[-3.24140214,  1.1272942 ]]).Finally you can compute the cost (error) of the trained model using our fitted parameters.<jupyter_code>computeCost(X, t, w)<jupyter_output><empty_output><jupyter_text>Expected result: 4.515955503078912.Now let's plot the linear model along with the data to visually see how well it fits.<jupyter_code>#TODO: to complete
x = np.linspace(data.Population.min(), data.Population.max(), 100)
f = ... ##### COMPLETE YOUR CODE HERE#####

fig, ax = plt.subplots(figsize=(10,7))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Traning Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')<jupyter_output><empty_output><jupyter_text>Looks pretty good!  Since the gradient decent function also outputs a vector with the cost at each training iteration, we can plot that as well.  Notice that the cost always decreases - this is an example of a convex optimization problem.<jupyter_code>#TODO: plot the error vs. training epoch
fig, ax = plt.subplots(figsize=(10,7))
ax.plot... #use np.arange(iters) to create an array ...
##### COMPLETE YOUR CODE HERE#####<jupyter_output><empty_output><jupyter_text>Part 2: Linear regression with multiple variables This exercise also included a housing price data set with 2 variables (size of the house in square feet and number of bedrooms) and a target (price of the house).  Let's use the techniques we already applied to analyze that data set as well.<jupyter_code>!wget https://www.dropbox.com/s/5b5hnnnn8d4y2o5/ex1data2.txt?dl=0 \
   -O ex1data2.txt<jupyter_output>--2020-02-04 19:28:34--  https://www.dropbox.com/s/5b5hnnnn8d4y2o5/ex1data2.txt?dl=0
Resolving www.dropbox.com (www.dropbox.com)... 172.16.58.3, fc00:e968:6179::de52:7100
Connecting to www.dropbox.com (www.dropbox.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /s/raw/5b5hnnnn8d4y2o5/ex1data2.txt [following]
--2020-02-04 19:28:34--  https://www.dropbox.com/s/raw/5b5hnnnn8d4y2o5/ex1data2.txt
Reusing existing connection to www.dropbox.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://uc64f44bdc997cb1a7884dcf8896.dl.dropboxusercontent.com/cd/0/inline/AxeHKXXTTfPNSDZ-yJA48-xyuSZ7s7pH-Fwd2PW8KEdVXTq2kbTaBQUUhpJxJU9QIWhldcDXD5AWsiVziNfODVUT5e-BEuqPrrBLKu1Z18K0sA/file# [following]
--2020-02-04 19:28:34--  https://uc64f44bdc997cb1a7884dcf8896.dl.dropboxusercontent.com/cd/0/inline/AxeHKXXTTfPNSDZ-yJA48-xyuSZ7s7pH-Fwd2PW8KEdVXTq2kbTaBQUUhpJxJU9QIWhldcDXD5AWsiVziNfODVUT5e-BEuqPrrBLKu1Z18K0sA/file
Resolving uc64[...]<jupyter_text>The notebook will start by loading and displaying some values from this dataset. You then complete the code to:	- Subtract the mean value of each feature from the dataset. 		- After subtracting the mean, additionally scale (divide) the feature values by their respective “standard deviations.”<jupyter_code>file = 'ex1data2.txt'
data2 = pd.read_csv(file, header=None, names=['Size', 'Bedrooms', 'Price'])
data2.head()<jupyter_output><empty_output><jupyter_text>Exercise: add another pre-processing step - normalizing the features. Question: why is it necessary?<jupyter_code>##TODO, normalize your features
data2 = ...##### COMPLETE YOUR CODE HERE#####


#Now show the normalized data
data2.head()<jupyter_output><empty_output><jupyter_text>Now let's repeat our pre-processing steps from part 1 and run the linear regression procedure on the new data set.<jupyter_code>eta=0.01

# add ones column
data2.insert... ##### COMPLETE YOUR CODE HERE#####

# set X (training data) and t (target variable)
cols = data2.shape[1]
X2 = data2.iloc...##### COMPLETE YOUR CODE HERE#####
t2 = ...##### COMPLETE YOUR CODE HERE#####

# convert to matrices/array
X2 = ....##### COMPLETE YOUR CODE HERE#####
t2 = ....##### COMPLETE YOUR CODE HERE#####

# initialize w
# TODO: what is shape of w? why?
# ..........................
w2 = ....##### COMPLETE YOUR CODE HERE#####

# perform linear regression on the data set
w2, cost2 = gradientDescent(X2, t2, w2, eta, iters)

# get the cost (error) of the model
computeCost(X2, t2, w2)<jupyter_output><empty_output><jupyter_text>We can take a quick look at the training progess for this one as well.<jupyter_code>#TODO: plot the error vs. training epoch
fig, ax = plt.subplots(figsize=(10,7))
ax.plot(np.arange(iters), cost, 'r')
ax.set_xlabel('Iterations')
ax.set_ylabel('Cost')
ax.set_title('Error vs. Training Epoch')<jupyter_output><empty_output><jupyter_text>Question: What can you see from this figure ?Answer: ....Exercise: Try out different learning rates for the dataset and find a learning rate that converges quickly.<jupyter_code>###WRITE YOUR CODE HERE
....<jupyter_output><empty_output><jupyter_text>Instead of implementing these algorithms from scratch, we could also use scikit-learn's linear regression function.  Let's apply scikit-learn's linear regressio algorithm to the data from part 1 and see what it comes up with.<jupyter_code>from sklearn import linear_model
model = linear_model.LinearRegression()
model.fit(X, t)<jupyter_output><empty_output><jupyter_text>Here's what the scikit-learn model's predictions look like.<jupyter_code>x = np.array(X[:, 1])
f = model.predict(X).flatten()

fig, ax = plt.subplots(figsize=(10,7))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Traning Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11695906432748537, 'fraction_numerical': 0.026242690058479533, 'mean_word_length': 4.733864207879296, 'pattern_counts': {'":': 0, '<': 98, '<?xml version=': 0, '>': 98, 'https://': 8, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '11314', 'license': 'MIT', 'n_tokens_mistral': 4732, 'n_tokens_neox': 4411, 'n_words': 1525, 'path': 'TensorFlow/TP1 ML/ML_TP1_P1.ipynb', 'repo_name': 'Kuga23/Deep-Learning', 'size': '13680'}