<jupyter_start><jupyter_text>Lab 1: Logistic Regression with Pytorch------------------------------------------------------*Deep Learning. Master in Big Data Analytics**<NAME> <EMAIL>*------------------------------------------------------In this first lab of the course, you will learn basic aspects about how to use [Pytorch](https://pytorch.org/) to train a logistic regressor in a binary classification problem. Pytorch is a powerful library to address Machine Learning problems in general, and deep learning problems in particular, as it provides with automatic differentiation tools. Namely, in our programs we will define functions $f_{\mathbf{w}}(x)$ of given input data $x$ and some parameters $\mathbf{w}$ to be optimized, and we will use Pytorch's [autograd package](https://pytorch.org/docs/stable/autograd.html) to automatically compute $\nabla_{\mathbf{w}} f_{\mathbf{w}}(x)$, and then use the gradient into a Stochastic Gradient Descent (SGD) routine.But lets go step by step. First of all, we will load some database and perform some pre-processing. To do so, we will use the **[Python Data Analysis Library](https://pandas.pydata.org/)** (a.k.a. pandas). You can find in Aula Global a notebook with a gentle introduction to Pandas.<jupyter_code>import pandas as pd # pd is simply an alias 
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline 
%config InlineBackend.figure_format = 'retina' #High quality figures<jupyter_output><empty_output><jupyter_text>Part I. Pre-processing a real datasetThe [Breast Cancer Wisconsin (Diagnostic) Data Set](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)) contains 10 features related to breast tumors that have been diagnosed as benign or malignant. You have a description of the 10 features in the link above. In the same link you can download the data set as a CVS file. We will import it directly using the following class<jupyter_code>def load_spam():
    data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/breast-cancer-wisconsin.data',header=None)
    data.columns=['ID', 'Clump Thickness', 'Uniformity of Cell Size',
       'Uniformity of Cell Shape', 'Marginal Adhesion',
       'Single Epithelial Cell Size', 'Bare Nuclei', 'Bland Chromatin',
       'Normal Nucleoli', 'Mitoses', 'Class']
    return data
''' Data is stored in a Dataframe, a particular data type implemented in Pandas'''

data = load_spam()<jupyter_output><empty_output><jupyter_text>Lets visualize the first 10 entries of the database with the `.head()` method<jupyter_code>data.head(10)<jupyter_output><empty_output><jupyter_text>With the `.shape()` method we can check how many datapoints we have<jupyter_code>print('There are %d data points. Each one of dimension %d' %(data.shape[0],data.shape[1]))<jupyter_output>There are 699 data points. Each one of dimension 11<jupyter_text>We will do the following preprocessing steps. All of them are implemented in Pandas (it is advisory for very large databases):- Remove the 'ID' column. We won't use it anymore- Analyze missing data (Encoded in this database with a '?' by the column mode). - The tumor class is equal to 2 for bening tumors and to 4 for malignat tumors. We will replace this by 0 and 1 respectively- Create a train set and a test set- We will normalize all input variables so they all have zero mean and unit variance  Remove ID columWe can use the `.drop()` method. We use the input `inplace=True` to override the existing Dataframe.<jupyter_code>data.drop('ID',axis=1,inplace=True)
# Lets check the result

data.head(10)<jupyter_output><empty_output><jupyter_text>About missing dataPandas considers values like `NaN` and `None` to represent missing data. The `pandas.isnull` function can be used to tell whether or not a value is missing. Let's use `apply()` across all of the columns in our DataFrame to figure out which values are missing. But first we  change the `?` missing codification by `NaN`.<jupyter_code>data.replace('?', np.nan,inplace=True)
empty = data.apply(lambda col: pd.isnull(col))
empty.sum()<jupyter_output><empty_output><jupyter_text>Observe that only one of the columns has 16 missing entries. For simplicity, let's remove this column. Be aware that in general we will impute missing data with some representative value.<jupyter_code>data.drop('Bare Nuclei',axis=1,inplace=True)
empty = data.apply(lambda col: pd.isnull(col))
empty.sum()<jupyter_output><empty_output><jupyter_text>Create a binary class labelTo do so, we will create a simple function to convert the label and store the result in a new colum of the dataframe<jupyter_code>def convert_to_binary(l):
    return (l-2.0)/2
# We create the new column

data['Binary Class'] = data[['Class']].apply(lambda l: convert_to_binary(l))

data.drop('Class',axis=1,inplace=True)<jupyter_output><empty_output><jupyter_text>Lets take a look to the histogram of the binary class label<jupyter_code>data['Binary Class'].hist()<jupyter_output><empty_output><jupyter_text>Create a train and test setsWe will simply split the dataset at random. 80% of the data will go to the training set, the rest to the test set. We will use [Numpy's random permutation function](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.permutation.html) and the `.iloc()` method.<jupyter_code>frac_train = 0.8

data_train = int(data.shape[0]*frac_train) # Training points

np.random.seed(seed=10) #To fix the random seed. So we all get the same partition

mask = np.random.permutation(data.shape[0]) # Random order of data indexes

train_data = data.iloc[list(mask[:data_train])].copy()

test_data = data.iloc[list(mask[data_train:])].copy()<jupyter_output><empty_output><jupyter_text>Note that we store the train and test data in separate Dataframes, so we can normalize them without modifying the oringinal data (We could add more columns to the original Dataframe, but this would be later on annoying to index). Also, the train/test partition and normalization can also easily done using Numpy and the [sklearn library](https://scikit-learn.org/stable/). You will see this in the Machine Learning course.  Normalize input variables In general, it is not recommended that input variables (or features) are defined in different ranges. I.e., $x_1$ for instance takes values in the $[-1,1]$ set and $x_2$ takes values in the $[-10^6,10^6]$ range. To improve both the numerical robustness of our estimators and **enhance interpretability**, we will apply a linear normalization preprocessing stage to use as input to the linear regression features with (sample) mean 0 and (sample) variance equal to 1. Given the unnormalized feature matrix $\mathbf{X}_{N\times D}$, we compute the sample mean per feature column, $\mu_j$, and the sample variance per  feature column, $\sigma^2_j$ for $j=1,\ldots,D$. Then, each row of the normalized feature matrix $\overline{\mathbf{X}}_{N\times D}$ is obtained as follows:$$\overline{\mathbf{x}}^{(i)}= \left[\frac{x_1^{(i)}-\mu_1}{\sqrt{\sigma^2_1}}, ~~ \frac{x_1^{(i)}-\mu_2}{\sqrt{\sigma^2_2}}, \ldots, \frac{x_1^{(D)}-\mu_1}{\sqrt{\sigma^2_D}}\right]$$**Note this does not affect to the logistic regression solution**, we are simply re-scaling the parameter vector $\boldsymbol{w}$. Using this scaling, we can now effectively compare the effect that each variable has in the logistic regression solution: **the larger $|w_j|$ is, the more effect the $j$-th feature has in the estimation of the class**.One important aspect is that the test set is normalized **using the train set statistics (mean and variance)**<jupyter_code>means = train_data.mean()
stds = train_data.std()
for idx,column_name in enumerate(train_data.columns[:-1]):
    print(column_name)
    train_data[column_name] = (train_data[column_name]-means[idx])/(stds[idx])
    test_data[column_name] = (test_data[column_name]-means[idx])/(stds[idx])<jupyter_output>Clump Thickness
Uniformity of Cell Size
Uniformity of Cell Shape
Marginal Adhesion
Single Epithelial Cell Size
Bland Chromatin
Normal Nucleoli
Mitoses<jupyter_text>Lets take a look to the resulting dataframe<jupyter_code>train_data.head(10)<jupyter_output><empty_output><jupyter_text>Part II. Tensors in PytorchNote: a big part of the following material is a personal wrap-up of [Facebook's Deep Learning Course in Udacity](https://www.udacity.com/course/deep-learning-pytorch--ud188). So all credit goes for them!!PyTorch in a lot of ways behaves like  Numpy arrays. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients and another module specifically for building neural networks. It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.  Creating and manipulating tensors<jupyter_code># First, import PyTorch
import torch
# To check what torch version you're using

torch.__version__<jupyter_output><empty_output><jupyter_text>Lets create a function to manipulate torch tensors. Let's program our own logistic regressor$$f(\mathbf{x}) = \frac{1}{1+\exp-(\mathbf{w}^T\mathbf{x}+w_0)}=\sigma(\cdot)$$where $\mathbf{w}$ and $w_0$ (the bias or intercept) are initialized with Pytorch tensors containing random numbers drawn from a i.i.d. Normal Gaussian distribution. Also, to test the function, we will generate data points $\mathbf{x}$ also as random tensors. First we define the sigmoid function $\sigma(\cdot)$<jupyter_code>def sigmoid(x):
    """ Sigmoid activation function 
    
        Arguments
        ---------
        x: torch.Tensor
    """
    return 1/(1+torch.exp(-x))<jupyter_output><empty_output><jupyter_text>Now we generate the parameters $\mathbf{w},w_0$ and one input vector (feature vector) $\mathbf{x}$<jupyter_code>torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
feature_vector = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(feature_vector)
# and a true bias term
bias = torch.randn((1, 1))<jupyter_output><empty_output><jupyter_text>Above we generated data we can use to get the output of our simple neural network. This is all just random for now, going forward we'll start using normal data. Going through each relevant line:`features = torch.randn((1, 5))` creates a tensor with shape `(1, 5)`, one row and five columns, that contains values randomly distributed according to the normal distribution with a mean of zero and standard deviation of one. `weights = torch.randn_like(features)` creates another tensor with the same shape as `features`, again containing values from a normal distribution.Finally, `bias = torch.randn((1, 1))` creates a single value from a normal distribution.PyTorch tensors can be added, multiplied, subtracted, etc, just like Numpy arrays. In general, you'll use PyTorch tensors pretty much the same way you'd use Numpy arrays.Just like in Numpy, you can check the values and dimensions of a tensor:<jupyter_code>print(feature_vector)
m,n = feature_vector.shape
print("The dimensions are {0},{1}".format(m,n))<jupyter_output>tensor([[-0.1468,  0.7861,  0.9468, -1.1143,  1.6908]])
The dimensions are 1,5<jupyter_text>> **Exercise**: Using the `sigmoid()` function, calculate the output a binary logistic regressor with input feature vector `feature_vector`, weights `weights`, and bias `bias`. You can use `*` for Tensor elemenwise product and the function  [`torch.sum()`](https://pytorch.org/docs/stable/torch.htmltorch.sum) to sum all the elements of a tensor in a certain dimension.<jupyter_code>### YOUR CODE HERE
print(f'The answer of the sigmoid function with the given values is: {sigmoid(torch.sum(weights*feature_vector)+bias)[0][0]}')<jupyter_output>The answer of the sigmoid function with the given values is: 0.15950655937194824<jupyter_text>Do the same but using matrix multiplication with `torch.mm()`You can do the multiplication and sum in the same operation using a matrix multiplication. In general, you'll want to use matrix multiplications since they are more efficient and accelerated using modern libraries and high-performance computing on GPUs.  For this we can use [`torch.mm()`](https://pytorch.org/docs/stable/torch.htmltorch.mm) or [`torch.matmul()`](https://pytorch.org/docs/stable/torch.htmltorch.matmul) which is somewhat more complicated and supports broadcasting.First, check the dimension of `feature_vector` and `weights` using the method `.shape`<jupyter_code>print(feature_vector.shape)
print(weights.shape)<jupyter_output>torch.Size([1, 5])
torch.Size([1, 5])<jupyter_text>See what comes if you try to directly multiply both tensors<jupyter_code>#torch.mm(feature_vector,weights)<jupyter_output><empty_output><jupyter_text>What's happening here is our tensors aren't the correct shapes to perform a matrix multiplication. There are a few options here: [`weights.reshape()`](https://pytorch.org/docs/stable/tensors.htmltorch.Tensor.reshape), and [`weights.view()`](https://pytorch.org/docs/stable/tensors.htmltorch.Tensor.view).* `weights.reshape(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)` sometimes, and sometimes a clone, as in it copies the data to another part of memory.* `weights.view(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)`.Any of the two methods will work for this. So, now we can reshape `weights` to have five rows and one column with something like `weights.view(5, 1)`.> **Exercise**: Calculate the output of the logistic regressor using matrix multiplication.<jupyter_code># YOUR CODE HERE
# using the function torch.matmul() for matrix multiplications:

y = sigmoid(torch.matmul(feature_vector,weights.reshape(5,1))+bias)
print("The ouput of the LR is {0}".format(y[0][0])) 
# we see that we get the same result for both cases<jupyter_output>The ouput of the LR is 0.15950657427310944<jupyter_text>Numpy to Torch and backPyTorch has a feature for converting between Numpy arrays and Torch tensors. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.<jupyter_code>a = np.random.rand(4,3)
a
b = torch.from_numpy(a)
b
b.numpy()<jupyter_output><empty_output><jupyter_text>The **memory is shared between the Numpy array and Torch tensor**, so if you change the values in-place of one object, the other will change as well.<jupyter_code># Multiply PyTorch Tensor by 2, in place (Note the _)
b.mul_(2)
# Numpy array matches new values from Tensor
a<jupyter_output><empty_output><jupyter_text>Part III. Define a logistc regression and the loss function in Pytorch The `nn.()` modulePyTorch provides a module `nn` that makes building neural networks much simpler.<jupyter_code>from torch import nn<jupyter_output><empty_output><jupyter_text>You should to get use to inspect [Pytorch official documentation](https://pytorch.org/docs/stable/index.html) to understand the structure and usability of methods and classes required to work with Pytorch and train neural networks.With the following code I show you how to create a Logistic Regression network whose parameters will be later on optimized for our data.<jupyter_code>class LR(nn.Module):
    def __init__(self,dimx):
        super().__init__()
        
        self.weights = nn.Parameter(torch.randn(dimx,1),requires_grad = True)
        
        self.bias = nn.Parameter(torch.randn(1,1),requires_grad = True)
        
        # Define sigmoid activation and softmax output 
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.sigmoid(torch.matmul(x,self.weights)+self.bias)
        return x<jupyter_output><empty_output><jupyter_text>Let's go through this step by step.```pythonclass Network(nn.Module):```Here we're inheriting from `nn.Module`. Combined with `super().__init__()` this creates a class that tracks the architecture and provides a lot of useful methods and attributes. It is mandatory to inherit from `nn.Module` when you're creating a class for your neural network. The name of the class itself can be anything.```pythonself.weights = nn.Parameter(torch.randn(dimx,1),requires_grad = True)self.bias = nn.Parameter(torch.randn(1,1),requires_grad = True)```The Parameter class defines a kind of Tensor that is to be considered a module parameter. Namely, we will optimize the values of these tensors. Note that we define a **random initialization** with `torch.randn()`.```pythondef forward(self, x):```PyTorch networks created with `nn.Module` must have a `forward` method defined. It takes in a tensor `x` and passes it through the operations you defined in the `__init__` method.  A more compact definition of the network using pre-defined layersThe `.nn` package provides us with predefined layers and operators so that the implementation of our networks is indeed much easier (and compact). The following code performs exactly as the class `LR` defined above.<jupyter_code>class LR_compact(nn.Module):
    def __init__(self,dimx):
        super().__init__()
        
        self.output = nn.Linear(dimx,1)
    
        # Define sigmoid activation and softmax output 
        self.sigmoid = nn.Sigmoid()
        
    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.output(x)
        x = self.sigmoid(x)
        return x<jupyter_output><empty_output><jupyter_text>When we define a logistic regressor by instantiating the class `LR_compact`, we can access the weights and the bias as `self.output.weight` and `self.output.bias`  Evaluate the logistic regressor using our dataLets go back to the database we just preprocessed above. Consider now the first 10 datapoints in the training set.<jupyter_code>x = np.array(train_data[:10]).astype(np.float32) # train_data is a Dataframe, can easily be converted to a numpy array
y = x[:,-1]    # Last Column is the class
x = x[:,:-1]   # We remove the last column from x
print("The dimensions of x are ({0},{1})\n".format(x.shape[0],x.shape[1]))

print(x)<jupyter_output>The dimensions of x are (10,8)

[[ 1.300525    2.2856638   2.2764957   2.4554093   2.2863305   2.6582384
   1.3306657   0.8670005 ]
 [-1.2125387  -0.7003592  -0.7478111  -0.6429514  -0.55274856 -1.010787
  -0.60810614 -0.3386047 ]
 [ 0.22349761 -0.7003592  -0.7478111  -0.6429514  -0.55274856 -0.60311747
  -0.2849775  -0.3386047 ]
 [ 0.22349761 -0.7003592  -0.41177702 -0.6429514  -0.55274856 -1.010787
  -0.60810614 -0.3386047 ]
 [ 0.22349761 -0.7003592  -0.7478111   0.04557319  0.39361116 -0.195448
  -0.2849775  -0.3386047 ]
 [ 2.018543   -0.03679854  0.93235934 -0.29868913 -0.07956871  0.21222152
   2.3000517   0.26419792]
 [ 0.22349761 -0.7003592  -0.7478111  -0.6429514  -0.55274856 -0.195448
  -0.60810614 -0.3386047 ]
 [-0.8535297  -0.7003592  -0.7478111  -0.6429514  -0.55274856 -1.010787
  -0.60810614 -0.3386047 ]
 [ 1.300525    1.6221032   1.6044276  -0.6429514  -0.55274856  1.0275605
   2.3000517  -0.3386047 ]
 [-1.2125387  -0.7003592  -0.7478111  -0.6429514  -0.55274856 -0.195448[...]<jupyter_text>> **Exercise:** Instantiate the class `LR_compact` with the dimension of every data point in `x`<jupyter_code>## YOUR CODE HERE
# must define the dimension of the input vector, in this case the number of columns of x: x.shape[1]
my_classifier = LR_compact(x.shape[1])<jupyter_output><empty_output><jupyter_text>> **Exercise:** Print the current parameters of the classifier (weight vector and bias)<jupyter_code>## YOUR CODE HERE
# weights are generated randomly at the start
print(f'the weights of the classifier are: {my_classifier.output.weight}')
# the bias of the classifier (are also generated randomly):
print(f'the bias of the classifier are: {my_classifier.output.bias}')<jupyter_output>the bias of the classifier are: Parameter containing:
tensor([0.0896], requires_grad=True)<jupyter_text>To evaluate the logistic regressor output for the data in `x`, we make use of the class `forward` method. Do not forget to feed the function with a **torch tensor**.<jupyter_code>output = my_classifier.forward(torch.tensor(x))
# this is the output using the sigmoid function with the randomly generated weights and biases
print(output)<jupyter_output>tensor([[0.7749],
        [0.4365],
        [0.4343],
        [0.4641],
        [0.4662],
        [0.7065],
        [0.3877],
        [0.4378],
        [0.6772],
        [0.3827]], grad_fn=<SigmoidBackward>)<jupyter_text>Above we can see the operation that created `output`, a sigmoid operation `SigmoidBackward`. The autograd module keeps track of **all operations that affect Parameter tensors** and use such information to calculate the gradient for each one. In this way, it's able to calculate the gradients for a chain of operations, with respect to any one tensor.  Evaluating the binary cross entropy function using `nn.BCE()`Let's reduce the tensor `y` to a scalar value, the binary cross entropy. We will use the class `nn.BCE()`. See how to use the function in the [official documentation](https://pytorch.org/docs/stable/nn.html).<jupyter_code>bce = nn.BCELoss()
# the bce function needs the x and y tensors to compute the 'difference' between them.
# for the y vector we need to reshape it in order to have the same shape as the x vector.
loss = bce(output,torch.tensor(y).view(output.shape[0],1))

# this is the loss 'difference' between the y and x vectors
print(loss)<jupyter_output>tensor(0.4936, grad_fn=<BinaryCrossEntropyBackward>)<jupyter_text>Above we can see the operation that created `loss`, a binary cross entropy `BinaryCrossEntropyBackward`.Now that we know how to calculate a loss, Torch provides a module, `autograd`, for automatically calculating the gradients of the loss w.r.t. the tensors. Autograd works by keeping track of operations performed on tensors, then going backwards through those operations, calculating gradients along the way.To make sure PyTorch keeps track of operations on a tensor and calculates the gradients, you need to set `requires_grad = True` on a tensor. You can do this at creation with the `requires_grad` keyword, or at any time with `x.requires_grad_(True)`. By default, all tensors created by instantiating the class `nn.Parameter` have `requires_grad = True`.> **Exercise**: Check that both the weights and the bias of the object have this flag set to `True`<jupyter_code># YOUR CODE HERE 
# We need to have the requires_grad flag activated, so that pytorch can keep track of the operations and calculate the gradients

my_classifier.output.weight.requires_grad
my_classifier.output.bias.requires_grad
# we see that they do require grads<jupyter_output><empty_output><jupyter_text>You can turn off gradients for a block of code with the `torch.no_grad()` content:```pythonx = torch.zeros(1, requires_grad=True)>>> with torch.no_grad():...     y = x * 2>>> y.requires_gradFalse```Also, you can turn on or off gradients altogether with `torch.set_grad_enabled(True|False)`. Turning off gradients is recommended for instance when you are just evaluating you network, i.e., when training is finished and only parameter evaluation is required.  The `.backward()` operatorThe gradients are computed with respect to some variable `z` with `z.backward()`. This does a backward pass through the operations that created `z`. This means that when we calculate the loss and call `loss.backward()`, the gradients of the loss function w.r.t. the parameters are calculated. These gradients are used to update the weights with gradient descent.After calling  `loss.backward()` the gradient of `loss` w.r.t. to any tensor `x` can be accessed as `x.grad`> **Exercise**: Print the gradient of the loss function w.r.t. the model parameters for the `LR` object (or the `LR_compact`) before and after running `loss.backward()`.<jupyter_code>#YOUR CODE HERE
# before running the loss backward funcion, we see that there is no gradient
print(my_classifier.output.weight.grad)

loss.backward()

# after running it we observe that the gradient has been calculated
print(my_classifier.output.weight.grad)
# if we try to run it again, we will get an error, since the gradient has already been computed<jupyter_output>None
tensor([[-0.2280, -0.3135, -0.3398, -0.1871, -0.1535, -0.2849, -0.3256, -0.1182]])<jupyter_text>> **Exercise**: Complete the following code, in which given the tensor $u$ (of size $10\times1$), you have to evaluate the gradient of $y=u^T~u$ w.r.t. every component of $u$. Does it coincide with the analytical solution?<jupyter_code>u = torch.randn(10,1,requires_grad=True)

#YOUR CODE HERE (2 lines of code)
y = torch.matmul(u.view(1,10),u)
y.backward()



# Turn off gradients for validation, saves memory and computations
with torch.no_grad():
    plt.stem(u.grad.numpy(),label='Autograd',use_line_collection=True)
    plt.plot(2*u.detach().numpy(),'-g',label='$2x$')
    plt.grid()
    plt.legend()<jupyter_output><empty_output><jupyter_text>`.detach()` method constructs a new view on a tensor which is declared not to need gradients. You can see a nice example in this [link](http://www.bnikolic.co.uk/blog/pytorch-detach.html). Typically, we will use detach to export values to numpy to perform model evaluation. In Pytorch, you cannot call the `.numpy()` operators in Tensor that has `require_grad=True`.An important aspects to implement Gradient Descent Loops is that if call again the `.backward()` operator, then **gradients are accumulated** in the variable `x.grad`. If we recompute `y`, we can check that gradients are accumulated by calling again `y.backward()`<jupyter_code>y = torch.matmul(u.view(1,10),u)
y.backward()

plt.stem(u.grad.numpy(),label='Autograd',use_line_collection=True)
plt.plot(2*u.detach().numpy(),'-g',label='$2x$')
plt.plot(4*u.detach().numpy(),'-b',label='$4x$')
plt.grid()
plt.legend()<jupyter_output><empty_output><jupyter_text>To **reset gradients**, Pytorch optimizers provide us with the appropiate method.  Part IV. Train the logistic regressor with the complete datasetWhen we create a network with PyTorch, all of the parameters are initialized with `requires_grad = True`. This means that when we calculate the loss and call `loss.backward()`, the gradients for the parameters are calculated. These gradients are used to update the weights with gradient descent. Below you can see an example of calculating the gradients using a backwards pass.<jupyter_code># We use all data points  

x = np.array(train_data).astype(np.float32)
# Last Column is the class
y = x[:,-1]  
x = x[:,:-1]
output = my_classifier.forward(torch.tensor(x))

loss = bce(output,torch.tensor(y).view(output.shape[0],1))

loss.backward()

print(my_classifier.output.weight.grad.shape)<jupyter_output>torch.Size([1, 8])<jupyter_text>Defining an optimizerThere's one last piece we need to start training, an optimizer that we'll use to update the weights with the gradients. We get these from PyTorch's [`optim` package](https://pytorch.org/docs/stable/optim.html). For example we can use stochastic gradient descent with `optim.SGD`. You can see how to define an optimizer below.With the following code, we can define the optimizer:<jupyter_code>from torch import optim

# Optimizers require the parameters to optimize and a learning rate
optimizer = optim.SGD(my_classifier.parameters(), lr=0.01)<jupyter_output><empty_output><jupyter_text>To perform a SGD iteration, we simply call `optmizer.step()`By the way, `my_classifier.parameters()` returns a [generator object](https://realpython.com/introduction-to-python-generators/). Namely, a list with all the parameters that you read one by one or iterate through a for loop.<jupyter_code>params = my_classifier.parameters()

print(type(params))
# We print the first set of parameters (weight vector)
print(next(params))
# We print the second set of parameters (the bias)
print(next(params))<jupyter_output><class 'generator'>
Parameter containing:
tensor([[ 0.0148,  0.0659,  0.2685,  0.0909,  0.1876, -0.2735,  0.2517,  0.1217]],
       requires_grad=True)
Parameter containing:
tensor([0.0896], requires_grad=True)<jupyter_text>If we call again the method `next()` over params, Python will yield a `StopIteration` error:<jupyter_code>#print(next(params))<jupyter_output><empty_output><jupyter_text>Creating a class with both the network and a training methodObject Oriented Programming (OOP) in Python is a versatil and convenient method to implement our own software and create reusable sofware. Indeed, along this whole notebook, we have been using classes all the time.[Here](https://www.programiz.com/python-programming/object-oriented-programming) you can find a short introduction to OOP in Python.With the following code, we will define a class that incorporates the definition of the logistic regression network and **a method to train the parameters**. Go carefully through the code and try to understand it line by line.<jupyter_code>''' This class inherits from the LR_compact class. So it has the same atributes
and methods, and some others that we will add. 
'''
class LR_extended(LR_compact):
    
    # extended initiallitation method for the extended class LR_extended
    def __init__(self,dimx,sgd_iterations=1000,lr=0.001,print_every=200):
        
        # takes the initialization method from its parent class 
        # super().__init__(arguments from parent class, without 'self')
        super().__init__(dimx)  #To initialize LR_compact!
        
        self.lr = lr #Learning Rate
        
        # self.parameters are taken from the parent class
        self.optim = optim.SGD(self.parameters(), self.lr)
        
        #Stochastic gradient descent steps
        self.sgd_iterations = sgd_iterations 
        
        # type of critetion applied
        self.criterion = nn.BCELoss()
        
        # A list to store the loss evolution along training
        self.loss_during_training = [] 
        
        # print every number of steps (200)
        self.print_every = print_every
        
    # train method for the extended class only
    def train(self,x,y):
        
        # Stochastic gradient descent loop 
        for iter in range(int(self.sgd_iterations)):
        
            self.optim.zero_grad()  #TO RESET GRADIENTS!

            # get the output of the last computed values
            out = self.forward(x)

            # compute the loss function with those values
            loss = self.criterion(out,y.view(x.shape[0],1))

            # append this error to the error list
            self.loss_during_training.append(loss.item())
            # get the recently computed gradients
            loss.backward()

            self.optim.step()
        
            if(iter % self.print_every == 0): # Every 1000 iterations
                
                print("Training loss after %d iterations: %f" 
                      %(iter,self.loss_during_training[-1]))<jupyter_output><empty_output><jupyter_text>Lets instantiate the class and train the logistic regressor. See how compact and easy is now (OOP is great!).<jupyter_code># instantiate the linear regression with the data we have
# to instantiate we need to pass the dimension of x (columns) and the other parameters are optional
# but we will put 3000 max iterations and a learning rate of 0.001
my_LR = LR_extended(x.shape[1],sgd_iterations=3000,lr=0.001)

# now we apply the train method, which needs the x and y variables 
my_LR.train(torch.tensor(x),torch.tensor(y))<jupyter_output>Training loss after 0 iterations: 0.768621
Training loss after 200 iterations: 0.595879
Training loss after 400 iterations: 0.488465
Training loss after 600 iterations: 0.417329
Training loss after 800 iterations: 0.367228
Training loss after 1000 iterations: 0.330191
Training loss after 1200 iterations: 0.301776
Training loss after 1400 iterations: 0.279332
Training loss after 1600 iterations: 0.261186
Training loss after 1800 iterations: 0.246234
Training loss after 2000 iterations: 0.233715
Training loss after 2200 iterations: 0.223091
Training loss after 2400 iterations: 0.213972
Training loss after 2600 iterations: 0.206064
Training loss after 2800 iterations: 0.199146<jupyter_text>Note that if you believe that training isn't finished yet, you can simply call again the `.train()` method.<jupyter_code>my_LR.train(torch.tensor(x),torch.tensor(y))<jupyter_output>Training loss after 0 iterations: 0.193047
Training loss after 200 iterations: 0.187633
Training loss after 400 iterations: 0.182797
Training loss after 600 iterations: 0.178453
Training loss after 800 iterations: 0.174531
Training loss after 1000 iterations: 0.170973
Training loss after 1200 iterations: 0.167733
Training loss after 1400 iterations: 0.164770
Training loss after 1600 iterations: 0.162051
Training loss after 1800 iterations: 0.159548
Training loss after 2000 iterations: 0.157236
Training loss after 2200 iterations: 0.155095
Training loss after 2400 iterations: 0.153107
Training loss after 2600 iterations: 0.151256
Training loss after 2800 iterations: 0.149529<jupyter_text>> **Exercise**: Train from scratch the logistic regresson for 20.000 iterations and plot the evolution of the binary cross entropy loss.<jupyter_code># Your code here

# we need to instantiate the class again: 

my_LR = LR_extended(x.shape[1], sgd_iterations=2e4, lr=0.001, print_every=9000)

# and train the model again: 

my_LR.train(torch.tensor(x),torch.tensor(y))


# here we will print the evolution of the training error by using the attribute 
# loss_during_training that is storing all the values
plt.plot(my_LR.loss_during_training)
plt.xlabel('Iterations')
plt.ylabel('Error')
plt.title('Evolution of the training error')
plt.show()<jupyter_output><empty_output><jupyter_text>> **Exercise**: Print all model parameters (e.g. weights and bias)<jupyter_code>#Your code here
# Now we will print the model parameters. These values were constantly updated
# Inside the class my_LR, since it inhereted the parameters from the parent class
# LR, that had the following attributes: LR.output.weight and LR.output.bias

print(f'The model after ttraining has the weights: {my_LR.output.weight[0]}')
print(f'And the following bias: {my_LR.output.bias[0]}')<jupyter_output>The model after ttraining has the weights: tensor([0.9383, 0.5689, 0.8222, 0.7527, 0.3529, 0.7652, 0.6367, 0.5520],
       grad_fn=<SelectBackward>)
And the following bias: -0.6298190951347351<jupyter_text>Evaluating the model in the test setNow that we have trained our logistic regressor, it is time to compare its performance in both the training and the test sets.> **Exercise**: Compute the logistic regressor output for both the training data and the test data.<jupyter_code># the forward method will compute the results with the already trained parameters
out_train = my_LR.forward(torch.tensor(x))

#Your code here
# convert the test_data dataframe as a numpy array
xtest = np.array(test_data).astype(np.float32)

# Last Column is the class
ytest = xtest[:,-1]   
xtest = xtest[:,:-1]

out_test = my_LR.forward(torch.tensor(xtest))<jupyter_output><empty_output><jupyter_text>Recall that the logistic regressor output is an estimated class probability. Assuming we put the threshold at a probability of 0.5, lets count the number of errors in both sets.<jupyter_code># convert all output values that are greater than 0.5 to 1, and those lower to 0
error_rate_train = np.sum((out_train.detach().numpy()>=0.5) 
                          == y.reshape([-1,1]))/y.shape[0]

error_rate_test = np.sum((out_test.detach().numpy()>=0.5) 
                         == ytest.reshape([-1,1]))/ytest.shape[0]

print(error_rate_train)

print(error_rate_test)<jupyter_output>0.9534883720930233
0.9714285714285714<jupyter_text>Part V. Training a Logistic Regressor with mini-batch stochastic gradient descentFinally, we are going to implement an scalable version of the gradient descent training implemented above. At every iteration, instead of evaluating the gradient using all data points, a small minibatch of data will be used.To make sure that all training data points are evenly used to evaluate the gradient, we will use a short iterating function. Instead of predefining a certain number of SGD iterations, we define a certain number of **epochs**. After every epoch **all datapoints** have been used in the optimizer once. > **Exercise**: Complete the code for the following class<jupyter_code>''' This class inherits from the LR_compact class. So it has the same atributes
and methods, and some others that we will add. 
'''
class LR_stochastic(LR_compact):
    
    # now we initialize it with the number of epochs as well
    def __init__(self,dimx,num_train_data,
                 epochs=100,lr=0.001,batch_size=50,print_every=20):
        
        super().__init__(dimx)  #To initialize LR2!
        
        self.lr = lr #Learning Rate
        
        # take parameters from parent class
        self.optim = optim.SGD(self.parameters(), self.lr)
        
        self.epochs = epochs
        
        self.print_every = print_every  # Print loss every 
        
        self.criterion = nn.BCELoss()
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        
        self.batch_size = batch_size
        
        self.num_train = num_train_data
        
        self.num_batchs = np.floor(self.num_train/self.batch_size)
        
        
    def train(self,x,y):
        
        # SGD Loop
        
        # first loop for the epochs
        for e in range(int(self.epochs)):
            
            # Random data permutation at each epoch
            # generate random indexes for the batch to go around
            idx = np.random.permutation(self.num_train)
            
            # put the loss back to zero
            running_loss = 0.
            
            # second loop for the iterations
            for i in range(int(self.num_batchs)):
        
                self.optim.zero_grad()  #TO RESET GRADIENTS!
            
                # Indexes of the datapoints that enter the batch
                idx_batch = idx[i*self.batch_size:(i+1)*self.batch_size]

                # generate the outputs for the selected batch of points
                out = self.forward(x[idx_batch,:]) 

                
                loss = self.criterion(out,y[idx_batch].view(self.batch_size,1))

                # accumulate the losses
                running_loss += loss.item()

                #Your code here. Compute gradients
                loss.backward()
                
                #Your code here. Perform one SGD step
                self.optim.step()
                # finish the iterations inside an epoch
                
            self.loss_during_training.append(running_loss/self.num_batchs)

            if(e % self.print_every == 0 or e == int(self.epochs)-1): 

                print("Training loss after %d epochs: %f" 
                      %(e,self.loss_during_training[-1]))<jupyter_output><empty_output><jupyter_text>> **Exercise**: Train a logistic regressor with minibatch-stochastic gradient descent with different batch sizes (10,20,50 and 100) and plot the evolution of the loss function for all cases in the same plot. Also, compare with the evolution of the loss function when all data is used. Note that when all datapoints are used, every GD iteration correspond to a **full epoch**.>> Finally, observe if in all cases the parameters of the models (weights and biases) are similar.<jupyter_code>#Your code here
print('batch: 10')
# batch = 10
my_LR10 = LR_stochastic(x.shape[1], x.shape[0], epochs=500, batch_size=10, print_every=100)
my_LR10.train(torch.tensor(x), torch.tensor(y))

print('batch: 20')
# batch = 20
my_LR20 = LR_stochastic(x.shape[1], x.shape[0], epochs=500, batch_size=20, print_every=100)
my_LR20.train(torch.tensor(x), torch.tensor(y))

print('batch: 50')
# batch = 50
my_LR50 = LR_stochastic(x.shape[1], x.shape[0], epochs=500, batch_size=50, print_every=100)
my_LR50.train(torch.tensor(x), torch.tensor(y))

print('batch: 100')
# batch = 100
my_LR100 = LR_stochastic(x.shape[1], x.shape[0], epochs=500, batch_size=100, print_every=100)
my_LR100.train(torch.tensor(x), torch.tensor(y))


plt.plot(my_LR10.loss_during_training, '-b', label='batch=10')
plt.plot(my_LR20.loss_during_training, '-g', label='batch=20')
plt.plot(my_LR50.loss_during_training, '-y', label='batch=50')
plt.plot(my_LR100.loss_during_training, '-r', label='batch=100')
#plt.plot(my_LR.loss_during_training, '-v')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.title('Evolution of the loss function w.r.t batch size')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>> **Exercise**: Add a method to the above class that evaluates the classification error rate for a given set of data points and the corresponding labels. Hint: we have done this previously in the notebook, you only  have to encapsulate it in the class as a new method.>>Recall to disable gradient computation for these computations (it saves time and operations!)<jupyter_code># We need to create another child class from the parent LR
''' This class inherits from the LR_compact class. So it has the same atributes
and methods, and some others that we will add. 
'''
class LR_stochastic(LR_compact):
    
    # now we initialize it with the number of epochs as well
    def __init__(self,dimx,num_train_data,
                 epochs=100,lr=0.001,batch_size=50,print_every=20):
        
        super().__init__(dimx)  #To initialize LR2!
        
        self.lr = lr #Learning Rate
        
        # take parameters from parent class
        self.optim = optim.SGD(self.parameters(), self.lr)
        
        self.epochs = epochs
        
        self.print_every = print_every  # Print loss every 
        
        self.criterion = nn.BCELoss()
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        
        self.batch_size = batch_size
        
        self.num_train = num_train_data
        
        self.num_batchs = np.floor(self.num_train/self.batch_size)
        

    def error_rate(self, x, y):

      with torch.no_grad():

        out = self.forward(x)
        error = np.sum((out.detach().numpy()>=0.5) == y.reshape([-1,1]))/y.shape[0]
        
      return error


    def train(self,x,y):
        
        # SGD Loop
        
        # first loop for the epochs
        for e in range(int(self.epochs)):
            
            # Random data permutation at each epoch
            # generate random indexes for the batch to go around
            idx = np.random.permutation(self.num_train)
            
            # put the loss back to zero
            running_loss = 0.
            
            # second loop for the iterations
            for i in range(int(self.num_batchs)):
        
                self.optim.zero_grad()  #TO RESET GRADIENTS!
            
                # Indexes of the datapoints that enter the batch
                idx_batch = idx[i*self.batch_size:(i+1)*self.batch_size]

                # generate the outputs for the selected batch of points
                out = self.forward(x[idx_batch,:]) 

                
                loss = self.criterion(out,y[idx_batch].view(self.batch_size,1))

                # accumulate the losses
                running_loss += loss.item()

                #Your code here. Compute gradients
                loss.backward()
                
                #Your code here. Perform one SGD step
                self.optim.step()
                # finish the iterations inside an epoch
                
            self.loss_during_training.append(running_loss/self.num_batchs)

            if(e % self.print_every == 0 or e == int(self.epochs)-1): 

                print("Training loss after %d epochs: %f" 
                      %(e,self.loss_during_training[-1]))

my_LR = LR_stochastic(x.shape[1], x.shape[0], batch_size=20, print_every=100, epochs=500)

my_LR.train(torch.tensor(x), torch.tensor(y))
my_LR.error_rate(torch.tensor(x), y)<jupyter_output>Training loss after 0 epochs: 0.623416
Training loss after 100 epochs: 0.194229
Training loss after 200 epochs: 0.152751
Training loss after 300 epochs: 0.136588
Training loss after 400 epochs: 0.128842
Training loss after 499 epochs: 0.118356<jupyter_text>Part VI. Regularize the model weights > **Exercise**: Implement a class for regularized logistic regression, in which you penalize the $L_2$ norm of the weights (do not penalize for the bias). Use a pre-defined regularization parameter that you specify when instantiate the class.>> Recall that $L_2$ is simply the sum of the squared elements of a vector.  For a given tensor `u` you can get this > norm by calling `torch.norm(u)`, which returns the squared-root of the norm. The idea is that you add this norm to the loss function, with a regularization parameter $\lambda$:>> $$\mathcal{L}_{reg} = \mathcal{L}_{BCE} + \lambda ||\mathbf{w}||^2_2$$>> Train the model with SGD and compare the train/test performance as you vary $\lambda$. Hint: Since in this case the > unregularized LR model performs really well on both train and test datasets, there is actually no need for   regularization. So do not expect much improvement, rather the contrary. We will reuse this code in future projects.<jupyter_code># now we apply the same function but with the added regularization parameter to the ridge regression
# We need to create another child class from the parent LR
''' This class inherits from the LR_compact class. So it has the same atributes
and methods, and some others that we will add. 
'''
class LR_stochastic_reg(LR_compact):
    
    # now we initialize it with the number of epochs as well
    def __init__(self,dimx,num_train_data,
                 epochs=100,lr=0.001,batch_size=50,print_every=20,lam=1.0):
        
        super().__init__(dimx)  #To initialize LR2!
        
        self.lr = lr #Learning Rate
        
        # take parameters from parent class
        self.optim = optim.SGD(self.parameters(), self.lr)
        
        self.epochs = epochs
        
        self.print_every = print_every  # Print loss every 
        
        self.criterion = nn.BCELoss()
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        
        self.batch_size = batch_size
        
        self.num_train = num_train_data
        
        self.num_batchs = np.floor(self.num_train/self.batch_size)

        # regularization parameter

        self.lam = lam
        

    def error_rate(self, x, y):

      with torch.no_grad():

        out = self.forward(x)
        error = np.sum((out.detach().numpy()>=0.5) == y.reshape([-1,1]))/y.shape[0]

      self.accuracy = error 
      return error


    def train(self,x,y):
        
        # SGD Loop
        
        # first loop for the epochs
        for e in range(int(self.epochs)):
            
            # Random data permutation at each epoch
            # generate random indexes for the batch to go around
            idx = np.random.permutation(self.num_train)
            
            # put the loss back to zero
            running_loss = 0.
            
            # second loop for the iterations
            for i in range(int(self.num_batchs)):
        
                self.optim.zero_grad()  #TO RESET GRADIENTS!
            
                # Indexes of the datapoints that enter the batch
                idx_batch = idx[i*self.batch_size:(i+1)*self.batch_size]

                # generate the outputs for the selected batch of points
                out = self.forward(x[idx_batch,:]) 

                
                loss = self.criterion(out,y[idx_batch].view(self.batch_size,1)) + self.lam * torch.norm(self.output.weight)

                # accumulate the losses
                running_loss += loss.item()

                #Your code here. Compute gradients
                loss.backward()
                
                #Your code here. Perform one SGD step
                self.optim.step()
                # finish the iterations inside an epoch
                
            self.loss_during_training.append(running_loss/self.num_batchs)

            if(e % self.print_every == 0 or e == int(self.epochs)-1): 

                print("Training loss after %d epochs: %f" 
                      %(e,self.loss_during_training[-1]))

my_LRR = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.5)
my_LRR.train(torch.tensor(x),torch.tensor(y))
my_LRR.error_rate(torch.tensor(x),y)
# trying out different lambdas
# lambda = 0.1
my_LRR_1 = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.1)
my_LRR_1.train(torch.tensor(x),torch.tensor(y))
# lambda = 0.25
my_LRR_25 = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.25)
my_LRR_25.train(torch.tensor(x),torch.tensor(y))
# lambda = 05
my_LRR_5 = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.5)
my_LRR_5.train(torch.tensor(x),torch.tensor(y))
# lambda = 0.75
my_LRR_75 = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.75)
my_LRR_75.train(torch.tensor(x),torch.tensor(y))
# lambda = 1
my_LRR_11 = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.11)
my_LRR_11.train(torch.tensor(x),torch.tensor(y))
plt.plot(my_LRR_1.loss_during_training, '-b', label='lambda: 0.1')
plt.plot(my_LRR_25.loss_during_training, '-g', label='lambda: 0.25')
plt.plot(my_LRR_5.loss_during_training, '-y', label='lambda: 0.5')
plt.plot(my_LRR_75.loss_during_training, '-r', label='lambda: 0.75')
plt.plot(my_LRR_11.loss_during_training, '-v', label='lambda: 1')
plt.legend()
plt.show()
# testing error:
# this is the final testing accuracy rate of the model
my_LRR_1.error_rate(torch.tensor(xtest), ytest)
sum(((my_LRR_1.forward(torch.tensor(xtest)).detach().numpy()) >= 0.5) == ytest.reshape([-1,1]))
len(ytest.reshape([-1,1]))
# if we try the testing accuracy rate of the least performing model, we get: 
my_LRR_75.error_rate(torch.tensor(xtest), ytest)
# we see that its accuracy is indeed lower than the one obtained with the lambda=1 model
my_LRR_1 = LR_stochastic_reg(x.shape[1], x.shape[0], lam=0.1)
my_LRR_1.train(torch.tensor(x),torch.tensor(y))
my_LRR_1.accuracy<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09930906871911971, 'fraction_numerical': 0.036160705497923266, 'mean_word_length': 3.771036814425244, 'pattern_counts': {'":': 0, '<': 185, '<?xml version=': 0, '>': 222, 'https://': 18, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '49', 'dir': 'jupyter-structured-clean-dedup', 'id': '637461', 'license': 'Apache-2.0', 'n_tokens_mistral': 16713, 'n_tokens_neox': 15052, 'n_words': 5409, 'path': 'Deep_Learning/Intro_Pytorch/JijonVorbeck_Lab_1_Pytorch_Log_Regression.ipynb', 'repo_name': 'Joseantonio-96/BigData_Uni', 'size': '50822'}