<jupyter_start><jupyter_text>Practice: Basic Artificial Neural NetworksCredits: this notebook belongs to [Practical DL](https://docs.google.com/forms/d/e/1FAIpQLScvrVtuwrHSlxWqHnLt1V-_7h2eON_mlRR6MUb3xEe5x9LuoA/viewform?usp=sf_link) course by Yandex School of Data Analysis.We will start working with neural networks on the practice session. Your homework will be to finish the implementation of the layers. Our goal is simple, yet an actual implementation may take some time :). We are going to write an Artificial Neural Network (almost) from scratch. The software design was heavily inspired by [PyTorch](http://pytorch.org) which is the main framework of our course<jupyter_code>%matplotlib inline
from time import time, sleep
import numpy as np
import matplotlib.pyplot as plt
from IPython import display<jupyter_output><empty_output><jupyter_text>Framework Implement everything in `modules.ipynb`. Read all the comments thoughtfully to ease the pain. Please try not to change the prototypes.Do not forget, that each module should return **AND** store `output` and `gradInput`.The typical assumption is that `module.backward` is always executed after `module.forward`,so `output` is stored, this would be useful for `SoftMax`.   Tech notePrefer using `np.multiply`, `np.add`, `np.divide`, `np.subtract` instead of `*`,`+`,`/`,`-` for better memory handling.Example: suppose you allocated a variable ```a = np.zeros(...)```So, instead of```a = b + c   will be reallocated, GC needed to free``` You can use: ```np.add(b,c,out = a)  puts result in `a````<jupyter_code>%run modules.ipynb
import mnist<jupyter_output><empty_output><jupyter_text>Toy example Use this example to debug your code, start with logistic regression and then test other layers. You do not need to change anything here. This code is provided for you to test the layers. Also it is easy to use this code in MNIST task.<jupyter_code># Generate some data
N = 500

X1 = np.random.randn(N,2) + np.array([2,2])
X2 = np.random.randn(N,2) + np.array([-2,-2])

Y = np.concatenate([np.ones(N),np.zeros(N)])[:,None]
Y = np.hstack([Y, 1-Y])

X = np.vstack([X1,X2])
plt.scatter(X[:,0],X[:,1], c = Y[:,0], edgecolors= 'none')<jupyter_output><empty_output><jupyter_text>Define a **logistic regression** for debugging.<jupyter_code>net = Sequential()
net.add(Linear(2, 2))
net.add(LogSoftMax())

criterion = ClassNLLCriterion()

print(net)

# Test something like that then 

# net = Sequential()
# net.add(Linear(2, 4))
# net.add(ReLU())
# net.add(Linear(4, 2))
# net.add(LogSoftMax())<jupyter_output>Linear 2 -> 2
LogSoftMax<jupyter_text>Start with batch_size = 1000 to make sure every step lowers the loss, then try stochastic version.<jupyter_code># Iptimizer params
optimizer_config = {'learning_rate' : 1e-1, 'momentum': 0.9}
optimizer_state = {}

# Looping params
n_epoch = 20
batch_size = 128
# batch generator
def get_batches(dataset, batch_size):
    X, Y = dataset
    n_samples = X.shape[0]
        
    # Shuffle at the start of epoch
    indices = np.arange(n_samples)
    np.random.shuffle(indices)
    
    for start in range(0, n_samples, batch_size):
        end = min(start + batch_size, n_samples)
        
        batch_idx = indices[start:end]
    
        yield X[batch_idx], Y[batch_idx]<jupyter_output><empty_output><jupyter_text>Train Basic training loop. Examine it.<jupyter_code>loss_history = []

for i in range(n_epoch):
    for x_batch, y_batch in get_batches((X, Y), batch_size):

        net.zeroGradParameters()

        # Forward
        predictions = net.forward(x_batch)
        loss = criterion.forward(predictions, y_batch)

        # Backward
        dp = criterion.backward(predictions, y_batch)
        net.backward(x_batch, dp)

        # Update weights
        sgd_momentum(net.getParameters(), 
                     net.getGradParameters(), 
                     optimizer_config,
                     optimizer_state)      

        loss_history.append(loss)

    # Visualize
    display.clear_output(wait=True)
    plt.figure(figsize=(8, 6))

    plt.title("Training loss")
    plt.xlabel("#iteration")
    plt.ylabel("loss")
    plt.plot(loss_history, 'b')
    plt.show()

    print('Current loss: %f' % loss)<jupyter_output><empty_output><jupyter_text>Digit classification  We are using old good [MNIST](http://yann.lecun.com/exdb/mnist/) as our dataset.<jupyter_code>import mnist
X_train, y_train, X_val, y_val, X_test, y_test = mnist.load_dataset()<jupyter_output><empty_output><jupyter_text>One-hot encode the labels first.<jupyter_code>plt.imshow(X_train[0], cmap='gray')
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()

y_train_onehot = encoder.fit_transform(y_train.reshape(-1, 1)).toarray()
y_test_onehot = encoder.fit_transform(y_test.reshape(-1, 1)).toarray()
y_val_onehot = encoder.fit_transform(y_val.reshape(-1, 1)).toarray()
X_train.shape, y_train_onehot.shape
X_train = X_train.reshape(X_train.shape[0], -1)
X_val = X_val.reshape(X_val.shape[0], -1)
X_test = X_test.reshape(X_test.shape[0], -1)<jupyter_output><empty_output><jupyter_text>- **Compare** `ReLU`, `ELU`, `LeakyReLU`, `SoftPlus` activation functions. You would better pick the best optimizer params for each of them, but it is overkill for now. Use an architecture of your choice for the comparison.- **Try** inserting `BatchNormalization` (folowed by `ChannelwiseScaling`) between `Linear` module and activation functions.- Plot the losses both from activation functions comparison and `BatchNormalization` comparison on one plot. Please find a scale (log?) when the lines are distinguishable, do not forget about naming the axes, the plot should be goodlooking.- Plot the losses for two networks: one trained by momentum_sgd, another one trained by Adam. Which one performs better?- Hint: good logloss for MNIST should be around 0.5.<jupyter_code># Your code goes here. ################################################<jupyter_output><empty_output><jupyter_text>`ReLU`, `ELU`, `LeakyReLU`, `SoftPlus` comparison<jupyter_code># Iptimizer params
optimizer_config = {'learning_rate' : 1e-1, 'momentum': 0.9}
optimizer_state = {}

# Looping params
n_epoch = 20
batch_size = 128
%run modules.ipynb
X_train.shape, y_train_onehot.shape
n_epoch = 1
batch_size = 256
relu_loss_history = []
elu_loss_history = []
leaky_relu_loss_history = []
softplus_loss_history = []
def train_loop_compare_activations(act_func, loss_history):
    net = Sequential()
    net.add(Linear(X_train.shape[1], 100))
    net.add(act_func)
    net.add(Linear(100, y_train_onehot.shape[1]))
    net.add(LogSoftMax())
    print(net)


    for i in range(n_epoch):
        for x_batch, y_batch in get_batches((X_train, y_train_onehot), batch_size):

            net.zeroGradParameters()

            # Forward
            predictions = net.forward(x_batch)
            loss = criterion.forward(predictions, y_batch)

            # Backward
            dp = criterion.backward(predictions, y_batch)
            net.backward(x_batch, dp)

            # Update weights
            sgd_momentum(net.getParameters(), 
                         net.getGradParameters(), 
                         optimizer_config,
                         optimizer_state)      

            loss_history.append(loss)

        # Visualize
        display.clear_output(wait=True)
        plt.figure(figsize=(8, 6))

        plt.title("Training loss")
        plt.xlabel("#iteration")
        plt.ylabel("loss")
        plt.plot(loss_history, 'b')
        plt.show()

        print('Current loss: %f' % loss)    
train_loop_compare_activations(ReLU(), relu_loss_history)
train_loop_compare_activations(ELU(), elu_loss_history)
train_loop_compare_activations(LeakyReLU(), leaky_relu_loss_history)
train_loop_compare_activations(SoftPlus(), softplus_loss_history)
plt.title("Training loss")
plt.xlabel("#iteration")
plt.ylabel("loss")

plt.plot(relu_loss_history, 'b', label='relu')
plt.plot(elu_loss_history, 'g', label='elu')
plt.plot(leaky_relu_loss_history, 'r', label='leakyrelu_loss_history')
plt.plot(softplus_loss_history, 'y', label='softplus_loss_histiry')
plt.legend()
plt.show()
# Inserting BatchNorm

relu_loss_history_with_batch_norm = []
net = Sequential()
net.add(Linear(X_train.shape[1], 100))
net.add(ReLU())
net.add(BatchNormalization())
net.add(Linear(100, y_train_onehot.shape[1]))
net.add(LogSoftMax())
print(net)


for i in range(n_epoch):
    for x_batch, y_batch in get_batches((X_train, y_train_onehot), batch_size):

        net.zeroGradParameters()

        # Forward
        predictions = net.forward(x_batch)
        loss = criterion.forward(predictions, y_batch)

        # Backward
        dp = criterion.backward(predictions, y_batch)
        net.backward(x_batch, dp)

        # Update weights
        sgd_momentum(net.getParameters(), 
                     net.getGradParameters(), 
                     optimizer_config,
                     optimizer_state)      

        relu_loss_history_with_batch_norm.append(loss)

    # Visualize
    display.clear_output(wait=True)
    plt.figure(figsize=(8, 6))

    plt.title("Training loss")
    plt.xlabel("#iteration")
    plt.ylabel("loss")
    plt.plot(loss_history, 'b')
    plt.show()

    print('Current loss: %f' % loss)    
# comparing relu without/with batchnorm

plt.title('training loss')
plt.xlabel('#iteration')
plt.ylabel('loss')
plt.plot(relu_loss_history, 'b', label='without batch norm')
plt.plot(relu_loss_history_with_batch_norm, 'g', label='with batchnorm')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Write your personal opinion on the activation functions, think about computation times too. Does `BatchNormalization` help?<jupyter_code># Your answer goes here. ################################################
BatchNormalization does _help<jupyter_output><empty_output><jupyter_text>**Finally**, use all your knowledge to build a super cool model on this dataset. Use **dropout** to prevent overfitting, play with **learning rate decay**. You can use **data augmentation** such as rotations, translations to boost your score. Use your knowledge and imagination to train a model. Don't forget to call `training()` and `evaluate()` methods to set desired behaviour of `BatchNormalization` and `Dropout` layers.<jupyter_code># Iptimizer params
optimizer_config = {'learning_rate' : 1e-1, 'momentum': 0.9}
optimizer_state = {}

# Looping params
n_epoch = 20
batch_size = 128
# Your code goes here. ################################################

net = Sequential()
net.add(Linear(X_train.shape[1], 256))
net.add(BatchNormalization())
net.add(LeakyReLU())
net.add(Dropout())
net.add(Linear(256, 64))
net.add(BatchNormalization())
net.add(LeakyReLU())
net.add(Linear(64, y_train_onehot.shape[1]))
net.add(LogSoftMax())
print(net)

criterion = ClassNLLCriterion()
n_epoch = 1
batch_size = 256

loss_history = []

for i in range(n_epoch):
    for x_batch, y_batch in get_batches((X_train, y_train_onehot), batch_size):

        net.zeroGradParameters()

        # Forward
        predictions = net.forward(x_batch)
        loss = criterion.forward(predictions, y_batch)

        # Backward
        dp = criterion.backward(predictions, y_batch)
        net.backward(x_batch, dp)

        # Update weights
        sgd_momentum(net.getParameters(), 
                     net.getGradParameters(), 
                     optimizer_config,
                     optimizer_state)      

        loss_history.append(loss)

    # Visualize
    display.clear_output(wait=True)
    plt.figure(figsize=(8, 6))

    plt.title("Training loss")
    plt.xlabel("#iteration")
    plt.ylabel("loss")
    plt.plot(loss_history, 'b')
    plt.show()

    print('Current loss: %f' % loss)<jupyter_output><empty_output><jupyter_text>Print here your accuracy on test set. It should be around 90%.<jupyter_code># Your answer goes here. ################################################
net.evaluate()
y_pred_onehot = net.forward(X_test)
y_pred = encoder.inverse_transform(y_pred_onehot).reshape(-1)
from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred)<jupyter_output><empty_output><jupyter_text>Comparing with PyTorch implementationThe last (and maybe the easiest step after compared to the previous tasks: build a network with the same architecture as above now with PyTorch.You can refer to the `week0_09` or `Lab3_part2` notebooks for hints.__Good Luck!__<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F
# Your beautiful code here.

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.model = nn.Sequential(
            nn.Flatten()
            nn.Linear(784, 256),
            nn.BatchNorm1d(256),
            nn.LeakyReLU(),
            nn.Dropout(),
            nn.Linear(256, 64),
            nn.BatchNorm1d(64),
            nn.LeakyReLU(),
            nn.Linear(64, 10),
            nn.LogSoftmax()
        )

    def forward(self, inp):
        out = self.model(inp)
        return out
mnist_net = Net()
loss = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(mnist_net.parameters(), lr=1.0e-3)
n_epoch = 1
batch_size = 128
X_train, y_train, X_val, y_val, X_test, y_test = mnist.load_dataset()
pytorch_loss_history = []

for epoch in range(n_epoch):
    for x_train, y_train in get_batches((X_train, y_train), batch_size):
        optimizer.zero_grad()

        x_train_tensor = torch.Tensor(x_train)
        y_train_tensor = torch.Tensor(y_train)
        predictions = mnist_net.forward(x_train_tensor)

        loss_value = loss(predictions, y_train_tensor)
        loss_value.backward()
        pytorch_loss_history.append(loss_value)

        optimizer.step()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12721850450974687, 'fraction_numerical': 0.011783532150130928, 'mean_word_length': 3.9474631162288594, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '375687', 'license': 'MIT', 'n_tokens_mistral': 4678, 'n_tokens_neox': 4426, 'n_words': 1076, 'path': 'Lab2_DL/main_notebook.ipynb', 'repo_name': 'YanLo/ml-mipt', 'size': '13748'}