<jupyter_start><jupyter_text>Four tasks*  Implement forward and backward for FCLayer (`layers/fc_layer.py`)*  Implement forward and backward for ReLULayer (`layers/relu_layer.py`)*  Implement forward and backward for SigmoidLayer (`layers/sigmoid_layer.py`)*  Implement sgd with momentum (`optimizer.py`)<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from tensorflow.examples.tutorials.mnist import input_data
from network import Network

from layers import FCLayer, ReLULayer, SigmoidLayer

from criterion import SoftmaxCrossEntropy
from optimizer import SGD, SGDwithMomentum

from solver import train, test
from plot import plot_loss_and_acc
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Set Hyperparameters<jupyter_code>batch_size = 100
max_epoch = 20
init_std = 0.01

learning_rate = 0.001
weight_decay = 0.005
momentum = 0.9

disp_freq = 50<jupyter_output><empty_output><jupyter_text>Dataset<jupyter_code>mnist = input_data.read_data_sets("MNIST_data/", validation_size=5000, one_hot=True)<jupyter_output>Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz<jupyter_text>Criterion and Optimizer<jupyter_code>criterion = SoftmaxCrossEntropy()
sgd = SGD(learning_rate, weight_decay)
sgd_momentum = SGDwithMomentum(learning_rate, weight_decay, momentum)<jupyter_output><empty_output><jupyter_text>MLP with sigmoid activation function Build and train a MLP with one hidden layer with 256 units using sigmoid activation function and crossentropy loss Before executing the following code, you should implement FCLayer and SigmoidLayer.  Build the model (see Framework_demo.ipynb)<jupyter_code>SigmoidMLP = Network()
SigmoidMLP.add(FCLayer(784, 256, init_std))
SigmoidMLP.add(SigmoidLayer())
SigmoidMLP.add(FCLayer(256, 10, init_std))<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>SigmoidMLP, sigmoid_loss, sigmoid_acc = train(SigmoidMLP, criterion, sgd, mnist, max_epoch, batch_size, disp_freq)<jupyter_output>Epoch [0][20]	 Batch [0][550]	 Training Loss 2.3180	 Accuracy 0.0700
Epoch [0][20]	 Batch [50][550]	 Training Loss 2.3142	 Accuracy 0.1075
Epoch [0][20]	 Batch [100][550]	 Training Loss 2.3020	 Accuracy 0.1214
Epoch [0][20]	 Batch [150][550]	 Training Loss 2.2708	 Accuracy 0.1579
Epoch [0][20]	 Batch [200][550]	 Training Loss 2.2065	 Accuracy 0.2236
Epoch [0][20]	 Batch [250][550]	 Training Loss 2.1052	 Accuracy 0.2882
Epoch [0][20]	 Batch [300][550]	 Training Loss 1.9868	 Accuracy 0.3459
Epoch [0][20]	 Batch [350][550]	 Training Loss 1.8699	 Accuracy 0.3934
Epoch [0][20]	 Batch [400][550]	 Training Loss 1.7633	 Accuracy 0.4355
Epoch [0][20]	 Batch [450][550]	 Training Loss 1.6633	 Accuracy 0.4743
Epoch [0][20]	 Batch [500][550]	 Training Loss 1.5762	 Accuracy 0.5065
Epoch [0]	 Average training loss 1.5004	 Average training accuracy 0.5335
Epoch [0]	 Average validation loss 0.6880	 Average validation accuracy 0.8204

Epoch [1][20]	 Batch [0][550]	 Training Loss 0.7556	 Accuracy 0.7700
[...]<jupyter_text>Test<jupyter_code>test(SigmoidMLP, criterion, mnist, batch_size, disp_freq)<jupyter_output>Testing
Batch [0][100]	 Accuracy 0.9700
Batch [50][100]	 Accuracy 0.9237
The test accuracy is 0.9411.<jupyter_text>MLP with ReLU activation function Build and train a MLP with one hidden layer with 256 units using ReLU activation function and crossentropy loss Before executing the following code, you should implement **ReLULayer**.  Build the model<jupyter_code>ReLUMLP = Network()
ReLUMLP.add(FCLayer(784, 256, init_std))
ReLUMLP.add(ReLULayer())
ReLUMLP.add(FCLayer(256, 10, init_std))<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>ReLUMLP, ReLU_loss, ReLU_acc = train(ReLUMLP, criterion, sgd, mnist, max_epoch, batch_size, disp_freq)<jupyter_output>Epoch [0][20]	 Batch [0][550]	 Training Loss 2.3004	 Accuracy 0.1600
Epoch [0][20]	 Batch [50][550]	 Training Loss 2.1622	 Accuracy 0.4380
Epoch [0][20]	 Batch [100][550]	 Training Loss 1.7224	 Accuracy 0.5691
Epoch [0][20]	 Batch [150][550]	 Training Loss 1.3909	 Accuracy 0.6536
Epoch [0][20]	 Batch [200][550]	 Training Loss 1.1824	 Accuracy 0.7040
Epoch [0][20]	 Batch [250][550]	 Training Loss 1.0404	 Accuracy 0.7376
Epoch [0][20]	 Batch [300][550]	 Training Loss 0.9341	 Accuracy 0.7628
Epoch [0][20]	 Batch [350][550]	 Training Loss 0.8544	 Accuracy 0.7813
Epoch [0][20]	 Batch [400][550]	 Training Loss 0.7950	 Accuracy 0.7949
Epoch [0][20]	 Batch [450][550]	 Training Loss 0.7489	 Accuracy 0.8057
Epoch [0][20]	 Batch [500][550]	 Training Loss 0.7070	 Accuracy 0.8162
Epoch [0]	 Average training loss 0.6737	 Average training accuracy 0.8239
Epoch [0]	 Average validation loss 0.3093	 Average validation accuracy 0.9138

Epoch [1][20]	 Batch [0][550]	 Training Loss 0.4103	 Accuracy 0.8800
[...]<jupyter_text>Test<jupyter_code>test(ReLUMLP, criterion, mnist, batch_size, disp_freq)<jupyter_output>Testing
Batch [0][100]	 Accuracy 1.0000
Batch [50][100]	 Accuracy 0.9727
The test accuracy is 0.9796.<jupyter_text>Plot<jupyter_code>plot_loss_and_acc({'Sigmoid': [sigmoid_loss, sigmoid_acc], 'ReLU': [ReLU_loss, ReLU_acc]})<jupyter_output><empty_output><jupyter_text>Sgd with momentum Before executing the following code, you should implement SGDwithMomentum<jupyter_code>ReLUMLP_momentum = Network()

# TODO build ReLUMLP with FCLayer and ReLULayer
ReLUMLP_momentum.add(FCLayer(784, 256, init_std))
ReLUMLP_momentum.add(ReLULayer())
ReLUMLP_momentum.add(FCLayer(256, 10, init_std))<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>ReLUMLP_momentum, momentum_loss, momentum_acc = train(ReLUMLP_momentum, criterion, sgd_momentum, mnist, max_epoch, batch_size, disp_freq)<jupyter_output>Epoch [0][20]	 Batch [0][550]	 Training Loss 2.3023	 Accuracy 0.0600
Epoch [0][20]	 Batch [50][550]	 Training Loss 1.7699	 Accuracy 0.5786
Epoch [0][20]	 Batch [100][550]	 Training Loss 1.1406	 Accuracy 0.7137
Epoch [0][20]	 Batch [150][550]	 Training Loss 0.8908	 Accuracy 0.7708
Epoch [0][20]	 Batch [200][550]	 Training Loss 0.7407	 Accuracy 0.8054
Epoch [0][20]	 Batch [250][550]	 Training Loss 0.6426	 Accuracy 0.8290
Epoch [0][20]	 Batch [300][550]	 Training Loss 0.5703	 Accuracy 0.8472
Epoch [0][20]	 Batch [350][550]	 Training Loss 0.5158	 Accuracy 0.8607
Epoch [0][20]	 Batch [400][550]	 Training Loss 0.4728	 Accuracy 0.8717
Epoch [0][20]	 Batch [450][550]	 Training Loss 0.4374	 Accuracy 0.8811
Epoch [0][20]	 Batch [500][550]	 Training Loss 0.4071	 Accuracy 0.8890
Epoch [0]	 Average training loss 0.3843	 Average training accuracy 0.8949
Epoch [0]	 Average validation loss 0.1183	 Average validation accuracy 0.9670

Epoch [1][20]	 Batch [0][550]	 Training Loss 0.1745	 Accuracy 0.9400
[...]<jupyter_text>Test<jupyter_code>test(ReLUMLP_momentum, criterion, mnist, batch_size, disp_freq)
plot_loss_and_acc({'Sigmoid': [sigmoid_loss, sigmoid_acc],
                   'SGD': [ReLU_loss, ReLU_acc], 
                   'SGD_Momentum': [momentum_loss, momentum_acc]})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11084473275983642, 'fraction_numerical': 0.12043435340572557, 'mean_word_length': 5.898832684824903, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '327463', 'license': 'MIT', 'n_tokens_mistral': 3338, 'n_tokens_neox': 2728, 'n_words': 682, 'path': 'HW1_24072018_mlp/homework-mlp_completed/Problem_completed.ipynb', 'repo_name': 'georgosgeorgos/dlss_tsinghua', 'size': '7091'}