<jupyter_start><jupyter_text>Homework-1: Softmax for MNIST Classification In this homework, you need to-  implement and apply a softmax classifier to perform digits classification.<jupyter_code>import os,sys
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load MNIST DatasetThe **mnist_data_loader.py** is a script to load mnist dataset, which is included in [TensorFlow tutorial](https://github.com/tensorflow/tensorflow/blob/r0.7/tensorflow/examples/tutorials/mnist/input_data.py).<jupyter_code>import mnist_data_loader

mnist_dataset = mnist_data_loader.read_data_sets("./MNIST_data/", one_hot=True)

# training dataset
train_set = mnist_dataset.train 
# test dataset
test_set = mnist_dataset.test   

train_size = train_set.num_examples
test_size = test_set.num_examples
print()
print('Training dataset size: ', train_size)
print('Test dataset size: ', test_size)<jupyter_output>c:\users\primu\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
c:\users\primu\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
c:\users\primu\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
c:\users\primu\appd[...]<jupyter_text>Data ExampleTo have a better understand of MNIST dataset, we can visualize some examples in MNIST.<jupyter_code># You can choose any image in training set to visualize, example_id ∈ [0,60000)
example_id = 0 

image = train_set.images[example_id] # shape = 784 (28*28)
label = train_set.labels[example_id] # shape = 10
#print(image)
print(label)
plt.imshow(np.reshape(image,[28,28]))<jupyter_output>[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]<jupyter_text>Set HyerparametersYou can modify hyerparameters by yourself.<jupyter_code>batch_size = 100
max_epoch = 10
learning_rate = 0.01

# For regularization
lamda = 0.5<jupyter_output><empty_output><jupyter_text>TODOBefore executing the following code, you should accomplish **./softmax_classifier.py**.<jupyter_code>from softmax_classifier import softmax_classifier<jupyter_output><empty_output><jupyter_text>Train<jupyter_code># Weight Initialization
W = np.random.randn(28*28, 10) * 0.001

loss_set = []
accu_set = []
disp_freq = 100

# Training process
for epoch in range(0, max_epoch):
    iter_per_batch = train_size // batch_size
    for batch_id in range(0, iter_per_batch):
        batch = train_set.next_batch(batch_size) # get data of next batch
        input, label = batch
        
        # softmax_classifier
        loss, gradient, prediction = softmax_classifier(W, input , label, lamda)
        
        # Calculate accuracy
        label = np.argmax(label, axis=1) # scalar representation
        accuracy = sum(prediction == label) / float(len(label))
        
        loss_set.append(loss)
        accu_set.append(accuracy)
        
        # Update weights
        W = W - (learning_rate * gradient)
        if batch_id % disp_freq == 0:
            print("Epoch [{}][{}]\t Batch [{}][{}]\t Training Loss {:.4f}\t Accuracy {:.4f}".format(
                epoch, max_epoch, batch_id, iter_per_batch, 
                loss, accuracy))
    print()<jupyter_output>Epoch [0][10]	 Batch [0][600]	 Training Loss 12.3700	 Accuracy 0.0400
Epoch [0][10]	 Batch [100][600]	 Training Loss 5.4600	 Accuracy 0.7800
Epoch [0][10]	 Batch [200][600]	 Training Loss 4.1800	 Accuracy 0.8000
Epoch [0][10]	 Batch [300][600]	 Training Loss 3.2000	 Accuracy 0.8000
Epoch [0][10]	 Batch [400][600]	 Training Loss 2.8100	 Accuracy 0.8800
Epoch [0][10]	 Batch [500][600]	 Training Loss 2.2100	 Accuracy 0.8600

Epoch [1][10]	 Batch [0][600]	 Training Loss 2.9600	 Accuracy 0.8200
Epoch [1][10]	 Batch [100][600]	 Training Loss 3.2500	 Accuracy 0.8700
Epoch [1][10]	 Batch [200][600]	 Training Loss 4.6300	 Accuracy 0.7700
Epoch [1][10]	 Batch [300][600]	 Training Loss 3.3900	 Accuracy 0.8100
Epoch [1][10]	 Batch [400][600]	 Training Loss 1.9300	 Accuracy 0.8500
Epoch [1][10]	 Batch [500][600]	 Training Loss 3.2100	 Accuracy 0.8600

Epoch [2][10]	 Batch [0][600]	 Training Loss 3.6000	 Accuracy 0.8200
Epoch [2][10]	 Batch [100][600]	 Training Loss 2.4000	 Accuracy 0.8300
Epoch [2][...]<jupyter_text>Test<jupyter_code>correct = 0
iter_per_batch = test_size // batch_size

# Test process
for batch_id in range(0, iter_per_batch):
    batch = test_set.next_batch(batch_size)
    data, label = batch
    
    # We only need prediction results in testing
    _,_, prediction = softmax_classifier(W, data , label, lamda)
    label = np.argmax(label, axis=1)
    
    correct += sum(prediction == label)
    
accuracy = correct * 1.0 / test_size
print('Test Accuracy: ', accuracy)<jupyter_output>Test Accuracy:  0.8575<jupyter_text>Plot<jupyter_code># training loss curve
plt.figure()
plt.plot(loss_set, 'b--')
plt.xlabel('iteration')
plt.ylabel('loss')
# training accuracy curve
plt.figure()
plt.plot(accu_set, 'r--')
plt.xlabel('iteration')
plt.ylabel('accuracy');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12647110486562446, 'fraction_numerical': 0.0660460214298261, 'mean_word_length': 4.371698113207547, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '44016', 'license': 'Apache-2.0', 'n_tokens_mistral': 2310, 'n_tokens_neox': 2033, 'n_words': 514, 'path': 'HW1/homework1-softmax.ipynb', 'repo_name': 'primus2019/Deep-Learning-Assignments', 'size': '5693'}