<jupyter_start><jupyter_text>Lab 2 - Logistic Regression (LR) with MNISTThis lab corresponds to Module 2 of the "Deep Learning Explained" course. We assume that you have successfully completed Lab 1 (Downloading the MNIST data).In this lab we will build and train a Multiclass Logistic Regression model using the MNIST data.  Introduction**Problem**:Optical Character Recognition (OCR) is a hot research area and there is a great demand for automation. The MNIST data is comprised of hand-written digits with little background noise making it a nice dataset to create, experiment and learn deep learning models with reasonably small comptuing resources. **Goal**:Our goal is to train a classifier that will identify the digits in the MNIST dataset. **Approach**:There are 4 stages in this lab: - **Data reading**: We will use the CNTK Text reader.  - **Data preprocessing**: Covered in part A (suggested extension section). - **Model creation**: Multiclass Logistic Regression model.- **Train-Test-Predict**: This is the same workflow introduced in the lectures  Logistic Regression[Logistic Regression](https://en.wikipedia.org/wiki/Logistic_regression) (LR) is a fundamental machine learning technique that uses a linear weighted combination of features and generates probability-based predictions of different classes.  There are two basic forms of LR: **Binary LR** (with a single output that can predict two classes) and **multiclass LR** (with multiple outputs, each of which is used to predict a single class).  ![LR-forms](http://www.cntk.ai/jup/cntk103b_TwoFormsOfLR-v3.png) In **Binary Logistic Regression** (see top of figure above), the input features are each scaled by an associated weight and summed together.  The sum is passed through a squashing (aka activation) function and generates an output in [0,1].  This output value is then compared with a threshold (such as 0.5) to produce a binary label (0 or 1), predicting 1 of 2 classes.  This technique supports only classification problems with two output classes, hence the name binary LR.  In the binary LR example shown above, the [sigmoid][] function is used as the squashing function.[sigmoid]: https://en.wikipedia.org/wiki/Sigmoid_function In **Multiclass Linear Regression** (see bottom of figure above), 2 or more output nodes are used, one for each output class to be predicted.  Each summation node uses its own set of weights to scale the input features and sum them together. Instead of passing the summed output of the weighted input features through a sigmoid squashing function, the output is often passed through a [softmax][] function (which in addition to squashing, like the sigmoid, the softmax normalizes each nodes' output value using the sum of all unnormalized nodes). (Details in the context of MNIST image to follow)We will use multiclass LR for classifying the MNIST digits (0-9) using 10 output nodes (1 for each of our output classes).  In our approach, we will move the softmax function out of the model and into our Loss function used in training (details to follow).[softmax]: https://en.wikipedia.org/wiki/Softmax_function<jupyter_code># Import the relevant components
from IPython.display import Image
from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
import sys
import os

import cntk as C

%matplotlib inline<jupyter_output><empty_output><jupyter_text>In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU).<jupyter_code># Select the right target device when this notebook is being tested:
if 'TEST_DEVICE' in os.environ:
    if os.environ['TEST_DEVICE'] == 'cpu':
        C.device.try_set_default_device(C.device.cpu())
    else:
        C.device.try_set_default_device(C.device.gpu(0))
# Test for CNTK version
#if not C.__version__ == "2.0":
#    raise Exception("this lab is designed to work with 2.0. Current Version: " + C.__version__)<jupyter_output><empty_output><jupyter_text>Initialization<jupyter_code># Ensure we always get the same amount of randomness
np.random.seed(0)
C.cntk_py.set_fixed_random_seed(1)
C.cntk_py.force_deterministic_algorithms()

# Define the data dimensions
input_dim = 256  #784
num_output_classes = 11 #10<jupyter_output><empty_output><jupyter_text>Data readingThere are different ways one can read data into CNTK. The easiest way is to load the data in memory using NumPy / SciPy / Pandas readers. However, this can be done only for small data sets. Since deep learning requires large amount of data we have chosen in this course to show how to leverage built-in distributed readers that can scale to terrabytes of data with little extra effort. We are using the MNIST data you have downloaded using Lab 1 DataLoader notebook. The dataset has 60,000 training images and 10,000 test images with each image being 28 x 28 pixels. Thus the number of features is equal to 784 (= 28 x 28 pixels), 1 per pixel. The variable `num_output_classes` is set to 10 corresponding to the number of digits (0-9) in the dataset.In Lab 1, the data was downloaded and written to 2 CTF (CNTK Text Format) files, 1 for training, and 1 for testing. Each line of these text files takes the form:    |labels 0 0 0 1 0 0 0 0 0 0 |features 0 0 0 0 ...                                                   (784 integers each representing a pixel)    We are going to use the image pixels corresponding the integer stream named "features". We define a `create_reader` function to read the training and test data using the [CTF deserializer](https://cntk.ai/pythondocs/cntk.io.html?highlight=ctfdeserializercntk.io.CTFDeserializer). The labels are [1-hot encoded](https://en.wikipedia.org/wiki/One-hot). Refer to Lab 1 for data format visualizations.<jupyter_code># Read a CTF formatted text (as mentioned above) using the CTF deserializer from a file
def create_reader(path, is_training, input_dim, num_label_classes):
    
    labelStream = C.io.StreamDef(field='labels', shape=num_label_classes, is_sparse=False)
    featureStream = C.io.StreamDef(field='features', shape=input_dim, is_sparse=False)
    
    deserailizer = C.io.CTFDeserializer(path, C.io.StreamDefs(labels = labelStream, features = featureStream))
            
    return C.io.MinibatchSource(deserailizer,
       randomize = is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)
# Ensure the training and test data is generated and available for this lab.
# We search in two locations in the toolkit for the cached MNIST data set.
data_found = False

for data_dir in [os.path.join(".", "PLAID")]:
    train_file = os.path.join(data_dir, "train_log.txt")
    test_file = os.path.join(data_dir, "test_log.txt")
    if os.path.isfile(train_file) and os.path.isfile(test_file):
        data_found = True
        break
        
if not data_found:
    raise ValueError("Please generate the data by completing Lab1_MNIST_DataLoader")
    
print("Data directory is {0}".format(data_dir))<jupyter_output>Data directory is ./PLAID<jupyter_text>Model CreationA multiclass logistic regression (LR) network is a simple building block that has been effectively powering many ML applications in the past decade. The figure below summarizes the model in the context of the MNIST data.![mnist-LR](https://www.cntk.ai/jup/cntk103b_MNIST_LR.png)LR is a simple linear model that takes as input, a vector of numbers describing the properties of what we are classifying (also known as a feature vector, $\bf \vec{x}$, the pixels in the input MNIST digit image) and emits the *evidence* ($z$). For each of the 10 digits, there is a vector of weights corresponding to the input pixels as show in the figure. These 10 weight vectors define the weight matrix ($\bf {W}$) with dimension of 10 x 784.  Each feature in the input layer is connected with a summation node by a corresponding weight $w$ (individual weight values from the $\bf{W}$ matrix). Note there are 10 such nodes, 1 corresponding to each digit to be classified.  The first step is to compute the evidence for an observation. $$\vec{z} = \textbf{W} \bf \vec{x}^T + \vec{b}$$ where $\bf{W}$ is the weight matrix of dimension 10 x 784 and $\vec{b}$ is known as the *bias* vector with lenght 10, one for each digit. The evidence ($\vec{z}$) is not squashed (hence no activation). Instead the output is normalized using a [softmax](https://en.wikipedia.org/wiki/Softmax_function) function such that all the outputs add up to a value of 1, thus lending a probabilistic iterpretation to the prediction. In CNTK, we use the softmax operation combined with the cross entropy error as our Loss Function for training.<jupyter_code>#2nd submission, we use 2, 400 respectively
#3rd submission, we use 4, 100 respectively
#4rd submission, we use 4, 200 respectively

num_hidden_layers = 8
hidden_layers_dim = 400<jupyter_output><empty_output><jupyter_text>Network input and output: - **input** variable (a key CNTK concept): >An **input** variable is a container in which we fill different observations, in this case image pixels, during model learning (a.k.a.training) and model evaluation (a.k.a. testing). Thus, the shape of the `input` must match the shape of the data that will be provided.  For example, when data are images each of  height 10 pixels  and width 5 pixels, the input feature dimension will be 50 (representing the total number of image pixels).**Knowledge Check:** What is the input dimension of your chosen model? This is fundamental to our understanding of variables in a network or model representation in CNTK.<jupyter_code>input = C.input_variable(input_dim)
label = C.input_variable(num_output_classes)<jupyter_output><empty_output><jupyter_text>Logistic Regression network setupThe CNTK Layers module provides a Dense function that creates a fully connected layer which performs the above operations of weighted input summing and bias addition.<jupyter_code>def create_model(features):
    with C.layers.default_options(init = C.layers.glorot_uniform(), activation = C.ops.relu):
            h = features
            for _ in range(num_hidden_layers):
                h = C.layers.Dense(hidden_layers_dim)(h)
            r = C.layers.Dense(num_output_classes, activation = None)(h)
            return r<jupyter_output><empty_output><jupyter_text>`z` will be used to represent the output of a network.<jupyter_code># Scale the input to 0-1 range by dividing each pixel by 255.
#input_s = input/255
z = create_model(input)<jupyter_output><empty_output><jupyter_text>TrainingBelow, we define the **Loss** function, which is used to guide weight changes during training.  As explained in the lectures, we use the `softmax` function to map the accumulated evidences or activations to a probability distribution over the classes (Details of the [softmax function][] and other [activation][] functions).[softmax function]: http://cntk.ai/pythondocs/cntk.ops.htmlcntk.ops.softmax[activation]: https://github.com/Microsoft/CNTK/wiki/Activation-FunctionsWe minimize the cross-entropy between the label and predicted probability by the network.<jupyter_code>loss = C.cross_entropy_with_softmax(z, label)<jupyter_output><empty_output><jupyter_text>EvaluationBelow, we define the **Evaluation** (or metric) function that is used to report a measurement of how well our model is performing.For this problem, we choose the **classification_error()** function as our metric, which returns the average error over the associated samples (treating a match as "1", where the model's prediction matches the "ground truth" label, and a non-match as "0").<jupyter_code>label_error = C.classification_error(z, label)<jupyter_output><empty_output><jupyter_text>Configure trainingThe trainer strives to reduce the `loss` function by different optimization approaches, [Stochastic Gradient Descent][] (`sgd`) being one of the most popular. Typically, one would start with random initialization of the model parameters. The `sgd` optimizer would calculate the `loss` or error between the predicted label against the corresponding ground-truth label and using [gradient-decent][] generate a new set model parameters in a single iteration. The aforementioned model parameter update using a single observation at a time is attractive since it does not require the entire data set (all observation) to be loaded in memory and also requires gradient computation over fewer datapoints, thus allowing for training on large data sets. However, the updates generated using a single observation sample at a time can vary wildly between iterations. An intermediate ground is to load a small set of observations and use an average of the `loss` or error from that set to update the model parameters. This subset is called a *minibatch*.With minibatches, we sample observations from the larger training dataset. We repeat the process of model parameters update using different combination of training samples and over a period of time minimize the `loss` (and the error metric). When the incremental error rates are no longer changing significantly or after a preset number of maximum minibatches to train, we claim that our model is trained.One of the key optimization parameters is called the `learning_rate`. For now, we can think of it as a scaling factor that modulates how much we change the parameters in any iteration.With this information, we are ready to create our trainer. [optimization]: https://en.wikipedia.org/wiki/Category:Convex_optimization[Stochastic Gradient Descent]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent[gradient-decent]: http://www.statisticsviews.com/details/feature/5722691/Getting-to-the-Bottom-of-Regression-with-Gradient-Descent.html<jupyter_code># Instantiate the trainer object to drive the model training
learning_rate = 0.2
lr_schedule = C.learning_rate_schedule(learning_rate, C.UnitType.minibatch)
learner = C.sgd(z.parameters, lr_schedule)
trainer = C.Trainer(z, (loss, label_error), [learner])<jupyter_output><empty_output><jupyter_text>First let us create some helper functions that will be needed to visualize different functions associated with training.<jupyter_code># Define a utility function to compute the moving average sum.
# A more efficient implementation is possible with np.cumsum() function
def moving_average(a, w=5):
    if len(a) < w:
        return a[:]    # Need to send a copy of the array
    return [val if idx < w else sum(a[(idx-w):idx])/w for idx, val in enumerate(a)]


# Defines a utility that prints the training progress
def print_training_progress(trainer, mb, frequency, verbose=1):
    training_loss = "NA"
    eval_error = "NA"

    if mb%frequency == 0:
        training_loss = trainer.previous_minibatch_loss_average
        eval_error = trainer.previous_minibatch_evaluation_average
        if verbose: 
            print ("Minibatch: {0}, Loss: {1:.4f}, Error: {2:.2f}%".format(mb, training_loss, eval_error*100))
        
    return mb, training_loss, eval_error<jupyter_output><empty_output><jupyter_text>Run the trainerWe are now ready to train our fully connected neural net. We want to decide what data we need to feed into the training engine.In this example, each iteration of the optimizer will work on `minibatch_size` sized samples. We would like to train on all 60000 observations. Additionally we will make multiple passes through the data specified by the variable `num_sweeps_to_train_with`. With these parameters we can proceed with training our simple feed forward network.<jupyter_code># Initialize the parameters for the trainer
minibatch_size = 64
num_samples_per_sweep = 60000
num_sweeps_to_train_with = 10
num_minibatches_to_train = (num_samples_per_sweep * num_sweeps_to_train_with) / minibatch_size
# Create the reader to training data set
reader_train = create_reader(train_file, True, input_dim, num_output_classes)

# Map the data streams to the input and labels.
input_map = {
    label  : reader_train.streams.labels,
    input  : reader_train.streams.features
} 

# Run the trainer on and perform model training
training_progress_output_freq = 500

plotdata = {"batchsize":[], "loss":[], "error":[]}

for i in range(0, int(num_minibatches_to_train)):
    
    # Read a mini batch from the training data file
    data = reader_train.next_minibatch(minibatch_size, input_map = input_map)
    
    trainer.train_minibatch(data)
    batchsize, loss, error = print_training_progress(trainer, i, training_progress_output_freq, verbose=1)
    
    if not (loss == "NA" or error =="NA"):
        plotdata["batchsize"].append(batchsize)
        plotdata["loss"].append(loss)
        plotdata["error"].append(error)<jupyter_output>Minibatch: 0, Loss: 2.4005, Error: 98.44%
Minibatch: 500, Loss: 1.8469, Error: 67.19%
Minibatch: 1000, Loss: 1.1192, Error: 39.06%
Minibatch: 1500, Loss: 0.8105, Error: 29.69%
Minibatch: 2000, Loss: 0.8246, Error: 31.25%
Minibatch: 2500, Loss: 0.5015, Error: 18.75%
Minibatch: 3000, Loss: 0.5727, Error: 25.00%
Minibatch: 3500, Loss: 0.4437, Error: 17.19%
Minibatch: 4000, Loss: 0.0690, Error: 1.56%
Minibatch: 4500, Loss: 0.3484, Error: 10.94%
Minibatch: 5000, Loss: 0.1216, Error: 1.56%
Minibatch: 5500, Loss: 0.0283, Error: 1.56%
Minibatch: 6000, Loss: 0.1454, Error: 4.69%
Minibatch: 6500, Loss: 0.0140, Error: 0.00%
Minibatch: 7000, Loss: 0.0305, Error: 1.56%
Minibatch: 7500, Loss: 0.0108, Error: 0.00%
Minibatch: 8000, Loss: 0.0221, Error: 1.56%
Minibatch: 8500, Loss: 0.0171, Error: 0.00%
Minibatch: 9000, Loss: 0.0160, Error: 1.56%<jupyter_text>Let us plot the errors over the different training minibatches. Note that as we progress in our training, the loss decreases though we do see some intermediate bumps.<jupyter_code># Compute the moving average loss to smooth out the noise in SGD
plotdata["avgloss"] = moving_average(plotdata["loss"])
plotdata["avgerror"] = moving_average(plotdata["error"])

# Plot the training loss and the training error
import matplotlib.pyplot as plt

plt.figure(1)
plt.subplot(211)
plt.plot(plotdata["batchsize"], plotdata["avgloss"], 'b--')
plt.xlabel('Minibatch number')
plt.ylabel('Loss')
plt.title('Minibatch run vs. Training loss')

plt.show()

plt.subplot(212)
plt.plot(plotdata["batchsize"], plotdata["avgerror"], 'r--')
plt.xlabel('Minibatch number')
plt.ylabel('Label Prediction Error')
plt.title('Minibatch run vs. Label Prediction Error')
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluation / Testing Now that we have trained the network, let us evaluate the trained network on the test data. This is done using `trainer.test_minibatch`.<jupyter_code>out = C.softmax(z)
# Read the data for evaluation
reader_eval = create_reader(test_file, False, input_dim, num_output_classes)

eval_minibatch_size = 1
eval_input_map = {input: reader_eval.streams.features} 

num_samples = 659
num_minibatches_to_test = num_samples // eval_minibatch_size
test_result = 0.0

results=[]

for i in range(num_minibatches_to_test):
    data = reader_eval.next_minibatch(eval_minibatch_size, input_map = eval_input_map)

    #img_label = data[label].asarray()
    img_data = data[input].asarray()
    predicted_label_prob = [out.eval(img_data[i]) for i in range(len(img_data))]
    pred = [np.argmax(predicted_label_prob[i]) for i in range(len(predicted_label_prob))]
    #print(predicted_label_prob)
    results.extend(pred )

#print(results)
np.savetxt(str(num_hidden_layers)+"x"+str(hidden_layers_dim)+"_log.csv", np.array(results).astype(int), fmt='%i',  delimiter=",")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0816439737771054, 'fraction_numerical': 0.022037317196167423, 'mean_word_length': 4.916169451073985, 'pattern_counts': {'":': 4, '<': 61, '<?xml version=': 0, '>': 60, 'https://': 10, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '11349', 'license': 'MIT', 'n_tokens_mistral': 6083, 'n_tokens_neox': 5580, 'n_words': 2377, 'path': 'LogisticRegression multi layers.ipynb', 'repo_name': 'MRYingLEE/Deep-Learning-Project', 'size': '19830'}