<jupyter_start><jupyter_text>MIT 6.036 Spring 2020: Homework 3**If you haven't already, please hit :**`File` -> `Save a Copy in Drive`**to copy this notebook to your Google drive, and work on a copy. If you don't do this, your changes won't be saved!**---This colab notebook provides code and a framework for problems 1-7 of the [ homework](https://lms.mitx.mit.edu/courses/course-v1:MITx+6.036+2019_Fall/courseware/Week3/week3_homework/).  You can work out your solutions here, then submit your results back on the homework page when ready. **Setup**First, download the code distribution for this homework that contains test cases and helper functions.Run the next code block to download and import the code for this lab.<jupyter_code>!rm -rf code_and_data_for_hw3*
!rm -rf mnist
!wget --no-check-certificate --quiet https://introml.odl.mit.edu/cat-soop/_static/6.036/homework/hw03/code_and_data_for_hw3.zip
!unzip code_and_data_for_hw3.zip

from code_for_hw3_part1 import *
import code_for_hw3_part2 as hw3
import numpy as np
import tqdm<jupyter_output>Archive:  code_and_data_for_hw3.zip
   creating: mnist/
  inflating: mnist/mnist_train0.png  
  inflating: mnist/mnist_train1.png  
  inflating: mnist/mnist_train2.png  
  inflating: mnist/mnist_train3.png  
  inflating: mnist/mnist_train4.png  
  inflating: mnist/mnist_train5.png  
  inflating: mnist/mnist_train6.png  
  inflating: mnist/mnist_train7.png  
  inflating: mnist/mnist_train8.png  
  inflating: mnist/mnist_train9.png  
  inflating: auto-mpg.tsv            
  inflating: code_for_hw3_part1.py   
  inflating: code_for_hw3_part2.py   
  inflating: hw3_part2_main.py       
  inflating: reviews.tsv             
  inflating: stopwords.txt           
Importing code_for_hw03
Imported tidy_plot, plot_separator, plot_data, plot_nonlin_sep, cv, rv, y, positive, score
Datasets: super_simple_separable_through_origin(), super_simple_separable(), xor(), xor_more()
Tests for part 2: test_linear_classifier_with_features, mul, make_polynomial_feature_fun, 
                  test_with_feature[...]<jupyter_text>Feature Transformation **Running Perceptron**In problems 1,2 and 3, you will have to run the Perceptron algorithm several times to obtain linear classifiers.We provide you with an implementation of the algorithm which you can use to obtain your results.The specifications for the `perceptron` method provided are:* `data` is a numpy array of dimension $d$ by $n$* `labels` is numpy array of dimension $1$ by $n$* `params` is a dictionary specifying extra parameters to this algorithm; your algorithm runs a number of iterations equal to $T$* `hook` is either None or a function that takes the tuple `(th, th0)` as an argument and displays the separator graphically. It should return a tuple of $\theta$ (a $d$ by 1 array) and $\theta_0$ (a 1 by 1 array).Note that you are free to modify the method. For example, a useful modification for this homework would be to make the method return the number of mistakes made on the input data, while it runs.<jupyter_code># Perceptron algorithm with offset.
# data is dimension d by n
# labels is dimension 1 by n
# T is a positive integer number of steps to run
def perceptron(data, labels, params = {}, hook = None):
    # if T not in params, default to 50
    T = params.get('T', 100)
    (d, n) = data.shape
    m = 0
    theta = np.zeros((d, 1)); theta_0 = np.zeros((1, 1))
    for t in tqdm.tqdm(range(T)):
        for i in range(n):
            x = data[:,i:i+1]
            y = labels[:,i:i+1]
            if y * positive(x, theta, theta_0) <= 0.0:
                theta = theta + y * x
                theta_0 = theta_0 + y
                m += 1
                if hook: hook((theta, theta_0), i)
    return theta, theta_0

def averaged_perceptron(data, labels, params = {}, hook = None):
    T = params.get('T', 50)
    (d, n) = data.shape

    theta = np.zeros((d, 1)); theta_0 = np.zeros((1, 1))
    theta_sum = theta.copy()
    theta_0_sum = theta_0.copy()
    for t in range(T):
        for i in range(n):
            x = data[:,i:i+1]
            y = labels[:,i:i+1]
            if y * positive(x, theta, theta_0) <= 0.0:
                theta = theta + y * x
                theta_0 = theta_0 + y
                if hook: hook((theta, theta_0))
            theta_sum = theta_sum + theta
            theta_0_sum = theta_0_sum + theta_0
    theta_avg = theta_sum / (T*n)
    theta_0_avg = theta_0_sum / (T*n)
    if hook: hook((theta_avg, theta_0_avg))
    return theta_avg, theta_0_avg

  
def eval_classifier(learner, data_train, labels_train, data_test, labels_test):
    th, th0 = learner(data_train, labels_train)
    return score(data_test, labels_test, th, th0)/data_test.shape[1]

def positive(x, th, th0):
    return np.sign(th.T@x + th0)

def score(data, labels, th, th0):
    return np.sum(positive(data, th, th0) == labels)

def xval_learning_alg(learner, data, labels, k):
    _, n = data.shape
    idx = list(range(n))
    np.random.seed(0)
    np.random.shuffle(idx)
    data, labels = data[:,idx], labels[:,idx]

    score_sum = 0
    s_data = np.array_split(data, k, axis=1)
    s_labels = np.array_split(labels, k, axis=1)
    for i in range(k):
        data_train = np.concatenate(s_data[:i] + s_data[i+1:], axis=1)
        labels_train = np.concatenate(s_labels[:i] + s_labels[i+1:], axis=1)
        data_test = np.array(s_data[i])
        labels_test = np.array(s_labels[i])
        score_sum += eval_classifier(learner, data_train, labels_train,
                                              data_test, labels_test)
    return score_sum/k
data =   np.array([[2, 3, 4, 5]])
labels = np.array([[1, 1, -1, -1]])
num_iterations = 5000 #TODO: change number of iterations
perceptron(data, labels, params = {'T':num_iterations}, hook = None)
data = np.array([[200, 800, 200, 800],
                [0.2,  0.2,  0.8,  0.8],
        #  [1,1,1,1]
        ])
labels = np.array ([[-1, -1, 1, 1]])
num_iterations = 300000 #TODO: change number of iterations

gt = 0

def h(p, t):
    global gt
    gt += 1

perceptron(data, labels, params = {'T':num_iterations}, hook = h)

print(gt)
data = np.array([[0.2, 0.8, 0.2, 0.8],
                [0.2,  0.2,  0.8,  0.8],
        #  [1,1,1,1]
        ])
labels = np.array ([[-1, -1, 1, 1]])
num_iterations = 300000 #TODO: change number of iterations

gt = 0

def h(p,t):
    global gt
    gt += 1

perceptron(data, labels, params = {'T':num_iterations}, hook = h)

print(gt)<jupyter_output>100%|██████████| 300000/300000 [00:08<00:00, 37196.53it/s]<jupyter_text>2D) Encoding Discrete ValuesIt is common to encode sets of discrete values, for machine learning, not as a single multi-valued feature, but using a one hot encoding. So, if there are $k$ values in the discrete set, we would transform that single multi-valued feature into $k$ binary-valued features, in which feature $i$ has value $+1$ if the original feature value was $i$ and has value $0$ (or $-1$) otherwise.Write a function `one_hot` that takes as input $x$, a single feature value (between $1$ and $k$), and $k$, the total possible number of values this feature can take on, and transform it to a numpy column vector of $k$ binary features using a one-hot encoding (remember vectors have zero-based indexing).<jupyter_code>def one_hot(x, k):
    # Make an empty column vector
    v = np.zeros((k, 1))
    # Set an entry to 1
    v[x-1, 0] = 1
    return v
#data_1 = np.array([one_hot(x, 6) for x in data[0,]]).T
data_1=data_1[0]
num_iterations = 5000 #TODO: change number of iterations
perceptron(data_1, labels, params = {'T':num_iterations}, hook = None)
data_2 = np.array([[1, 2, 3, 4, 5, 6]])
data_3 = np.array([one_hot(x, 6) for x in data_2[0,:]]).T
labels = np.array([[1, 1, -1, -1, 1, 1]])
data = data_3[0]
num_iterations = 5000 #TODO: change number of iterations
perceptron(data, labels, params = {'T':num_iterations}, hook = None)
test_one_hot(one_hot)<jupyter_output><empty_output><jupyter_text>3) Polynomial FeaturesOne systematic way of generating non-linear transformations of your input features is to consider the polynomials of increasing order.  Given a feature vector $x = [x_1, x_2, ..., x_d]^T$, we can map it into a new feature vector that contains all the factors in a polynomial of order $d$. For example, for $x = [x_1, x_2]^T$ and order 2, we get $$\phi(x) = [1, x_1, x_2, x_1x_2, x_1^2, x_2^2]^T$$ and for order 3, we get $$\phi(x) = [1, x_1, x_2, x_1x_2, x_1^2, x_2^2, x_1^2x_2, x_1x_2^2, x_1^3, x_2^3]^T.$$  In the code that has been loaded, we have defined `make_polynomial_feature_fun` that, given the order, returns a feature transformation function (analogous to $\phi$ in the description).  You should use it in doing this problem.<jupyter_code>## For example, make_polynomial_feature_fun could be used as follows:

# Generate transformation of order 2
transformation = make_polynomial_feature_fun(2)

# Data
data = np.zeros((5,1))

# Use transformation on data
print(transformation(data))

for i in [1, 10, 20, 30, 40, 50]:
  print((make_polynomial_feature_fun(i)(np.zeros((2,1)))).shape)<jupyter_output>[[1.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]
 [0.]]
(3, 1)
(66, 1)
(231, 1)
(496, 1)
(861, 1)
(1326, 1)<jupyter_text>Note that iterative animations, which update a plot within a loop, don't work the same way in colab, as with a local python console installation.  One workaround for colab to be able to show such plot iterations is to show all the plots, and this can be done for the test code using this patched function:<jupyter_code># T is a positive integer number of steps to run
def perceptron(data, labels, params = {}, hook = None):
    # if T not in params, default to 50
    T = params.get('T', 100)
    (d, n) = data.shape
    m = 0
    theta = np.zeros((d, 1)); theta_0 = np.zeros((1, 1))
    for t in tqdm.tqdm(range(T)):
        for i in range(n):
            x = data[:,i:i+1]
            y = labels[:,i:i+1]
            if y * positive(x, theta, theta_0) <= 0.0:
                theta = theta + y * x
                theta_0 = theta_0 + y
                m += 1
                if hook: hook((theta, theta_0))
    return theta, theta_0
def test_linear_classifier_with_features(dataFun, learner, feature_fun,
                             draw = True, refresh = True, pause = True):
    raw_data, labels = dataFun()
    data = feature_fun(raw_data) if feature_fun else raw_data
    if draw:
        def hook(params):
            ax = plot_data(raw_data, labels)   # create plot axis on each iteration
            (th, th0) = params
            predictor = lambda x1,x2: int(positive(feature_fun(cv([x1, x2])), th, th0))
            plot_nonlin_sep(
                predictor,
                ax = ax)
            plot_data(raw_data, labels, ax)
            plt.show()                         # force plot to push to the colab notebook and be displayed
            print('th', th.T, 'th0', th0)
            if pause: input('press enter here to continue:')
    else:
        hook = None
    th, th0 = learner(data, labels, hook = hook)
    if hook: hook((th, th0))
    print("Final score", int(score(data, labels, th, th0)))
    print("Params", np.transpose(th), th0)

def test_with_features(dataFun, order = 2, draw=True, pause=True):
    test_linear_classifier_with_features(
        dataFun,                        # data
        perceptron,                     # learner
        make_polynomial_feature_fun(order), # feature maker
        draw=draw,
        pause=pause)<jupyter_output><empty_output><jupyter_text>Here are a few tests you can run to see plots:<jupyter_code># Separable dataset 
print(super_simple_separable())
test_with_features(super_simple_separable, order=3, draw=True, pause=False)
# XOR dataset 
print(xor_more())
test_with_features(xor_more, order=3, draw=False, pause=False)
test_with_features(super_simple_separable, order=1, draw=False, pause=False)
test_with_features(super_simple_separable, order=2, draw=False, pause=False)
test_with_features(super_simple_separable, order=3, draw=False, pause=False)
#test_with_features(super_simple_separable_through_origin, order=3, draw=False, pause=False)
#test_with_features(super_simple_separable, order=1, draw=False, pause=False)
#test_with_features(xor, order=1, draw=False, pause=False)
#test_with_features(xor, order=2, draw=False, pause=False)
#test_with_features(xor, order=3, draw=False, pause=False)
#test_with_features(xor_more, order=1, draw=False, pause=False)
#test_with_features(xor_more, order=2, draw=False, pause=False)
#test_with_features(xor_more, order=3, draw=False, pause=False)
#test_with_features(xor_more, order=4, draw=False, pause=False)<jupyter_output>100%|██████████| 100/100 [00:00<00:00, 13638.24it/s]
100%|██████████| 100/100 [00:00<00:00, 10920.96it/s]
100%|██████████| 100/100 [00:00<00:00, 7656.07it/s]<jupyter_text>Experiments  4) Car Data - Evaluating Algorithmic and Feature ChoicesWe now want to build a classifier for the auto data, focusing on thenumeric data.  In the code file for this part of the assignment, we have supplied youwith the `load_auto_data` function, that can be used to read therelevant .tsv file.  It will return a list of dictionaries, one for each data item.We then have to specify what feature function to use for each columnin the data.  The file `hw3_part2_main.py` has an example for constructingthe data and label arrays using `raw` feature function for all the columns.Look at the definition of `features` in `hw3_part2_main.py`, this indicates a feature name touse and then a feature function, there are three defined in the`code_for_hw3_part2.py` file (`raw`, `standard` and `one_hot`).  `raw` just usesthe original value, `standard` subtracts out the mean value anddivides by the standard deviation and `one_hot` does the encodingdescribed in the notes.The function `auto_data_and_labels` will process the dictionaries andreturn data, labels where data are arrays ofdimension $(d, 392)$, with $d$ the total number of features specified,and labels is of dimension $(1, 392)$.  The data in the fileis sorted by class, but it will be shuffled when you read it in.<jupyter_code># Perceptron algorithm with offset.
# data is dimension d by n
# labels is dimension 1 by n
# T is a positive integer number of steps to run
def perceptron(data, labels, params = {}, hook = None):
    # if T not in params, default to 50
    T = params.get('T', 100)
    (d, n) = data.shape
    m = 0
    theta = np.zeros((d, 1)); theta_0 = np.zeros((1, 1))
    for t in tqdm.tqdm(range(T)):
        for i in range(n):
            x = data[:,i:i+1]
            y = labels[:,i:i+1]
            if y * positive(x, theta, theta_0) <= 0.0:
                theta = theta + y * x
                theta_0 = theta_0 + y
                m += 1
                if hook: hook((theta, theta_0))
    return theta, theta_0

def averaged_perceptron(data, labels, params = {}, hook = None):
    T = params.get('T', 50)
    (d, n) = data.shape

    theta = np.zeros((d, 1)); theta_0 = np.zeros((1, 1))
    theta_sum = theta.copy()
    theta_0_sum = theta_0.copy()
    for t in range(T):
        for i in range(n):
            x = data[:,i:i+1]
            y = labels[:,i:i+1]
            if y * positive(x, theta, theta_0) <= 0.0:
                theta = theta + y * x
                theta_0 = theta_0 + y
                if hook: hook((theta, theta_0))
            theta_sum = theta_sum + theta
            theta_0_sum = theta_0_sum + theta_0
    theta_avg = theta_sum / (T*n)
    theta_0_avg = theta_0_sum / (T*n)
    if hook: hook((theta_avg, theta_0_avg))
    return theta_avg, theta_0_avg

  
def eval_classifier(learner, data_train, labels_train, data_test, labels_test):
    th, th0 = learner(data_train, labels_train)
    return score(data_test, labels_test, th, th0)/data_test.shape[1]

def positive(x, th, th0):
    return np.sign(th.T@x + th0)

def score(data, labels, th, th0):
    return np.sum(positive(data, th, th0) == labels)

def xval_learning_alg(learner, data, labels, k):
    _, n = data.shape
    idx = list(range(n))
    np.random.seed(0)
    np.random.shuffle(idx)
    data, labels = data[:,idx], labels[:,idx]

    score_sum = 0
    s_data = np.array_split(data, k, axis=1)
    s_labels = np.array_split(labels, k, axis=1)
    for i in range(k):
        data_train = np.concatenate(s_data[:i] + s_data[i+1:], axis=1)
        labels_train = np.concatenate(s_labels[:i] + s_labels[i+1:], axis=1)
        data_test = np.array(s_data[i])
        labels_test = np.array(s_labels[i])
        score_sum += eval_classifier(learner, data_train, labels_train,
                                              data_test, labels_test)
    return score_sum/k
# Returns a list of dictionaries.  Keys are the column names, including mpg.
auto_data_all = hw3.load_auto_data('auto-mpg.tsv')

# The choice of feature processing for each feature, mpg is always raw and
# does not need to be specified.  Other choices are hw3.standard and hw3.one_hot.
# 'name' is not numeric and would need a different encoding.
features = [('cylinders', hw3.raw),
            ('displacement', hw3.raw),
            ('horsepower', hw3.raw),
            ('weight', hw3.raw),
            ('acceleration', hw3.raw),
            ## Drop model_year by default
            ## ('model_year', hw3.raw),
            ('origin', hw3.raw)]

# Construct the standard data and label arrays
auto_data, auto_labels = hw3.auto_data_and_labels(auto_data_all, features)
print('auto data and labels shape', auto_data.shape, auto_labels.shape)
# Your code here to process the auto data
features = [('cylinders', hw3.raw),
            ('displacement', hw3.raw),
            ('horsepower', hw3.raw),
            ('weight', hw3.raw),
            ('acceleration', hw3.raw),
            ('origin', hw3.raw)]

# Construct the standard data and label arrays
auto_data, auto_labels = hw3.auto_data_and_labels(auto_data_all, features)
print('auto data and labels shape', auto_data.shape, auto_labels.shape)

# acc= xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 10}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 10}), auto_data, auto_labels, k=10)
# print(acc)
#acc =xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 10}), auto_data, auto_labels, k=10)
#print(acc)
#acc =xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 50}), auto_data, auto_labels, k=10)
#print(acc)

features_new = [('cylinders', hw3.one_hot),
            ('displacement', hw3.standard),
            ('horsepower', hw3.standard),
            ('weight', hw3.standard),
            ('acceleration', hw3.standard),
            ('origin', hw3.one_hot)]

# Construct the standard data and label arrays
auto_data, auto_labels = hw3.auto_data_and_labels(auto_data_all, features_new)
print('auto data and labels shape', auto_data.shape, auto_labels.shape)

# acc= xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 1}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 1}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 10}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 10}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 50}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 50}), auto_data, auto_labels, k=10)
# print(acc)

# acc= xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 1}), auto_data, auto_labels, k=10)
# print(acc)
# acc= xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 1}), auto_data, auto_labels, k=10)
# print(acc)

averaged_perceptron(auto_data, auto_labels, {"T": 1})<jupyter_output>avg and std {'displacement': (388.3482142857143, 302.0458123396403), 'horsepower': (509.3545918367347, 333.6521151716361), 'weight': (2977.5841836734694, 848.3184465698365), 'acceleration': (15.541326530612228, 2.7553429127509963)}
entries in one_hot field {'cylinders': [3.0, 4.0, 5.0, 6.0, 8.0], 'origin': [1.0, 2.0, 3.0]}
auto data and labels shape (12, 392) (1, 392)<jupyter_text>5) Food Reviews - Evaluating Algorithmic and Feature ChoicesWe have supplied you with the `load_review_data`function, that can be used to read a .tsv file and return the labelsand texts. We have also supplied you with the `bag_of_words` function,which takes the raw data and returns a dictionary of unigramwords. The resulting dictionary is an input to`extract_bow_feature_vectors` which computes a feature matrix of onesand zeros that can be used as the input for the classificationalgorithms.  The file `hw3_part2_main.py` has code for constructingthe data and label arrays.  Using these arrays and our implementationof the learning algorithms, you will be able to compute $\theta$ and$\theta_0$.  You will need to add theimplementation of Perceptron and Averaged Perceptron.<jupyter_code># Returns lists of dictionaries.  Keys are the column names, 'sentiment' and 'text'.
# The train data has 10,000 examples
review_data = hw3.load_review_data('reviews.tsv')

# Lists texts of reviews and list of labels (1 or -1)
review_texts, review_label_list = zip(*((sample['text'], sample['sentiment']) for sample in review_data))

# The dictionary of all the words for "bag of words", mapping unique words to unique integer values
dictionary = hw3.bag_of_words(review_texts)

# The standard data arrays for the bag of words
review_bow_data = hw3.extract_bow_feature_vectors(review_texts, dictionary)
review_labels = hw3.rv(review_label_list)

print('review_bow_data and labels shape', review_bow_data.shape, review_labels.shape)
# Evaluate learning algorithm with different T values
xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 10}), review_bow_data, review_labels, k=10)
# Evaluate learning algorithm with different T values
xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 1}), review_bow_data, review_labels, k=10)
xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 1}), review_bow_data, review_labels, k=10)
xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 10}), review_bow_data, review_labels, k=10)
xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 10}), review_bow_data, review_labels, k=10)
xval_learning_alg(lambda data, labels: perceptron(data, labels, {"T": 50}), review_bow_data, review_labels, k=10)
xval_learning_alg(lambda data, labels: averaged_perceptron(data, labels, {"T": 50}), review_bow_data, review_labels, k=10)
# Your code here to process the food review data
th, th_0 = averaged_perceptron(review_bow_data, review_labels, {"T": 10})
dictionary
keyword_ind= np.argsort(th[:,0])
keyword_ind

most_negative = keyword_ind[:10]

most_positive = keyword_ind[-10:]

word_dict = hw3.reverse_dict(dictionary)

negative_words = [word_dict[i] for i in most_negative]

positive_words = [word_dict[i] for i in most_positive]
negative_words
positive_words<jupyter_output><empty_output><jupyter_text>6.2) Evaluating Features for MNIST DataThis problem explores how well the perceptron algorithm works to <ahref="http://neuralnetworksanddeeplearning.com/chap1.html">classifyimages of handwritten digits, from the well-known ("MNIST")dataset, buiding on your thoughts from lab about extracting featuresfrom images.  This exercise will highlight how important featureextraction is, before linear classification is done, using algorithmssuch as the Perceptron.Dataset setupOften, it may be easier to work with a vector whose spatial orientation is preserved.In previous parts, we have represented features as one long feature vector.For images, however, we often represent a $m$ by $n$ imageas a `(m,n)` array, rather than a `(mn,1)` array(as the previous parts have done).In the code file, we have supplied you with the `load_mnist_data` function,which will read from the provided image files and populate a dictionary,with image and label vectors for each numerical digit from 0 to 9.These images are already shaped as `(m,n)` arrays.<jupyter_code>mnist_data_all = hw3.load_mnist_data(range(10))

print('mnist_data_all loaded. shape of single images is', mnist_data_all[0]["images"][0].shape)

# HINT: change the [0] and [1] if you want to access different images
d0 = mnist_data_all[2]["images"]
d1 = mnist_data_all[4]["images"]
y0 = np.repeat(-1, len(d0)).reshape(1,-1)
y1 = np.repeat(1, len(d1)).reshape(1,-1)

# data goes into the feature computation functions
data = np.vstack((d0, d1))
# labels can directly go into the Perceptron algorithm
labels = np.vstack((y0.T, y1.T)).T
y0
y1
def raw_mnist_features(x):
    n_samples, m, n = x.shape
    return x.reshape((n_samples, m*n)).T
# change these implementations to support whole datasets

def raw_mnist_features(x):
    n_samples, m, n = x.shape
    return x.reshape((n_samples, m*n)).T

def row_average_features(x):
    return np.mean(x, axis=1,keepdims=True)

def col_average_features(x):
    return np.mean(x, axis=0,keepdims=True).T


def top_bottom_features(x):
    m = x.shape[0]
    n = x.shape[1]
    return np.array([[np.mean(x[:m // 2,]), np.mean(x[m // 2:,])]]).T
top_bottom_features(data)
# use this function to evaluate accuracy
acc = hw3.get_classification_accuracy(raw_mnist_features(data), labels)
print(acc)
def raw_mnist_features(x):
    data_points, m, n = x.shape
    return x.reshape((data_points, m*n)).T
raw_mnist_features(data)
data.shape
def row_average_features(x):
    """
    @param x (n_samples,m,n) array with values in (0,1)
    @return (m,n_samples) array where each entry is the average of a row
    """
    n_samples, m, n = x.shape
    return np.mean(x, axis=2,keepdims=True).T.reshape((m,n_samples))

def col_average_features(x):
    """
    @param x (n_samples,m,n) array with values in (0,1)
    @return (n,n_samples) array where each entry is the average of a column
    """
    n_samples, m, n = x.shape
    return np.mean(x, axis=1,keepdims=True).T.reshape((n,n_samples))

def top_bottom_features(x):
    """
    @param x (n_samples,m,n) array with values in (0,1)
    @return (2,n_samples) array where the first entry of each column is the average of the
    top half of the image = rows 0 to floor(m/2) [exclusive]
    and the second entry is the average of the bottom half of the image
    = rows floor(m/2) [inclusive] to m
    """
    n_samples, m, n = x.shape
    return np.array([np.mean(x[:,:m // 2,],axis=(1,2)), np.mean(x[:,m // 2:,],axis=(1,2))]).reshape((2,n_samples))

# HINT: change the [0] and [1] if you want to access different images
d0 = mnist_data_all[9]["images"]
d1 = mnist_data_all[0]["images"]
y0 = np.repeat(-1, len(d0)).reshape(1,-1)
y1 = np.repeat(1, len(d1)).reshape(1,-1)

# data goes into the feature computation functions
data = np.vstack((d0, d1))
# labels can directly go into the Perceptron algorithm
labels = np.vstack((y0.T, y1.T)).T
acc = hw3.get_classification_accuracy(row_average_features(data), labels)
print(acc)
acc = hw3.get_classification_accuracy(col_average_features(data), labels)
print(acc)
acc = hw3.get_classification_accuracy(top_bottom_features(data), labels)
print(acc)<jupyter_output>0.49749999999999994
{'fraction_non_alphanumeric': 0.1220449601160261, 'fraction_numerical': 0.0340826686004351, 'mean_word_length': 3.8764144271570014, 'pattern_counts': {'":': 21, '<': 41, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '357865', 'license': 'MIT', 'n_tokens_mistral': 10060, 'n_tokens_neox': 9364, 'n_words': 2907, 'path': 'MIT_6_036_HW03_Feature_Representation.ipynb', 'repo_name': 'XuRui314/6.036_Machine_Learning', 'size': '27580'}