<jupyter_start><jupyter_text>Homework 3For this homework, you will be working extensively in tensorflow. It is suggested that you spin up a Google Cloud VM with a GPU attached. Remember, instructions for doing so are found in Homework 0. Part 1: Homework 2, but on tensorflow Part 2: DNN on MNIST and CIFAR10 Part 3: VGG on MNIST and CIFAR10 (Optional) Part 4, getting state of the art (SOTA)  Part 1You don't have to repeat everything in homework 2, but rather pick one set of two features that worked well for you last homework, and implement logistic regression using tensorflow without using keras (you will practice using keras in parts 2 and 3). In other words, using tensorflow operations, please create a scalar-value loss function and let tensorflow create the training operation for logistic regression, which automatically computes the gradients and updates the weight parameters. Note that the logistic loss is a special case of the softmax cross entropy loss that you've seen when classifying MNIST.<jupyter_code># Import datasets
%matplotlib inline

from sklearn import datasets
import matplotlib.pyplot as plt
digits = datasets.load_digits()
from sklearn.model_selection import train_test_split
import numpy as np

number_a = 0
number_b = 3

digit_a_indexes = np.where(digits.target==number_a) 
digit_b_indexes = np.where(digits.target==number_b)
targets = np.concatenate((digits.target[digit_a_indexes], digits.target[digit_b_indexes]))
images = np.concatenate((digits.images[digit_a_indexes], digits.images[digit_b_indexes]))
X_train, X_test, y_train, y_test = train_test_split(images, targets, test_size=0.1, random_state=42)
def compute_features(vector):
    image = vector.reshape(8, 8) # get back original image shape
    def compute_feature_a(image):
        return np.mean(image*np.rot90(image, k = 2))
    def compute_feature_b(image):
        return np.mean(image)
    return compute_feature_a(image), compute_feature_b(image)

# Apply and plot your features
# for x train dataset
X_features_train = np.apply_along_axis(compute_features, 1, X_train.reshape(X_train.shape[0], X_train.shape[1]*X_train.shape[2]))
X_features_a = X_features_train[np.where(y_train==number_a)]
X_features_b = X_features_train[np.where(y_train==number_b)]

# for x test dataset
X_features_test = np.apply_along_axis(compute_features, 1, X_test.reshape(X_test.shape[0], X_test.shape[1]*X_test.shape[2]))
X_features_a_test = X_features_test[np.where(y_test==number_a)]
X_features_b_test = X_features_test[np.where(y_test==number_b)]


# for y train
y_train_label = np.copy(y_train)
y_train_label[y_train == number_a] = -1
y_train_label[y_train == number_b] = 1

# for y test
y_test_label = np.copy(y_test)
y_test_label[y_test == number_a] = -1
y_test_label[y_test == number_b] = 1
# cast as a float32; in general, you will work with float32 inputs:
X_features_train = X_features_train.astype(np.float32)
X_features_test = X_features_test.astype(np.float32)
print(X_features_train.shape)
print(X_features_test.shape)
y_train_label = y_train_label.astype(np.float32)
y_test_label = y_test_label.astype(np.float32)
print(y_train_label.shape)
print(y_test_label.shape)
# this can be either X_train/y_train or X_test/y_test, so we make a placeholder that we can feed into:
X_train_or_test = tf.placeholder(tf.float32, [None, 2], name='input_features')
y_train_or_test = tf.placeholder(tf.int32, [None], name='image_label')
# batch_size = 2

# # create a tf dataset, from which we can generate batches
# dataset = tf.data.Dataset.from_tensor_slices((X_train_or_test, y_train_or_test))
# dataset = dataset.repeat()
# dataset = dataset.batch(batch_size)
# batch_generator = dataset.make_initializable_iterator()
# X_batch, y_batch = batch_generator.get_next()  # batches symbolically generated
net = X_batch
# fully connected layers:
# net = tf.layers.flatten(net)
# net = tf.layers.dense(net, units=512, activation=tf.nn.relu) # was 256
net = tf.layers.flatten(net)
net = tf.layers.dense(net, units=2) # was 10

logits = net
weights = np.random.rand(2)
features = np.copy(X_features_train)
labels = np.copy(y_test_label)

loss = (0.5*np.matmul((labels - np.matmul(features,weights)).transpose(), (labels - np.matmul(features,weights))))/np.size(labels)
# loss = tf.losses.softmax_cross_entropy(onehot_labels=tf.one_hot(y_batch, depth=2), logits=logits) # original depth = 10
# loss = tf.losses.log_loss(onehot_labels=tf.one_hot(y_batch, depth=2), logits=logits) # original depth = 10
train_op = tf.train.GradientDescentOptimizer(learning_rate=.01).minimize(loss)
config = tf.ConfigProto(device_count={'GPU': 0})
sess = tf.InteractiveSession(config=config)
sess.run(tf.global_variables_initializer())
sess.run(batch_generator.initializer, feed_dict={X_train_or_test: X_features_train, y_train_or_test: y_train_label})
for i in range(1000):
    _, loss_i = sess.run([train_op, loss])
    if i%100 == 0:
        print(loss_i)
sess.run(batch_generator.initializer, feed_dict={X_train_or_test: X_features_test, y_train_or_test: y_test_label})
correct = 0
total = 0
for i in range(100):
    prediction, truth = sess.run([logits, y_batch])
    correct += np.sum(prediction.argmax(1)==truth)
    total += len(truth)
acc = correct/total
print(acc)<jupyter_output><empty_output><jupyter_text>Part 2:  DNN on MNIST and CIFAR10In our lab, you guys saw how to work with the MNIST dataset to perform image classification. We can attempt the MNIST classification problem with just fully connected layers. This means we will be optimizing for non-banded matrices (no convolutions).1. Calcualte the number of weight parameters you are optimizing for 1, 2 and 3 different fully connected layers (the total size of each layer is up to you).2. What is the max layer depth you can go before training loss does not converge? You can usually tell that something is not converging by examining the training loss vs. iteration curve.3. How does the number of parameters relate to the training loss and validation/test loss? Try to get a few data points to speak to this question.3. Keeping the maximum number of parameters possible while still maintaining convergence (i.e., a good training and validation/test loss), what happens when you swap the activation function to `tanh` instead of `relu`? How about `sigmoid`?4. After exploring the above, train a DNN model with the combination of hyperparameters that you believe will work best on MNIST.5. Using the same architecture, try training a DNN model on more difficult dataset such as Fashion MNIST or CIFAR10/100. Example download instructions are shown in the next problem. Must haves1. Make a curve of the final validation/test loss of your DNN after the loss plateaus as a function of the number of weight parameters used (final loss versus  parameters used). Note that you might see something like the curve below for a low number of parameters, but as the number of parameters increases, it will not look like this plot. 2. On the same figure, make the same curve as above, but use different activation functions in your architecture.3. Plot a point corresponding to your crafted DNN archiecture for question 4.4. Repeat 1-3 for CIFAR10The curves when reasonable  params are used look like the below![image.png](attachment:image.png)<jupyter_code>parameters = []
train_error = []
test_error = []
import tensorflow as tf
(X_train, y_train), (X_val, y_val) = tf.keras.datasets.mnist.load_data() #load dataset
y_train = tf.keras.utils.to_categorical(y_train, 10)
y_val = tf.keras.utils.to_categorical(y_val, 10)

X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)
X_val = X_val.reshape(X_val.shape[0], 28, 28, 1)

from matplotlib import pyplot as plt
%matplotlib inline

print('Training data shape', X_train.shape)
_, (ax1, ax2) = plt.subplots(1, 2)
ax1.imshow(X_train[0].reshape(28, 28), cmap=plt.cm.Greys);
ax2.imshow(X_train[1].reshape(28, 28), cmap=plt.cm.Greys);
n = 15 # number of test
for i in range(n):
    model = tf.keras.Sequential([])
    model.add(tf.keras.layers.Flatten(input_shape=(28, 28, 1)))
    for j in range(i):
        model.add(tf.keras.layers.Dense(100, activation='relu'))
    for k in range(i):
        model.add(tf.keras.layers.Dense(50, activation='relu'))
    model.add(tf.keras.layers.Dense(10, activation='softmax'))
    parameters.append(model.count_params())  #count parameters for you

    model.compile(optimizer=tf.train.AdamOptimizer(0.005),
                  loss='categorical_crossentropy', #10 classes for output
                  metrics=['accuracy']) # add mae: mean average error
    model.summary()
    early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=3)
    # tensorboard = tf.keras.callbacks.TensorBoard(log_dir='logs/{}'.format('model_name'))
    hist = model.fit(X_train, y_train,
                         batch_size=64,
                         epochs=5,
                         verbose=1,
                         validation_data=(X_val, y_val),
                         callbacks=[early_stop])
    train_error.append(100*(1-max(hist.history['acc'])))
    print('train_error: ', train_error)
    test_error.append(100*(1-max(hist.history['val_acc'])))
    print('test_error: ', test_error)
def plot_loss(parameters_list, train_errors_list, test_errors_list):
    ax = plt.subplot(111)
    ax.plot(parameters_list, train_errors_list,'r', label='train error')
    ax.plot(parameters_list, test_errors_list,'g', label='test error')
    ax.set(xlabel='# parameters', ylabel='validation error (%)')
    ax.legend()
    plt.title('Tanh Multiple Layers')
    plt.show()

print('parameter list: ', parameters)
print('train_error: ', train_error)
print('test_error: ', test_error)
plot_loss(parameters, train_error, test_error)
# tanh:
n = 15 # number of test
for i in range(n):
    model = tf.keras.Sequential([])
    model.add(tf.keras.layers.Flatten(input_shape=(28, 28, 1)))
    for j in range(i):
        model.add(tf.keras.layers.Dense(100, activation='tanh'))
    for k in range(i):
        model.add(tf.keras.layers.Dense(60, activation='tanh'))
    model.add(tf.keras.layers.Dense(10, activation='softmax'))
    parameters.append(model.count_params())  #count parameters for you

    model.compile(optimizer=tf.train.AdamOptimizer(0.001),
                  loss='categorical_crossentropy', #10 classes for output
                  metrics=['accuracy']) # add mae: mean average error
    model.summary()
    early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=3)
    # tensorboard = tf.keras.callbacks.TensorBoard(log_dir='logs/{}'.format('model_name'))
    hist = model.fit(X_train, y_train,
                         batch_size=64,
                         epochs=15,
                         verbose=1,
                         validation_data=(X_val, y_val),
                         callbacks=[early_stop])
    train_error.append(100*(1-max(hist.history['acc'])))
    print('train_error: ', train_error)
    test_error.append(100*(1-max(hist.history['val_acc'])))
    print('test_error: ', test_error)
print('parameter list: ', parameters)
print('train_error: ', train_error)
print('test_error: ', test_error)
plot_loss(parameters, train_error, test_error)
# sigmoid:
n = 15 # number of test
for i in range(n):
    model = tf.keras.Sequential([])
    model.add(tf.keras.layers.Flatten(input_shape=(28, 28, 1)))
    for j in range(i):
        model.add(tf.keras.layers.Dense(100, activation='sigmoid'))
    for k in range(i):
        model.add(tf.keras.layers.Dense(60, activation='sigmoid'))
    model.add(tf.keras.layers.Dense(10, activation='softmax'))
    parameters.append(model.count_params())  #count parameters for you

    model.compile(optimizer=tf.train.AdamOptimizer(0.001),
                  loss='categorical_crossentropy', #10 classes for output
                  metrics=['accuracy']) # add mae: mean average error
    model.summary()
    early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=3)
    hist = model.fit(X_train, y_train,
                         batch_size=64,
                         epochs=15,
                         verbose=1,
                         validation_data=(X_val, y_val),
                         callbacks=[early_stop])
    train_error.append(100*(1-max(hist.history['acc'])))
    print('train_error: ', train_error)
    test_error.append(100*(1-max(hist.history['val_acc'])))
    print('test_error: ', test_error)
def plot_loss(parameters_list, train_errors_list, test_errors_list):
    ax = plt.subplot(111)
    ax.plot(parameters_list, train_errors_list,'r', label='train error')
    ax.plot(parameters_list, test_errors_list,'g', label='test error')
    ax.set(xlabel='# parameters', ylabel='validation error (%)')
    ax.legend()
    plt.title('Sigmoid Multiple Layers')
    plt.show()
print('parameter list: ', parameters)
print('train_error: ', train_error)
print('test_error: ', test_error)
plot_loss(parameters, train_error, test_error)<jupyter_output>parameter list:  [7850, 85170, 98930, 112690, 126450, 140210, 153970, 167730, 181490, 195250, 209010, 222770, 236530, 250290, 264050]
train_error:  [26.25666666666666, 6.691666666666663, 7.033333333333336, 7.956666666666667, 9.431666666666672, 15.671666666666662, 18.845, 89.33833333333332, 88.76333333333332, 89.3, 89.24833333333333, 89.32166666666667, 89.30333333333334, 89.23333333333333, 89.025]
test_error:  [25.670000000000005, 6.020000000000003, 6.999999999999995, 7.689999999999997, 9.040000000000003, 14.139999999999997, 17.100000000000005, 88.64999999999999, 88.64999999999999, 88.64999999999999, 88.64999999999999, 88.64999999999999, 88.64999999999999, 88.64999999999999, 88.64999999999999]<jupyter_text>Part 3. VGG on CIFAR100 and CIFAR10VGG is a simple, but powerful CNN created in 2015. Read the VGG paper here: https://arxiv.org/pdf/1409.1556.pdfHere, we're going to try to reproduce the model's findings on the cifar10 and cifar100 dataset. Note that the paper takes 224 x 224 images, but cifar10 and 100 are only 32 x 32 images.1. Implement all of the layers for the VGG ConvNet Configuration A. Please use the shell code below as guide. Then, train this network on the Cifar10 and Cifar100 datasets.2. For Cifar10 and 100, VGG is probably overkill. Try changing the number of layers and number of filters without sacrificing too much performance accuracy. How many filters can you get rid of before you see the accuracy drop by more than 2%? Where in the architecture is it better to remove filters - towards the input layers, or more towards the output layers?3. For what you experiment with--report the parameter, validation loss curves for changing the number of i) layers, ii) filter size, iii) both.<jupyter_code># This is the same model in the other notebook, looks very simplified.
import tensorflow as tf
(X_train, y_train), (X_val, y_val) = tf.keras.datasets.cifar10.load_data() #32*32 RGB vectors
y_train = tf.keras.utils.to_categorical(y_train, 10)
y_val = tf.keras.utils.to_categorical(y_val, 10)

X_train = X_train.reshape(X_train.shape[0], 32, 32, 3)
X_val = X_val.reshape(X_val.shape[0], 32, 32, 3)

from matplotlib import pyplot as plt
%matplotlib inline

print('Training data shape', X_train.shape)
_, (ax1, ax2) = plt.subplots(1, 2)
ax1.imshow(X_train[0].reshape(32, 32, 3));
ax2.imshow(X_train[1].reshape(32, 32, 3));
# Example CNN used in class
model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(64, (5,5), padding='same', activation='relu', input_shape=(32, 32, 3)),
    tf.keras.layers.MaxPool2D(padding='same'),
    
    tf.keras.layers.Conv2D(128, (5,5), padding='same', activation='relu'),
    tf.keras.layers.MaxPool2D(padding='same'),
    
    tf.keras.layers.Conv2D(256, (5,5), padding='same', activation='relu'),
    tf.keras.layers.Conv2D(256, (5,5), padding='same', activation='relu'),
    tf.keras.layers.MaxPool2D(padding='same'),
    

    tf.keras.layers.Conv2D(512, (5,5), padding='same', activation='relu'),
    tf.keras.layers.Conv2D(512, (5,5), padding='same', activation='relu'),
    tf.keras.layers.MaxPool2D(padding='same'),
    
    tf.keras.layers.Conv2D(512, (5,5), padding='same', activation='relu'),
    tf.keras.layers.Conv2D(512, (5,5), padding='same', activation='relu'),
    tf.keras.layers.MaxPool2D(padding='same'),

    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(4096, activation='relu'),
    tf.keras.layers.Dense(1000, activation='relu'),
    tf.keras.layers.Dense(1000, activation='relu'),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer=tf.train.AdamOptimizer(0.0001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

model.summary()
early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=3)
hist = model.fit(X_train, y_train,
                 batch_size=64,
                 epochs=5,
                 verbose=1,
                 validation_data=(X_val, y_val),
                 callbacks=[early_stop])<jupyter_output>Train on 50000 samples, validate on 10000 samples
Epoch 1/5
  512/50000 [..............................] - ETA: 1:06:17 - loss: 2.3181 - acc: 0.0840
{'fraction_non_alphanumeric': 0.11856510598648963, 'fraction_numerical': 0.05864197530864197, 'mean_word_length': 4.393530150753769, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 27, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '223451', 'license': 'MIT', 'n_tokens_mistral': 6392, 'n_tokens_neox': 5615, 'n_words': 1471, 'path': 'homework/homework_3/Homework3RoujiaWang.ipynb', 'repo_name': 'RoujiaW/ML-Imaging', 'size': '17172'}