<jupyter_start><jupyter_text>PS7In this problem set, we will use the symplest type of recurrent network, and Elman network, to try to learn a symple language..  InstructionsRemember to do your problem set in Python 3. Fill in `YOUR CODE HERE`.Make sure: - that all plots are scaled in such a way that you can see what is going on (while still respecting specific plotting instructions) - that the general patterns are fairly represented.- to label all x- and y-axes, and to include a title.        **N.B.** The ideas in this notebook draw heavily from the readings <NAME>. (1990). Finding structure in time. _Cognitive Science, 14_, 179-211.<NAME>., & <NAME>. (1986). Past tenses of English verbs. In <NAME>. and <NAME>. (Eds.) _Parallel distributed processing: Explorations in the microstructure of cognition. Vol. 2: Applications_ (pp. 216-271). Cambridge, MA: MIT Press.If you are confused about some of the ideas in this notebook or would like further clarification, we recommend having a look there.<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>IntroductionOne of the most successful (and controversial) applications of neural        networks has been as models of human language. Specifically, contrary to the rules/symbols approach proposed by Chomsky, neural networks have been used to demonstrate that distributed representations can give rise to language learning. You will test whether asimple neural network is capable of learning the rule underlying acontext-free language.The language $a^nb^n$, being the set of all strings containing asequence of $a$'s followed by a sequence of $b$'s of the same lengthis a simple example of a language that can be generated by acontext-free grammar but not a finite-state grammar (because a finite state grammar cannot keep track of the number of times a string has occurred in the sentence). Human languagesexhibit similar long-range constraints -- for example, a plural nounat the start of a sentence affects conjugation of a verb at the end,regardless of what intervenes. Some criticisms of applications ofneural networks to human languages are based upon their apparentreliance on local sequential structure, which makes them seem muchmore similar to finite-state grammars than to context-freegrammars. In other words, simple neural networks will mostly use the most recent word to predict the next one. An interesting question to explore is thus whether arecurrent neural network can learn to generalize a simple rulecharacterizing a long-range dependency, such as the rule underlying$a^nb^n$.Recall that an "Elman" network, as discussed by Elman (1990), is arecurrent network where the activation of the hidden units at theprevious timestep are used as input to the hidden units on the currenttimestep. This type of network architecture allows thenetwork to learn about sequential dependencies in the input data. In this notebook we will evaluate whether such a network can learn an $a^nb^n$grammar. Here we formalize learning a grammar as being able to correctlypredict what the next item in a sequence should be given therules of the grammar. Therefore, the output node represents thenetworks's prediction for what the next item in the sequence (the nextinput) will be -- it outputs a $1$ if it thinks the current input willbe followed by an $a$, and outputs a $0$ if it thinks the currentinput will be followed by a $b$.  Q1. Data	[5pts, SOLO]We will use the `abdata.npz` dataset for this problem. Make sure that the data file is in the same directory as your notebook while working on the problem set.This dataset has two keys.- The array `train_data` contains the sequence we will use to train our network.- The array `test_data` contains the sequence we will use to evaluate thenetwork.In both `train_data` and `test_data` a $1$ represents an $a$ and a $0$ represents a $b$.`train_data` was constructed by concatenating a randomly orderedset of strings of the form $a^nb^n$, with $n$ ranging from 1 to 11.The frequency of sequences for a given value of $n$ in the training setare given by `np.ceil(50/n)`, thus making them inversely proportional to $n$.The `np.ceil` function returns the smallest integer greater or equal toits input. For example, `np.ceil(3)` is 3, but `np.ceil(3.1)` is4 . `test_data` contains an ordered sequence of strings of the form$a^nb^n$, with $n$ increasing from 1 to 18 over the length of thestring.Take a look at the data! You can first print the variables. Next plot the first 100 values of train_data and the first 100 values of test_data in two subplots. Make sure to label the axes and the titles. Upload your plot to gradescope as PS7_Q1.png<jupyter_code>ab_data = np.load("abdata.npz")
ab_data.keys()

# look at train_data
print('train_data')
train_data = ab_data['train_data']
print(train_data[:30])
print(len(train_data))

## look at test_data
print('test_data')
test_data = ab_data['test_data']
print(test_data)

## plot the data
#YOUR CODE HERE


figure.savefig('PS7_Q1.png')<jupyter_output><empty_output><jupyter_text>Q2. Input/output [3 pts HELP]In order to train your network, you will need both training *input* andtraining *output*. That is, you need a sequence of inputs of the form$a^nb^n$, and a corresponding sequence with the correct output foreach item in the input sequence.For this problem we're going to use `train_data[:-1]` as theinput training sequence, and `train_data[1:]` as the outputtraining sequence.Explain in gradescope why `train_data[:-1]` and `train_data[1:]` are appropriate input and outputsequences. If you're confused by what the sequences`train_data[:-1]` and `train_data[1:]` look like,try creating them in a cell and compare them to `train_data`.  Elman network (provided)We have provided you with a function, `train_Elman`, which takes four arguments:- `input` -- the training input sequence- `output` -- the training output sequence- `num_hidden` -- the number of hidden units- `num_iters` -- the number of training iterations: network needs to be trained on many iterations`train_Elman` will:1) create a network with one input node, the specified number of hidden units, and one output node2) train the network on the training data for the specified number of iterations.The network sees thetraining data one input at a time (in our case, it sees a single $1$or $0$ per time step).<jupyter_code>def train_Elman(inputs, outputs, num_hidden, num_iters):
    """
    Initializes and trains an Elman network. For details see Elman (1990).

    Parameters
    ----------
    inputs : numpy array
        A one dimensional sequence of input values to the network

    outputs : numpy array
        A one-dimensional sequence of desired output values for each of the
        items in inputs

    num_hidden : int
        The number of hidden units to use in the network

    num_iters : int
        The number of training iterations to run the network for

    Returns
    -------
    net : dict
        Dictionary object containing the trained network weights for each layer. 
        Key 1 corresponds to the weights from the visibles to the hidden units,
        key 2 corresponds to the weights from the hiddens to the output units.

    NOTE: Poorly-Python-ported from trainElman.m, which in turn was adapted from
    code from http://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/matlab/
    recurrent/ which in turn was adapted from Elman (1990) :-)
    """
    np.random.seed(seed=1)

    # Parameters
    # increment to the derivative of the transfer function (Fahlman's trick)
    DerivIncr = 0.2
    Momentum  = 0.05
    LearnRate = 0.001

    num_input  = 1
    num_output = 1
    num_train  = inputs.shape[0]

    if inputs.ndim == 2:
        num_input  = inputs.shape[0]
        num_output = outputs.shape[0]
        num_train  = inputs.shape[1]

    if not all([outputs.ndim == inputs.ndim,
               inputs.shape[0] == outputs.shape[0]]):
        raise ValueError('unequal number of input and output examples')

    # create a dictionary to hold the network weights
    net = {}
    net[1] = np.random.rand(num_hidden, num_input + num_hidden + 1) - 0.5
    net[2] = np.random.rand(num_output, num_hidden + 1) - 0.5

    # the context layer
    # zeros because it is not active when the network starts
    Result1 = np.zeros((num_hidden, num_train))

    # the row of ones is the bias
    Inputs = np.vstack((inputs, np.ones(num_train)))
    Desired = outputs

    delta_w1 = 0.
    delta_w2 = 0.

    # Training
    for ii in range(num_iters):
        # Recurrent state
        # includes current inputs, as well as the output of the hidden layer
        # from the previous time step
        Input1 = np.vstack((Inputs, np.hstack([np.zeros((num_hidden,1)),  Result1[:,:-1]])))

        # Forward propagate activations
        # input --> hidden
        NetIn1 = np.dot(net[1], Input1)
        Result1 = np.tanh(NetIn1)

        # Hidden --> output
        # we again add a row of ones for bias
        Input2 = np.vstack((Result1, np.ones(num_train)))
        NetIn2 = np.dot(net[2], Input2)
        Result2 = np.tanh(NetIn2)

        # Backprop errors
        # output --> hidden
        Result2Error = Result2 - Desired
        In2Error = Result2Error * (DerivIncr + np.cosh(NetIn2)**(-2))

        # hidden --> input
        Result1Error = np.dot(net[2].T, In2Error)
        In1Error = Result1Error[:-1, :] * (DerivIncr + np.cosh(NetIn1)**(-2))

        # Calculate weight updates
        dw2 = np.dot(In2Error, Input2.T)
        dw1 = np.dot(In1Error, Input1.T)

        delta_w2 = -LearnRate * dw2 + Momentum * delta_w2
        delta_w1 = -LearnRate * dw1 + Momentum * delta_w1

        net[2] = net[2] + delta_w2
        net[1] = net[1] + delta_w1
    return net<jupyter_output><empty_output><jupyter_text>Q3 - learn anbn language [HELP 5 pts]Complete the function `anbn_learner` below to train an "Elman"network with two hidden units using the provided function `train_Elman` (rememberto use the input **train_data[:-1]** and output **train_data[1:]** sequences from Q2).Train the network for *100 iterations*, and return the final output of the network.We provide test cases. If you got the function right, the following cell should print "Success". Otherwise, it will give you an error message that will help with debugging.Copy your code into gradescope.<jupyter_code>def anbn_learner(train_data):
    """
    Creates an "Elman" neural network with two hidden units and trains it
    on the provided data.

    Parameters
    ----------
    train_data: numpy array of shape (n,)
        the data on which to train the Elman network

    Returns
    -------
    net: dictionary with 2 keys
        a dictionary containing the weights of the network. Valid keys are 1 and 2. 
        key 1 is for the weights between the input and the hidden units, and 
        key 2 is for the weights between the hidden units and the output units.
    """
    # YOUR CODE HERE
    
"""Check that anbn_learner returns the correct output"""
from numpy.testing import assert_array_equal
from nose.tools import assert_equal, assert_almost_equal 

# check that abdata hasn't been modified
ab = np.load("data/abdata.npz")
assert_array_equal(test_data, ab['test_data'], "test_data array has changed")
assert_array_equal(train_data, ab['train_data'], "train_data array has changed")

# generate test data
traindata = np.zeros(20)
traindata[10:] = 1.

net = anbn_learner(traindata)

# check that net has the correct shape and type
assert_equal(type(net), dict, "net should be a dict of network weights")
assert_equal(len(net), 2, "incorrect number of layers in net")
assert_equal(list(net.keys()), [1,2], "keys for net should be 1 and 2")

# check the dimensions of the weight matrices
assert_equal(net[1].shape, (2,4), "invalid network weights for the input -> hidden layer")
assert_equal(net[2].shape, (1,3), "invalid network weights for the hidden -> output layer")

# check the weight matrix sums to the correct value on testdata
assert_almost_equal(np.sum(net[1]), -1.9326, places=4, msg="weights for input --> hidden layer are incorrect")
assert_almost_equal(np.sum(net[2]), 0.01825, places=4, msg="weights for hidden --> output layer are incorrect")
print("Success!")<jupyter_output><empty_output><jupyter_text>Q4 - checking the trained network [5 pts, solo]Once the network is trained (*your anbn_learner function should pass the test case in the previous cell)*, you can test it on a new set of sequencesand evaluate its predictions to see how well it has learned the target grammar. To generate predictions from the trained network, we use the provided function `predict_Elman`.  Use your `anbn_learner` function on `train_data` to train a network, then use your trained network (by passing it as the first input to `predict_Elman` function) to predict the sequences in `test_data`. A) The `predict_Elman` function returns an array of predicted values with the same dimensions as the input.Plot your prediction (sequence index number on x axis, predicted values on y axis) and upload the figure to gradescope as PS7_Q4.png.B) Look back at your plot of the test data in Q1 - what do you think of the network's predictions? Do your predictions approximate the testing data? Explain why you think they do, or why you think they do not, by referring to the mechanisms of recurrent networks and the nature of the data.<jupyter_code>def predict_Elman(net, inputs):
    """
    Uses the Elman network parameterized by the weights in net to generate
    predictions for the elements in inputs.

    Parameters
    ----------
    net : dict
        Dictionary object containing the trained network weights and
        recurrent connections as produced by train_Elman. Key 1 corresponds 
        to the weights from the visibles to the hidden units, key 2 
        corresponds to the weights from the hiddens to the output units.

    inputs : numpy array
        A one dimensional sequence of input values to the network

    Returns
    -------
    outputs : numpy array
        An array containing the predictions generated by the Elman network for the
        items in inputs.
        
    NOTE: Poorly-Python-ported from predictElman.m, which in turn was adapted from
    code from http://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/matlab/
    recurrent/ which in turn was adapted from Elman (1990) :-)
    """
    num_output = 1
    num_hidden = net[1].shape[0]
    num_train  = inputs.shape[0]

    if inputs.ndim == 2:
        num_train = inputs.shape[1]

    Inputs = np.vstack((inputs, np.ones([1, num_train])))
    Result1 = np.zeros([num_hidden, 1])

    outputs = np.zeros(num_train)

    for i in range(num_train):
        Input1 = np.append(Inputs[:, i], Result1)
        NetIn1 = np.dot(net[1], Input1)
        Result1 = np.tanh(NetIn1)

        Input2 = np.append(Result1, np.ones((1, 1)))
        NetIn2 = np.dot(net[2], Input2)
        outputs[i] = np.tanh(NetIn2)
    return outputs

## YOUR CODE HERE

figure.savefig('PS7_Q4.png')<jupyter_output><empty_output><jupyter_text>Q5 - Quantifying the model performance [2pts, HELP]How well does the network do at predicting the next letter? Has it learned the language? Let's look more carefully.To quantify how well the network performs we are going to look at how much the predicted sequence deviates from expectations. The squared error (SE) for a prediction $p_i$ in the prediction vector ${\bf p}$compared to a target value ${y_i}$ in the target vector ${\bf y}$ is\begin{equation}SE_i = (p_i-y_i)^2\end{equation}That is, the squared error is just the squared difference between thepredicted and target value.	Complete the function `squared_error`, which takes in an array of test data and an array ofpredictions. The function should return an error array **with the same number of elements as the test data**, containing the SE for eachof the predictions of the network compared against the corresponding value in `test_data`. Remember that the predictions refer to the _next_ item in the sequence(e.g.  `predictions[0]` should be compared to`test_data[1]`, etc.). You should append an $a$ (coded as a $1$) to the end of your test data to equate the array sizes (describing the start of a new sequence of $a^nb^n$). We have provided test cases again in the cell below. If your function is right, it should print success. Otherwise, it will give you an error with feedback.Copy your function into gradescope.<jupyter_code>def squared_error(predictions, test_data):
    """
    Uses equation 1 to compute the SE for each of the predictions made 
    by the network.

    Parameters
    ----------
    predictions: numpy array of shape (n,)
        an array of predictions from the Elman network
    
    test_data: numpy array of shape (n,) 
        the array of test data from which predictions were generated

    Returns
    -------
    se_vector: numpy array of shape (n,)
        an array containing the SE for each of items in predictions 
    """
    # YOUR CODE HERE


#YOUR OWN TESTS HERE
"""Check that squared_error returns the correct output"""
from nose.tools import assert_equal

# generate test data
pred = np.array([1, 0, 1])
test = np.array([0, 1, 0])
se = squared_error(pred, test)

# check that squared_error returns the correct output for testdata
#assert_equal(se.dtype, np.float64, "squared_error should return an array of floats")
assert_equal(se.shape, (3,), "squared_error returned an array of the incorrect size on the validate testdata")
assert_array_equal(se, np.zeros(3), "squared_error should return all zeros on the validate testdata")

# check that squared_error compares the correct elements
pred = np.zeros(1)
test = np.zeros(1)
se = squared_error(pred, test)
assert_equal(se, np.ones(1), "squared_error([0],[0]) should have returned a 1 (did you remember to append an a to testdata?")

print("Success!")<jupyter_output><empty_output><jupyter_text>--- Q6 [5 pts, SOLO]Train the network on the train_data, then apply it to the test_data: try to predict test data using the weights of the trained network. Measure the resulting squared error.Use matplotlib to plot a bar graph of the squared error for each training example. You should have the sequence iteration number on the x axis, and the error values on y axis. Don't forget to provide a title and labels your $x$ and $y$ axes!If you have difficulty interpreting this graph, you may want toexamine a few of the values in `test_data`, `predictions`, and your `mse_vector` to see how they are related.Upload your figure PS7_Q6.png to gradescope.<jupyter_code># YOUR CODE HERE

# create the figure
fig, axis = plt.subplots()
axis.set_xlim([0.0, 350.0]), axis.set_ylim([0.0,.7])

# YOUR CODE HERE
axis.bar(np.arange(len(se_vector)),se_vector)<jupyter_output><empty_output><jupyter_text>Q7 [SOLO, 5pts]To get a better idea of what is going on, let's have a look at the specific values in `test_data` where the prediction error spikes. Use the provided code below and look at its output. At which points in test_data do the large errors occur? Explain which parts of the data the network is predicting ok, and which parts it's not predicting well. Answer in gradescope.<jupyter_code># prints the 3 values preceding and 2 values following the spot where 
# the prediction error >= 0.5
error_spike_idxs = np.argwhere(se_vector >= 0.5) + 1
error_spike_idxs = error_spike_idxs[:-1]

for i in error_spike_idxs:
    print('3 values preceding MSE spike: {}\tValue at MSE spike: {}'
          '\t\t2 values following MSE spike: {}'\
          .format(test_data[i[0]-3:i[0]], test_data[i[0]], test_data[i[0]+1:i[0]+3]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08079530506931094, 'fraction_numerical': 0.0158352726904786, 'mean_word_length': 3.814174378957623, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '506217', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5783, 'n_tokens_neox': 5484, 'n_words': 2545, 'path': 'PS7/PS7_final_MR_AC.ipynb', 'repo_name': 'smitchaute/COGSCI-C131-PSYCH-123-FA20', 'size': '19797'}