<jupyter_start><jupyter_text>CS109B Data Science 2: Advanced Topics in Data Science  Advanced-Sections: Homework 2 - Echo-State Reservoir Computing**Harvard University****Spring 2021****Instructors**: <NAME>, <NAME>, & <NAME> **Authors**: <NAME>, <NAME><jupyter_code>#RUN THIS CELL 
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
import os
import pathlib
working_dir = pathlib.Path().absolute()
# Uncomment the line below to help debug if the path to included images don't show
#print(working_dir)
os.chdir(working_dir)<jupyter_output><empty_output><jupyter_text>INSTRUCTIONS- To submit your assignment follow the instructions given in Canvas.- This homework can be submitted in pairs.- If you submit individually but you have worked with someone, please include the name of your **one** partner below.- Please restart the kernel and run the entire notebook again before you submit. (Exception - you may skip the cells where you train neural networks, running the cells which load previously saved weights instead. However, **don't delete/overwrite the output that model.fit produced during training!**)**Names of person you have worked with goes here:**<jupyter_code>import numpy as np
from numpy import loadtxt

from matplotlib import pyplot as plt
%matplotlib inline

# This is a class for reservoir computing
# The pyESN.py file must be in the same directory with this notebook
from pyESN import ESN<jupyter_output><empty_output><jupyter_text>Outline- [**Question 1: Short-range forecast  [5pts total]**](q1)In this question you are asked to perform a short range prediction. In particular, you have to use the `95%` of the sequential points of the time-series `x(t)` and predict the final `5%` which  is considered the validation or testing set; in this homework the validation and the testing sets are the same.   - [1.1 [2.5pts]](q1.1)First, try manually to find a set of the hyper-parameters  `spectral_radius` and  `sparsity` that yields a prediction with relatively low validation/testing MSE (smaller than 0.25). Plot the training and the prediction along with the ground truth data. Also, show the residual between the ground truth and your prediction.     - [1.2 [2.5pts]](q1.2)Then make a more systematic hyper-parameter optimization by using a grid search for the hyper-parameters `spectral_radius` and  `sparsity`. The goal is to find an optimal set that gives the lowest MSE on the prediction (on the testing set). Make a 2D color plot to show the MSE for the different values of `spectral_radius` and `sparity`.Finally, you have to perform a prediction with the optimal hyper-parameter set. Plot the training and the prediction along with the ground truth data. Also, show the residual between the ground truth and your prediction. - [**Question 2: Long-range forecast [5 pts]:**](q2)Here, you are asked to make a  long-range prediction. Use the first `50%` of your data to train the RC network and then predict the final `50%`. This is a very long prediction and consequently, it is an extremely hard task.   - [2.1 [2pts]](q2.1)First, show that by using the RC as in the Question 1, it is imposible to make  good predictions (predictions with MSE smaller than 0.4). Make a grid search to check what is the lowest possible testing MSE that you can get.  - [2.2 [3pts]](q2.2)Then, use the concept of the `observers` and perform an inference prediction. Follow the steps of the Question 1. Make a grid search for the hyperparameters `spectral-radius` and `sparsity`. Visualize the MSE in prediction by using a 2D plot.  Then perform an inference prediction with the optimal set. Plot the training and prediction along with the ground truth. Also, show the residuals.    OverviewDuring the lecture we discussed about the formulation of the  *Reservoir Computing (RC); an echo-state recurrent neural network*. One of the examples that we discussed was the predictions of the chaotic time-series for the Mackey-Glass nonlinear dynamical system. This example can be found in the seminal paper: http://www.rctn.org/vs265/jaeger04-ESN.pdf. This is the paper that RC was introduced, so we encourage you to read this publication. In this homework, you are asked to work on the Rossler dynamical system. It is a very popular chaotic system  used to describe the evolution of chemical reactions. For more information check the Wikipedia:  https://en.wikipedia.org/wiki/R%C3%B6ssler_attractorFor the implementation of the RC you should the python class `pyESN` taken from https://github.com/cknd/pyESN. In this github repository you can also find the Mackey-Glass example. We encourage you to explore this library. Note, `pyESN` can also be found in the current homework directory.**The task:** Employ the RC network to predict the time evolution of  chaotic time series.  We provide you three time series `x(t), y(t), z(t)` (files: `x.dat`, `y.dat`, `z.dat`), which are solutions of the chaotic Rossler system. **In the Question 1**, you are asked to make a short-range forecast. It is a prediction where the network is learning from the past data and predicts the future behavior. In this case, we need a single input. The input signal is a constant function because the sequential steps are considered to be  equally-spaced (difference between time steps), hence, the input should be an array of ones. **In the Question 2**  you need to to make a long-time prediction. First, you should use a single constant input (similar to the question 1) where you have to confirm that the predictions are very bad. This is expected because we deal with  very difficult (chaotic) time-series. On the other hand, we saw in the class that  using the concept of the `observers`  we can perform extremely long-range predictions. To include the observers you need to use three different  inputs: a vector of ones (as in the question 1), and the two other known time series `y(t)` and `z(t)` (the observers). In this case the prediction is a response to the past behavior and to the present observers' signals. This kind of predictions are often called **inference**. As we discussed in the lecture,  RC is very sensitive to the hyper-parameter. In all the questions you are asked to find  the optimal set of hyper-parameter that gives the best predictions. For your convenience, we are asking you to optimize just two of the hyper-parameter, the `spectrum radius`  and `sparsity term`.  The rest hyper-parameters are given. For this homework you should use a two-dimensional grid to  search the two hyper-parameters, report the mean square error (MSE) for each hyper-parameter set, and keep the model with the lowest MSE.  The goals of this homework are for you to:1. learn the mechanics of RC2. confirm that RC training is extremely fast3. learn how to use RC for forecasting4. acknowledge that RC is sensitive to hyper-parameters (no free lunch)5. learn how to optimize hyper-parameters of RC6. evaluate RC forecasting7. learn observers-based RC for inference**Tip**: Running the entire notebook on the Jupyter-hub should take about 30 minutes. The most computationally expensive part  is the hyper-parameter optimization (grid-search). First, you can try a sparse hyper-parameter grid. Once everything runs smoothly, you can try a denser grid to get a better optimization.   A brief review of  `pyESN` library for the RC implementation You call the RC as:esn = ESN(n_inputs =  ,  $\quad$ $\quad$     n_outputs = ,  $\quad$ $\quad$     n_reservoir = , $\quad$ $\quad$     sparsity= , $\quad$ $\quad$     random_state= ,  $\quad$ $\quad$     spectral_radius = , $\quad$ $\quad$     noise= ) where  denotes the value that you choose. Brief explanation of the parameters:`n_inputs`: number of input dimensions `n_outputs`: number of output dimensions `n_reservoir`: number of reservoir neurons  `random_state`: seed for the random generator`sparsity`: proportion of recurrent weights set to zero `spectral_radius`: spectral radius of the recurrent weight matrix `noise`: noise added to each hidden neuron (regularization)  Along this homework you should fix the following hyper-parameters.`n_outputs = 1`, `n_reservoir = 1000`, `noise = 0.0001`, `random_state=42`  **Helper functions**We are providing three helper functions.  You can use them or you can write your custom functions. If you define any other helper function (e.g. a function for plotting your results), it should be placed after the next cell. Helper functions are not requested by this homework.The given helper functions calculate the mean square error `MSE`, the `residuals`, and prepare the data. The `prepareData` splits the data into training and testing sets, and create a training and testing array of ones (account to the time-input).  We do not need evaluation data sets. Note that the `prepareData` **does not** prepare the *observers*, you will need to do it manually in the question 2.<jupyter_code># HELPER FUNCTIONS GO HERE

def myMSE(prediction,target):
    return np.sqrt(np.mean((prediction.flatten() - target.flatten() )**2))

def residuals(prediction,target):
    return (target.flatten() - prediction.flatten())

    
def prepareData(target, train_perc=0.9, plotshow=False):
    datalen =  len(target)        
    trainlen = int(train_perc*datalen)
    testlen  = datalen-trainlen

# Train/Test sets
    trainTarget = target[:trainlen]
    testTarget  = target[trainlen:trainlen+testlen]    
    inputTrain = np.ones(trainlen)
    inputTest  = np.ones(testlen)
        
    if plotshow:
        plt.figure(figsize=(14,3))
        plt.plot(range(0,trainlen), trainTarget,'g',label='Train')
        plt.plot(range(trainlen,trainlen+testlen), testTarget,'-r',label='Test')
        plt.legend(loc=(0.1,1.1),fontsize=18,ncol=2)
        plt.tight_layout()
        
    return trainTarget, testTarget, inputTrain, inputTest<jupyter_output><empty_output><jupyter_text>**If you define any custom helper function, it should be put in the following cell**<jupyter_code># HELPER FUNCTIONS GO HERE
### your code here
def plotResults(trainlen, testlen, data, yfit, yhat, resTrain, resTest):
    plt.figure(figsize=(14,6))
    plt.subplot(2,1,1)
    plt.plot(range(0,trainlen+testlen), data,'k',  linewidth=2, label="data")
    plt.plot(range(0,trainlen),yfit,'--g',  linewidth=2, alpha=0.9, label="train")
    plt.plot(range(trainlen,trainlen+testlen), yhat,'--r', linewidth=2,  alpha=1, label="test")
    lo,hi = plt.ylim()
    plt.plot([trainlen,trainlen],[lo+np.spacing(1),hi-np.spacing(1)],'--b',alpha=0.8, linewidth=4)
    plt.tight_layout()
    plt.ylabel('x')
    plt.legend(loc=(0.1,1.1),fontsize=18,ncol=3)

    
    #
    plt.subplot(2,1,2)
    plt.plot(range(0,trainlen),resTrain,'--g',  linewidth=2, alpha=0.9, label="train")
    plt.plot(range(trainlen,trainlen+testlen), resTest,'--r', linewidth=2,  alpha=1, label="test")
    lo,hi = plt.ylim()
    plt.plot([trainlen,trainlen],[lo+np.spacing(1),hi-np.spacing(1)],'--b',alpha=0.8, linewidth=4)
    plt.tight_layout()
    plt.ylabel('Residuals')<jupyter_output><empty_output><jupyter_text>**Inspect the datasets**Load and plot three time series<jupyter_code>### your code here
x = loadtxt('data/x.dat')
y = loadtxt('data/y.dat')
z = loadtxt('data/z.dat')

fig, ax = plt.subplots(1, figsize=(20, 15))
n=len(x)
ax.plot(range(n), x, label='x', color='red')
ax.plot(range(n), y, label='y', color='blue')
ax.plot(range(n), z, label='z', color='green')
ax.set_xlabel('Time')
ax.set_ylabel('Amplitude')
ax.set_title('Visualization of the 3 time series')
ax.legend()
plt.show(fig)<jupyter_output><empty_output><jupyter_text>Question 1 [Return to Outline](contents)- [1: Short-range forecast](q1) [5pts total] In this question you are asked to perform a short range prediction. In particular, you have to use the `95%` of the sequential points of the time-series `x(t)` and predict the final `5%` which  is considered the validation or testing set; in this homework the validation and the testing sets are the same. - [1.1:](q1.1)First, try manually to find a set of the hyper-parameters  `spectral_radius` and  `sparsity` that yields a prediction with relatively low validation/testing MSE (smaller than 0.25). Plot the training and the prediction along with the ground truth data. Also, show the residual between the ground truth and your prediction. - [1.2:](q1.2)Then make a more systematic hyper-parameter optimization by using a grid search for the hyper-parameters `spectral_radius` and  `sparsity`. The goal is to find an optimal set that gives the lowest MSE on the prediction (on the testing set). Make a 2D color plot to show the MSE for the different values of `spectral_radius` and `sparity`.Finally, you have to perform a prediction with the optimal hyper-parameter set. Plot the training and the prediction along with the ground truth data. Also, show the residual between the ground truth and your prediction.   1.1 Manual hyper-parameter search and prediction Set the target time-series and give the name `target`<jupyter_code>### your code here
target=x<jupyter_output><empty_output><jupyter_text>**Prepare your data:** The target time-series should be splitted  into training and testing sets. Plot the time-series and use different colors to indicate the training and testing sets.  You might want to use the given helper function `prepareData()` or you can do it by yourself.<jupyter_code>### your code here
trainTarget, testTarget, inputTrain, inputTest = prepareData(target, train_perc=0.95, plotshow=True)<jupyter_output><empty_output><jupyter_text>Make a quick prediction until find a testing MSE < 0.25: Try around `spectral_radius = [1.2, 2.6]` and   `sparsity = [0.16,  0.24]`.<jupyter_code>### your code here
n_outputs = 1
n_reservoir = 1000
noise = 0.0001
random_state=42
spectral_radius = [1.2, 2.6]
sparsity = [0.16, 0.24]

esn = ESN(n_inputs = 1,
      n_outputs = n_outputs,
      n_reservoir = n_reservoir,
      noise = noise,
      spectral_radius = 1.2,
      sparsity=0.16,          
      random_state=random_state)
yfit = esn.fit(inputTrain, trainTarget)
yhat = esn.predict(inputTest)
mse = myMSE(yhat, testTarget)
resTrain = residuals(yfit, trainTarget)
resTest = residuals(yhat, testTarget)
print(f"For the set of parameters: spectral_radius = 1.2 & sparsity = 0.16, the MSE value on the test set it: {mse:.4f}")
trainlen = int(len(x)*0.95)
testlen = int(len(x)*0.05)
plotResults(trainlen, testlen, x, yfit, yhat, resTrain, resTest)<jupyter_output><empty_output><jupyter_text>1.2 Hyper-parameters optimizationMake a grid search on the hyper-parameters `spectra-radius` and `sparsity`. Visualize the result by ploting the testing MSE in a 2D color plot.**Tips:** Search in the range: `spectral_radius = [1.2, 2.6]` and   `sparsity = [0.16,  0.24]`. The RC is more sensitive on the `sparsity`. For a dense grid you can consider the steps 0.05 and 0.01 for the `spectra-radius` and `sparsity`, respectively.<jupyter_code>### your code here
spectral_radius_list = np.arange(1.2,2.6,0.2)
sparsity_list = np.arange(0.16,0.24,0.01)

mse_lst = []
yfit_lst = []
yhat_lst = []
resTrain_lst = []
resTest_lst = []
sp_rad_spar_lst = []

for sp_rad in spectral_radius_list:
    for spar in sparsity_list:
        esn = ESN(n_inputs = 1,
                  n_outputs = n_outputs,
                  n_reservoir = n_reservoir,
                  noise = noise,
                  spectral_radius = sp_rad,
                  sparsity=spar,          
                  random_state=random_state)
        yfit = esn.fit(inputTrain, trainTarget)
        yfit_lst.append(yfit)
        yhat = esn.predict(inputTest)
        yhat_lst.append(yhat)
        sp_rad_spar_lst.append([sp_rad,spar])
        mse = myMSE(yhat, testTarget)
        mse_lst.append(mse)
        resTrain = residuals(yfit, trainTarget)
        resTrain_lst.append(resTrain)
        resTest = residuals(yhat, testTarget)
        resTest_lst.append(resTest)<jupyter_output><empty_output><jupyter_text>**plot the results**<jupyter_code>### your code here
fig, ax = plt.subplots(figsize = (12,12))
ax.set_xlabel('Sparsity', fontsize = 12)
ax.set_ylabel('Spectral Radius', fontsize = 12)
ax.set_title('Figure: Heatmap for the Gridsearch of Hyperparameters: Spectral Radius & Sparsity', fontsize = 16)
im = ax.imshow(np.array(mse_lst).reshape(8, 7))
cbar = ax.figure.colorbar(im, ax=ax)

plt.xticks(range(len(sparsity_list)), [round(float(spa),2) for spa in sparsity_list])
plt.yticks(range(len(spectral_radius_list)), [round(float(spa),2) for spa in spectral_radius_list])
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Optimal prediction Plot the training and the prediction along with the ground truth data. Also, show the residual between the ground truth and your prediction.<jupyter_code>### your code here
min_idx = np.argmin(mse_lst)
yfit =  yfit_lst[min_idx]
yhat = yhat_lst[min_idx]
resTrain = resTrain_lst[min_idx]
resTest = resTest_lst[min_idx]

plotResults(trainlen, testlen, x, yfit, yhat, resTrain, resTest)
print(f"For the set of parameters: spectral_radius: {np.round(sp_rad_spar_lst[min_idx][0], 2)} & sparsity: {np.round(sp_rad_spar_lst[min_idx][1], 2)}, the MSE value on the test set it: {mse_lst[min_idx]:.4f}")<jupyter_output>For the set of parameters: spectral_radius: 1.4 & sparsity: 0.18, the MSE value on the test set it: 0.0230<jupyter_text>Question 2 [Return to Outline](contents)- [2: Long-range forecast](q2) [5pts total] Here, you are asked to make a  long-range prediction. Use the first `50%` of your data to train the RC network and then predict the final `50%`. This is a very long prediction and consequently, it is an extremely hard task. - [2.1:](q2.1)First, show that by using the RC as in the Question 1, it is imposible to make  good predictions (predictions with MSE smaller than 0.4). Make a grid search to check what is the lowest possible testing MSE that you can get.- [2.2:](q2.2)Then, use the concept of the `observers` and perform an inference prediction. Follow the steps of the Question 1. Make a grid search for the hyperparameters `spectral-radius` and `sparsity`. Visualize the MSE in prediction by using a 2D plot.  Then perform an inference prediction with the optimal set. Plot the training and prediction along with the ground truth. Also, show the residuals.  2.1 Predict using the RC as in Question 1 **Prepare your data:** Split the target time-series   in training and testing sets.<jupyter_code>### your code here
trainTarget, testTarget, inputTrain, inputTest = prepareData(target, train_perc=0.5, plotshow=True)
trainlen = int(len(x)*0.5)
testlen = int(len(x)*0.5)<jupyter_output><empty_output><jupyter_text>We can see that the best MSE is obtained with spectral radius of {} and sparsity of {}, it is not good enough ($>0.4$), this will therefore yield bad predictions  2.2 Hyperparameter Optimization**Tips:** Search in the range: `spectral_radius = [1.3, 1.6]` and   `sparsity = [0.17,  0.23]`. Again, the RC is more sensitive on the `sparsity`. For a dense grid you can consider the steps 0.05 and 0.01 for the `spectra-radius` and `sparsity`, respectively.<jupyter_code>### your code here
spectral_radius_list = np.arange(1.3,1.75,0.05)
sparsity_list = np.arange(0.17,0.24,0.01)

mse_lst = []
yfit_lst = []
yhat_lst = []
resTrain_lst = []
resTest_lst = []
sp_rad_spar_lst = []

for sp_rad in spectral_radius_list:
    for spar in sparsity_list:
        esn = ESN(n_inputs = 1,
                  n_outputs = n_outputs,
                  n_reservoir = n_reservoir,
                  noise = noise,
                  spectral_radius = sp_rad,
                  sparsity=spar,          
                  random_state=random_state)
        yfit = esn.fit(inputTrain, trainTarget)
        yfit_lst.append(yfit)
        yhat = esn.predict(inputTest)
        yhat_lst.append(yhat)
        sp_rad_spar_lst.append([sp_rad,spar])
        mse = myMSE(yhat, testTarget)
        mse_lst.append(mse)
        resTrain = residuals(yfit, trainTarget)
        resTrain_lst.append(resTrain)
        resTest = residuals(yhat, testTarget)
        resTest_lst.append(resTest)
np.array(mse_lst).shape<jupyter_output><empty_output><jupyter_text>**plot the results**<jupyter_code>### your code here
fig, ax = plt.subplots(figsize = (12,12))
ax.set_xlabel('Sparsity', fontsize = 12)
ax.set_ylabel('Spectral Radius', fontsize = 12)
ax.set_title('Figure: Heatmap for the Gridsearch of Hyperparameters: Spectral Radius & Sparsity', fontsize = 16)
im = ax.imshow(np.array(mse_lst).reshape(7, -1))
cbar = ax.figure.colorbar(im, ax=ax)

plt.xticks(range(len(sparsity_list)), [round(float(spa),2) for spa in sparsity_list])
plt.yticks(range(len(spectral_radius_list)), [round(float(spa),2) for spa in spectral_radius_list])
plt.tight_layout()
plt.show()
min_idx = np.argmin(mse_lst)
yfit =  yfit_lst[min_idx]
yhat = yhat_lst[min_idx]
resTrain = resTrain_lst[min_idx]
resTest = resTest_lst[min_idx]
plotResults(trainlen, testlen, x, yfit, yhat, resTrain, resTest)
print(f"For the set of parameters: spectral_radius: {sp_rad_spar_lst[min_idx][0]} & sparsity: {sp_rad_spar_lst[min_idx][1]:.4f}, the MSE value on the test set it: {mse_lst[min_idx]:.4f}")<jupyter_output>For the set of parameters: spectral_radius: 1.3 & sparsity: 0.2200, the MSE value on the test set it: 0.9552<jupyter_text>Inference: Observers Prepare your `observers`. The given `prepareData()` function does not prepare the observers, so you need to that manually.<jupyter_code>### your code here
trainlen,testlen = 10000,10000

# Input  data
ttrain = np.ones(trainlen)
ttest  = np.ones(testlen)

ytrain = y[:trainlen]
ytest  = y[trainlen:trainlen+testlen]

ztrain = z[:trainlen]
ztest  = z[trainlen:trainlen+testlen]

# Output data
xtrain = x[:trainlen]
xtest  = x[trainlen:trainlen+testlen]<jupyter_output><empty_output><jupyter_text>**Make a quick prediction to see the improvement (without optimizing the hyper-parameters yet)**<jupyter_code>### your code here

inputTrain = np.stack((ttrain,ytrain,ztrain), axis=1)
inputTest = np.stack((ttest,ytest,ztest), axis=1)

esn = ESN(n_inputs = 3,
          n_outputs = 1,
          n_reservoir = 3000,
          spectral_radius = 1.5,
          sparsity=.2,
          noise =0.0001,
          random_state=42
         )

# Train and predict 
xfit = esn.fit(inputTrain, xtrain)
xhat = prediction = esn.predict(inputTest)

train_mse=myMSE(xfit,trainTarget)
test_mse=myMSE(prediction,testTarget)
print("Training mean square error: ",train_mse)
print("Testing  mean square error: ",test_mse)<jupyter_output>Training mean square error:  8.097717449130698e-06
Testing  mean square error:  0.8976688286089369<jupyter_text>**Hyper-parameter optimization****Tips:** Search in the range: `spectral_radius = [1.4, 2.2]` and   `sparsity = [0.18,  0.24]`. The RC is more sensitive on the `sparsity`. For a dense grid you can consider the steps 0.1 and 0.01 for the `spectra-radius` and `sparsity`, respectively.  Do not run these cells !!<jupyter_code>### your code here
spectral_radius_list = np.arange(1.4,2.3,0.1)
sparsity_list = np.arange(0.18,0.25,0.01)

mse_lst = []
yfit_lst = []
yhat_lst = []
resTrain_lst = []
resTest_lst = []
sp_rad_spar_lst = []

for sp_rad in spectral_radius_list:
    for spar in sparsity_list:
        esn = ESN(n_inputs = 3,
                  n_outputs = n_outputs,
                  n_reservoir = 3000,
                  noise = noise,
                  spectral_radius = sp_rad,
                  sparsity=spar,          
                  random_state=random_state)
        yfit = esn.fit(inputTrain, xtrain)
        yfit_lst.append(yfit)
        yhat = esn.predict(inputTest)
        yhat_lst.append(yhat)
        sp_rad_spar_lst.append([sp_rad,spar])
        mse = myMSE(yhat, xtest)
        mse_lst.append(mse)
        resTrain = residuals(yfit, xtrain)
        resTrain_lst.append(resTrain)
        resTest = residuals(yhat, xtest)
        resTest_lst.append(resTest)<jupyter_output><empty_output><jupyter_text>![Grid%20Search.png](attachment:Grid%20Search.png)<jupyter_code>### your code here
mse_array = np.array(mse_lst).reshape(9, 7)
fig, ax = plt.subplots(figsize = (12,12))
ax.set_xlabel('Sparsity', fontsize = 12)
ax.set_ylabel('Spectral Radius', fontsize = 12)
ax.set_title('Figure: Heatmap for the Gridsearch of Hyperparameters: Spectral Radius & Sparsity', fontsize = 16)
im = ax.imshow(mse_array)
cbar = ax.figure.colorbar(im, ax=ax)

plt.xticks(range(len(sparsity)), [round(float(spa),2) for spa in sparsity])
plt.yticks(range(len(spectral_radius)), [round(float(spa),2) for spa in spectral_radius])
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Loaded result For the set of parameters: spectral_radius: 2.2 & sparsity: 0.2300, the MSE value on the test set it: 0.2687 **Plot here the optimal prediction (inference). As in Q1, show the fitting and prediction data along the ground truth as well as plot the residuals.**<jupyter_code>### your code here
min_idx = np.argmin(mse_lst)
yfit =  yfit_lst[min_idx]
yhat = yhat_lst[min_idx]
resTrain = resTrain_lst[min_idx]
resTest = resTest_lst[min_idx]

plotResults(trainlen, testlen, x_data, yfit, yhat, resTrain, resTest)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10233386075949367, 'fraction_numerical': 0.020727848101265822, 'mean_word_length': 4.4065440547476475, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '322875', 'license': 'MIT', 'n_tokens_mistral': 8334, 'n_tokens_neox': 7902, 'n_words': 2802, 'path': 'content/HW/HW5/cs109b_hw5_209.ipynb', 'repo_name': 'DavidAssaraf106/2021-CS109B', 'size': '25319'}