<jupyter_start><jupyter_text>Examples of hyperparameter optimizationIn this notebook, we provide examples for how to optimize hyperparameters of a decoder and then use the decoder with those hyperparameters. We demonstrate how to use 2 different hyperparameter optimization packages, ["BayesianOptimization"](https://github.com/fmfn/BayesianOptimization) and ["hyperopt"](http://hyperopt.github.io/hyperopt/). Both give similar performance. In the arXiv manuscript, I used "BayesianOptimization" (simply because I discovered it first). - The first few sections (1-3) just import packages, load the files, and preprocess them - Section 4 shows examples of [BayesianOptimization](https://github.com/fmfn/BayesianOptimization) for 3 decoders: Wiener Cascade, XGBoost, and Feedforward Neural Net - Section 5 shows examples of [hyperopt](http://hyperopt.github.io/hyperopt/) for 3 decoders: Wiener Cascade, XGBoost, and Feedforward Neural Net  - Section 6 shows examples of making test-set predictions using the decoders with the fit hyperparameters Note that the example using the Wiener Cascade is quick, but the examples with XGBoost and the Feedforward Neural Net are slower (depending on your computer, potentially 10's of minutes).  1. Import PackagesBelow, we import both standard packages, and functions from the accompanying .py files<jupyter_code>#Import standard packages
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from scipy import io
from scipy import stats
import pickle
import time
# If you would prefer to load the '.h5' example file rather than the '.pickle' example file. You need the deepdish package
# import deepdish as dd 

#Import function to get the covariate matrix that includes spike history from previous bins
from preprocessing_funcs import get_spikes_with_history

#Import metrics
from metrics import get_R2
from metrics import get_rho

#Import decoder functions
from decoders import WienerCascadeDecoder
from decoders import WienerFilterDecoder
from decoders import DenseNNDecoder
from decoders import SimpleRNNDecoder
from decoders import GRUDecoder
from decoders import LSTMDecoder
from decoders import XGBoostDecoder
from decoders import SVRDecoder

#Import hyperparameter optimization packages
#If either are not installed, give a warning
try:
    from bayes_opt import BayesianOptimization
except ImportError:
    print("\nWARNING: BayesianOptimization package is not installed. You will be unable to use section 4.")
    pass

try:
    from hyperopt import fmin, hp, Trials, tpe, STATUS_OK
except ImportError:
    print("\nWARNING: hyperopt package is not installed. You will be unable to use section 5.")
    pass
#Turn off deprecation warnings

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)<jupyter_output><empty_output><jupyter_text>2. Load DataThe data for this example can be downloaded at this [link](https://www.dropbox.com/sh/n4924ipcfjqc0t6/AACPWjxDKPEzQiXKUUFriFkJa?dl=0&preview=example_data_s1.pickle). It was recorded by <NAME> from Lee Miller's lab at Northwestern.The data that we load is in the format described below. We have another example notebook, "Example_format_data", that may be helpful towards putting the data in this format.Neural data should be a matrix of size "number of time bins" x "number of neurons", where each entry is the firing rate of a given neuron in a given time binThe output you are decoding should be a matrix of size "number of time bins" x "number of features you are decoding"<jupyter_code># folder='' #ENTER THE FOLDER THAT YOUR DATA IS IN
folder='/home/jglaser/Data/DecData/' 
# folder='/Users/jig289/Dropbox/Public/Decoding_Data/'

with open(folder+'example_data_s1.pickle','rb') as f:
#     neural_data,vels_binned=pickle.load(f,encoding='latin1') #If using python 3
    neural_data,vels_binned=pickle.load(f) #If using python 2

# #If you would prefer to load the '.h5' example file rather than the '.pickle' example file.
# data=dd.io.load(folder+'example_data_s1.h5')
# neural_data=data['neural_data']
# vels_binned=data['vels_binned']<jupyter_output><empty_output><jupyter_text>3. Preprocess Data  3A. User InputsThe user can define what time period to use spikes from (with respect to the output).I am using fewer bins in this example than in the manuscript and other examples, to make it run faster<jupyter_code>bins_before=3 #How many bins of neural data prior to the output are used for decoding
bins_current=1 #Whether to use concurrent time bin of neural data
bins_after=3 #How many bins of neural data after the output are used for decoding<jupyter_output><empty_output><jupyter_text>3B. Format Covariates  Format Input Covariates<jupyter_code># Format for recurrent neural networks (SimpleRNN, GRU, LSTM)
# Function to get the covariate matrix that includes spike history from previous bins
X=get_spikes_with_history(neural_data,bins_before,bins_after,bins_current)

# Format for Wiener Filter, Wiener Cascade, XGBoost, and Dense Neural Network
#Put in "flat" format, so each "neuron / time" is a single feature
X_flat=X.reshape(X.shape[0],(X.shape[1]*X.shape[2]))<jupyter_output><empty_output><jupyter_text>Format Output Covariates<jupyter_code>#Set decoding output
y=vels_binned<jupyter_output><empty_output><jupyter_text>3C. Split into training / testing / validation setsNote that hyperparameters should be determined using a separate validation set. Then, the goodness of fit should be be tested on a testing set (separate from the training and validation sets).  User Options<jupyter_code>#Set what part of data should be part of the training/testing/validation sets
#I made the ranges smaller for this example so that the hyperparameter optimization runs faster
training_range=[0.6, 0.7] 
testing_range=[0.7, 0.8]
valid_range=[0.8,0.9]<jupyter_output><empty_output><jupyter_text>Split Data<jupyter_code>num_examples=X.shape[0]

#Note that each range has a buffer of"bins_before" bins at the beginning, and "bins_after" bins at the end
#This makes it so that the different sets don't include overlapping neural data
training_set=np.arange(np.int(np.round(training_range[0]*num_examples))+bins_before,np.int(np.round(training_range[1]*num_examples))-bins_after)
testing_set=np.arange(np.int(np.round(testing_range[0]*num_examples))+bins_before,np.int(np.round(testing_range[1]*num_examples))-bins_after)
valid_set=np.arange(np.int(np.round(valid_range[0]*num_examples))+bins_before,np.int(np.round(valid_range[1]*num_examples))-bins_after)

#Get training data
X_train=X[training_set,:,:]
X_flat_train=X_flat[training_set,:]
y_train=y[training_set,:]

#Get testing data
X_test=X[testing_set,:,:]
X_flat_test=X_flat[testing_set,:]
y_test=y[testing_set,:]

#Get validation data
X_valid=X[valid_set,:,:]
X_flat_valid=X_flat[valid_set,:]
y_valid=y[valid_set,:]<jupyter_output><empty_output><jupyter_text>3D. Process CovariatesWe normalize (z_score) the inputs and zero-center the outputs.Parameters for z-scoring (mean/std.) should be determined on the training set only, and then these z-scoring parameters are also used on the testing and validation sets.<jupyter_code>#Z-score "X" inputs. 
X_train_mean=np.nanmean(X_train,axis=0)
X_train_std=np.nanstd(X_train,axis=0)
X_train=(X_train-X_train_mean)/X_train_std
X_test=(X_test-X_train_mean)/X_train_std
X_valid=(X_valid-X_train_mean)/X_train_std

#Z-score "X_flat" inputs. 
X_flat_train_mean=np.nanmean(X_flat_train,axis=0)
X_flat_train_std=np.nanstd(X_flat_train,axis=0)
X_flat_train=(X_flat_train-X_flat_train_mean)/X_flat_train_std
X_flat_test=(X_flat_test-X_flat_train_mean)/X_flat_train_std
X_flat_valid=(X_flat_valid-X_flat_train_mean)/X_flat_train_std

#Zero-center outputs
y_train_mean=np.mean(y_train,axis=0)
y_train=y_train-y_train_mean
y_test=y_test-y_train_mean
y_valid=y_valid-y_train_mean<jupyter_output><empty_output><jupyter_text>4. Optimize Hyperparameters of decoders using "BayesianOptimization" - The general idea is that we will try to find the decoder hyperparameters that produce the highest R2 values on the validation set.  - We will provide examples for a few decoders (Wiener Cascade, XGBoost, Feedforward Neural Net)  A potential downside of BayesianOptimization is that it optimizes over a continuous space. So if a hyperparameter has integer values, the optimizer may unnecessarily test many nearby values (e.g. 2.05, 2.1, and 2.2) which are all treated the same (as 2), when it could just test the one integer value (2).  4A. Wiener Cascade (Linear Nonlinear Model) - The hyperparameter we are trying to optimize is "degree" (the degree of the polynomial). - Note that a sophisticated hyperparameter optimization technique is not needed for this decoder with a single hyperparameter - you could easily do a grid search. However, we show the example since it's the simplest and runs quickly.  Define a function that returns the metric we are trying to optimize (R2 value of the validation set) as a function of the hyperparameter (degree)<jupyter_code>def wc_evaluate(degree):
    model_wc=WienerCascadeDecoder(degree) #Define model
    model_wc.fit(X_flat_train,y_train) #Fit model
    y_valid_predicted_wc=model_wc.predict(X_flat_valid) #Validation set predictions
    return np.mean(get_R2(y_valid,y_valid_predicted_wc)) #R2 value of validation set (mean over x and y position/velocity)<jupyter_output><empty_output><jupyter_text>Set range of hyperparameters, and run optimization<jupyter_code>#Define Bayesian optimization, and set limits of hyperparameters 
#Here, we set the limit of "degree" to be [1, 6.99], so we test degrees 1,2,3,4,5,6
wcBO = BayesianOptimization(wc_evaluate, {'degree': (1, 6.99)}, verbose=0)
#Set number of initial runs (init_points) and subsequent tests (n_iter), and do the optimization
#kappa is a parameter that sets exploration vs exploitation in the algorithm
#We set kappa=10 (greater than the default) so there is more exploration when there are more hyperparameters
wcBO.maximize(init_points=5, n_iter=5, kappa=10)<jupyter_output><empty_output><jupyter_text>Get best hyperparametersNote that you can also find out more information about each tested hyperparameter in "wcBO.res": (each hyperparameter tested and the resulting R2 value)<jupyter_code>#Print out the best parameters and associated R2 value (called "max_val")
wcBO.res['max']
#Assign the best hyperparameter to a variable
best_params=wcBO.res['max']['max_params'] 
degree=best_params['degree']<jupyter_output><empty_output><jupyter_text>4B. XGBoostThe hyperparameters we are trying to optimize are: - "max_depth" (maximum depth of the trees) - "num_round" (number of trees for fitting) - "eta" (learning rate) Note that this example can be somewhat slow (depending on your computer, potentially 10's of minutes).  Define a function that returns the metric we are trying to optimize (R2 value of the validation set) as a function of the hyperparameters<jupyter_code>def xgb_evaluate(max_depth,num_round,eta):
    #The parameters need to be in the correct format for the decoder, so we do that below
    max_depth=int(max_depth) 
    num_round=int(num_round) 
    eta=float(eta) 
    #Define model
    model_xgb=XGBoostDecoder(max_depth=max_depth, num_round=num_round, eta=eta) 
    model_xgb.fit(X_flat_train,y_train) #Fit model
    y_valid_predicted_xgb=model_xgb.predict(X_flat_valid) #Get validation set predictions
    return np.mean(get_R2(y_valid,y_valid_predicted_xgb)) #Return mean validation set R2<jupyter_output><empty_output><jupyter_text>Set range of hyperparameters, and run optimizationIf you want to keep track of progress, set verbose=1 in the cell below<jupyter_code>#Do bayesian optimization, and set limits of hyperparameters
xgbBO = BayesianOptimization(xgb_evaluate, {'max_depth': (2, 6.99), 'num_round': (100,600.99), 'eta': (0.01, 0.8)},verbose=0) #Define Bayesian optimization, and set limits of hyperparameters    
#Set number of initial runs and subsequent tests, and do the optimization. Also, we set kappa=10 (greater than the default) so there is more exploration when there are more hyperparameters
xgbBO.maximize(init_points=10, n_iter=10, kappa=10)<jupyter_output><empty_output><jupyter_text>Get best hyperparametersNote that you can also find out more information about each tested hyperparameter in "xgbBO.res": (each hyperparameter tested and the resulting R2 value)<jupyter_code>#Print out the best parameters and associated R2 value (called "max_val")
xgbBO.res['max']
#Assign the best hyperparameters to variables, and put them in the correct format
best_params=xgbBO.res['max']['max_params'] #Get the hyperparameters that give rise to the best fit
num_round=np.int(best_params['num_round']) #We want the integer value associated with the best "num_round" parameter (which is what the xgb_evaluate function does above)
max_depth=np.int(best_params['max_depth']) #We want the integer value associated with the best "max_depth" parameter (which is what the xgb_evaluate function does above)
eta=best_params['eta']<jupyter_output><empty_output><jupyter_text>4C. Feedforward (Dense) Neural NetThe hyperparameters we are trying to optimize are: - "num_units" (the number of hidden units in each layer) - "frac_dropout" (the proportion of units that are dropped out" - "n_epochs" (the number of epochs used for fitting)Note that this example can be somewhat slow (depending on your computer, potentially 10's of minutes).  Define a function that returns the metric we are trying to optimize (R2 value of the validation set) as a function of the hyperparameters<jupyter_code>def dnn_evaluate(num_units,frac_dropout,n_epochs):
    #The parameters need to be in the correct format for the decoder, so we do that below
    num_units=int(num_units)
    frac_dropout=float(frac_dropout)
    n_epochs=int(n_epochs)
    #Declare and fit decoder
    model_dnn=DenseNNDecoder(units=[num_units,num_units],dropout=frac_dropout,num_epochs=n_epochs)
    model_dnn.fit(X_flat_train,y_train)
    #Make predictions and get R2 values on validation set
    y_valid_predicted_dnn=model_dnn.predict(X_flat_valid)
    return np.mean(get_R2(y_valid,y_valid_predicted_dnn))<jupyter_output><empty_output><jupyter_text>Set range of hyperparameters, and run optimizationIf you want to keep track of progress, set verbose=1 in the cell below<jupyter_code>#Do bayesian optimization, and set limits of hyperparameters
dnnBO = BayesianOptimization(dnn_evaluate, {'num_units': (50, 700.99), 'frac_dropout': (0,.5), 'n_epochs': (2,15.99)},verbose=0)

#Set number of initial runs (init_points) and subsequent tests (n_iter), and do the optimization
#kappa is a parameter that sets exploration vs exploitation in the algorithm - 10 seems to work pretty welldnnBO = BayesianOptimization(dnn_evaluate, {'num_units': (50, 500), 'frac_dropout': (0.,.5), 'n_epochs': (2,15)})
dnnBO.maximize(init_points=10, n_iter=10, kappa=10)<jupyter_output>/opt/anaconda/anaconda2/lib/python2.7/site-packages/keras/models.py:826: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
  warnings.warn('The `nb_epoch` argument in `fit` '<jupyter_text>Get best hyperparametersNote that you can also find out more information about each tested hyperparameter in "dnnBO.res": (each hyperparameter tested and the resulting R2 value)<jupyter_code>#Print out the best parameters and associated R2 value
dnnBO.res['max']
#Assign the best hyperparameters to variables, and put them in the correct format
best_params=dnnBO.res['max']['max_params']
frac_dropout=float(best_params['frac_dropout'])
n_epochs=np.int(best_params['n_epochs'])
num_units=np.int(best_params['num_units'])<jupyter_output><empty_output><jupyter_text>5. Optimize Hyperparameters of decoders using "Hyperopt - The general idea is that we will try to find the decoder hyperparameters that produce the highest R2 values on the validation set.  - We will provide examples for a few decoders (Wiener Cascade, XGBoost, Feedforward Neural Net)  5A. Wiener Cascade - The hyperparameter we are trying to optimize is "degree" (the degree of the polynomial). - Note that a sophisticated hyperparameter optimization technique is not needed for this decoder with a single hyperparameter - you could easily do a grid search. However, we show the example since it's the simplest.  Define a function that returns the metric we are trying to optimize (R2 value of the validation set) as a function of the hyperparameter (degree) - hyperopt minimizes the parameter, so we will return -R2 (in order to maximize R2)<jupyter_code>def wc_evaluate2(degree):
    
    model_wc=WienerCascadeDecoder(degree) #Define model
    model_wc.fit(X_flat_train,y_train) #Fit model
    y_valid_predicted_wc=model_wc.predict(X_flat_valid) #Validation set predictions
    return -np.mean(get_R2(y_valid,y_valid_predicted_wc)) #-R2 value of validation set (mean over x and y position/velocity)<jupyter_output><empty_output><jupyter_text>Set range of hyperparameters, and run optimization<jupyter_code>#The range of values I'll look at for the parameter
#"hp.quniform" will allow us to look at integer (rather than continuously spaced) values.
#Below we consider values of "degree" starting at 1, going until 6, and spaced at values of 1 (i.e., 1,2,3,4,5,6)
space = hp.quniform('degree', 1, 6, 1)

#object that holds iteration results
trials = Trials()
#Do optimization
#Set the number of evaluations below (10 in this example)
hyperoptBest = fmin(wc_evaluate2, space, algo=tpe.suggest, max_evals=10, trials=trials)<jupyter_output><empty_output><jupyter_text>Get best hyperparametersNote that you can also find out more information about each tested hyperparameter in the "trials" object. "trials.results" will give the R2 value for each hyperparameters tested, and "trials.vals" will give you the values of the hyperparameters.<jupyter_code>print("R2_validation",-trials.best_trial['result']['loss'])
print(hyperoptBest)
degree=hyperoptBest['degree']<jupyter_output>{'degree': 5.0}<jupyter_text>5B. XGBoostThe hyperparameters we are trying to optimize are: - "max_depth" (maximum depth of the trees) - "num_round" (number of trees for fitting) - "eta" (learning rate) Note that this example can be somewhat slow (depending on your computer, potentially 10's of minutes).  Define a function that returns the metric we are trying to optimize (R2 value of the validation set) as a function of the hyperparameter (degree) - hyperopt minimizes the parameter, so we will return -R2 (in order to maximize R2)<jupyter_code>def xgb_evaluate2(params):
    #Put parameters in correct formats
    num_round=int(params['num_round'])
    eta=float(params['eta'])
    max_depth=int(params['max_depth'])
    model_xgb=XGBoostDecoder(max_depth=max_depth, num_round=num_round, eta=eta) #Define model
    model_xgb.fit(X_flat_train,y_train) #Fit model
    y_valid_predicted_xgb=model_xgb.predict(X_flat_valid) #Get validation set predictions
    return -np.mean(get_R2(y_valid,y_valid_predicted_xgb)) #Return mean validation set R2<jupyter_output><empty_output><jupyter_text>Set range of hyperparameters, and run optimization<jupyter_code>#The range of values I'll look at for the parameter
#"hp.quniform" will allow us to look at integer (rather than continuously spaced) values.
#So for "num_round", we are looking at values between 100 and 600 by 50 (100,150,200,...600)
#"hp.uniform" looks at continuously spaced values
space = {
    'eta': hp.uniform('eta', 0.01, 0.8),
    'num_round': hp.quniform('num_round', 100,600,50),
    'max_depth': hp.quniform('max_depth', 2,6,1),
}

#object that holds iteration results
trials = Trials()
#Do optimization
#Set the number of evaluations below (20 in this example)
hyperoptBest = fmin(xgb_evaluate2, space, algo=tpe.suggest, max_evals=20, trials=trials)<jupyter_output><empty_output><jupyter_text>Get best hyperparametersNote that you can also find out more information about each tested hyperparameter in the "trials" object. "trials.results" will give the R2 value for each hyperparameters tested, and "trials.vals" will give you the values of the hyperparameters.<jupyter_code>print("R2_validation",-trials.best_trial['result']['loss'])
print(hyperoptBest)

best_params=hyperoptBest #Just renamed so it was in the same format as I used with BayesOptimization
num_round=np.int(best_params['num_round']) #We want the integer value associated with the best "num_round" parameter (which is what the xgb_evaluate function does above)
max_depth=np.int(best_params['max_depth']) #We want the integer value associated with the best "max_depth" parameter (which is what the xgb_evaluate function does above)
eta=best_params['eta']<jupyter_output>{'num_round': 550.0, 'eta': 0.11823310650011055, 'max_depth': 5.0}<jupyter_text>5C. Feedforward (Dense) Neural NetThe hyperparameters we are trying to optimize are: - "num_units" (the number of hidden units in each layer) - "frac_dropout" (the proportion of units that are dropped out" - "n_epochs" (the number of epochs used for fitting)Note that this example can be somewhat slow (depending on your computer, potentially 10's of minutes).  Define a function that returns the metric we are trying to optimize (R2 value of the validation set) as a function of the hyperparameter (degree) - hyperopt minimizes the parameter, so we will return -R2 (in order to maximize R2)<jupyter_code>def dnn_evaluate2(params):
    #Put parameters in proper format
    num_units=int(params['num_units'])
    frac_dropout=float(params['frac_dropout'])
    n_epochs=int(params['n_epochs'])
    model_dnn=DenseNNDecoder(units=[num_units,num_units],dropout=frac_dropout,num_epochs=n_epochs) #Define model
    model_dnn.fit(X_flat_train,y_train) #Fit model
    y_valid_predicted_dnn=model_dnn.predict(X_flat_valid) #Get validation set predictions
    return -np.mean(get_R2(y_valid,y_valid_predicted_dnn)) #Return -R2 value of validation set<jupyter_output><empty_output><jupyter_text>Set range of hyperparameters, and run optimization<jupyter_code>#The range of values I'll look at for the parameter
#"hp.quniform" will allow us to look at integer (rather than continuously spaced) values.
#So for "num_units", we are looking at values between 50 and 700 by 10 (50,60,70,...700)
#"hp.uniform" looks at continuously spaced values
space = {
    'frac_dropout': hp.uniform('frac_dropout', 0., 0.5),
    'num_units': hp.quniform('num_units', 50,700,10),
    'n_epochs': hp.quniform('n_epochs', 2,15,1),
}

#object that holds iteration results
trials = Trials()
#Do optimization
#Set the number of evaluations below (20 in this example)
hyperoptBest = fmin(dnn_evaluate2, space, algo=tpe.suggest, max_evals=20, trials=trials)<jupyter_output><empty_output><jupyter_text>Get best hyperparametersNote that you can also find out more information about each tested hyperparameter in the "trials" object. "trials.results" will give the R2 value for each hyperparameters tested, and "trials.vals" will give you the values of the hyperparameters.<jupyter_code>print("R2_validation",-trials.best_trial['result']['loss'])
print(hyperoptBest)

best_params=hyperoptBest #Just renamed so it was in the same format as I used with BayesOptimization
frac_dropout=float(best_params['frac_dropout'])
n_epochs=np.int(best_params['n_epochs'])
num_units=np.int(best_params['num_units'])<jupyter_output>{'frac_dropout': 0.35466311053401717, 'num_units': 550.0, 'n_epochs': 11.0}<jupyter_text>6. Use the optimal hyperparameters to fit the decoder on the test setThis can be run after running either section 4 or section 5 (both don't need to be run).  6A. Wiener Cascade<jupyter_code>#"degree" was determined during hyperparameter optimization
model_wc=WienerCascadeDecoder(degree) #Declare model w/ fit hyperparameter
model_wc.fit(X_flat_train,y_train) #Fit model on training data
y_test_predicted_wc=model_wc.predict(X_flat_test) #Get test set predictions
#Print R2 values on test set
R2s_wc=get_R2(y_test,y_test_predicted_wc)
print('R2s_wc:', R2s_wc)<jupyter_output>('R2s_wc:', array([ 0.65910948,  0.71765739]))<jupyter_text>6B. XGBoost<jupyter_code># Run model w/ above hyperparameters
model_xgb=XGBoostDecoder(max_depth=max_depth, num_round=num_round, eta=eta) #Declare model w/ fit hyperparameters
model_xgb.fit(X_flat_train,y_train) #Fit model
y_test_predicted_xgb=model_xgb.predict(X_flat_test) #Get test set predictions
#Print R2 values on test set
R2s_xgb=get_R2(y_test,y_test_predicted_xgb)
print('R2s:', R2s_xgb)<jupyter_output>('R2s:', array([ 0.71761818,  0.7576485 ]))<jupyter_text>6C. Feedforward Neural Net<jupyter_code># Run model w/ above hyperparameters
model_dnn=DenseNNDecoder(units=[num_units,num_units],dropout=frac_dropout,num_epochs=n_epochs) #Declare model w/ fit hyperparameters
model_dnn.fit(X_flat_train,y_train) #Fit model
y_test_predicted_dnn=model_dnn.predict(X_flat_test) #Get test set predictions
#Print R2 values on test set
R2s_dnn=get_R2(y_test,y_test_predicted_dnn)
print('R2s:', R2s_dnn)<jupyter_output>('R2s:', array([ 0.76140749,  0.80849555]))
{'fraction_non_alphanumeric': 0.08551630217095249, 'fraction_numerical': 0.017784186493631338, 'mean_word_length': 6.481869943062631, 'pattern_counts': {'":': 3, '<': 111, '<?xml version=': 0, '>': 111, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '499366', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 8217, 'n_tokens_neox': 7793, 'n_words': 2630, 'path': 'Example_hyperparam_opt.ipynb', 'repo_name': 'joeyo/Neural_Decoding', 'size': '24975'}