<jupyter_start><jupyter_text>Bayesian hyperparameter searching using Hyperopt- The python libary hyperopt has an implementation of a [Tree Parzen Estimator](https://papers.nips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf) based bayesian hyperoptimizer.Practically speaking the fmin function attempts to optimize the hyperparameter set using the tree parzen estimator (TPE) as a surrogate function for the actual objective function (which involves cross-validating a neural network). This TPE updates its preference for certain paramater values in a bayesian fashion, based on information gained from the last set of hyperparameters chosen and the resulting loss.<jupyter_code>from hyperopt import hp, Trials, fmin, tpe, space_eval, STATUS_OK, STATUS_FAIL
from timeit import default_timer as timer
from statistics import mean<jupyter_output><empty_output><jupyter_text>Experimenting with various search spacesI think it's wise to keep alpha as a uniform distribution. I'll start with normal distributions for the number now neurons in each layer and see where that takes things. 1. Search space 1: normal distributions with large variance.     - layer 1: ~N(mu = 150 neurons, sigma = 65)    - layer 2: ~N(mu = 75 neurons, sigma = 32)    - alpha: uniform {0.1, 0.01,... 10^-7}<jupyter_code># Define the search space. Dictionaries are in key:value where the value is a probabilistic space to 
# "randomly" chose from (I believe as TPE algorithm works, it changes updates probabilistic distributions)
search_space =  hp.choice("layers", [
    {
        "type": "two_layer",
        "k": hp.quniform("2layers_k",1,7,1), # k is selecting for bias alpha, where alpha = 10^-k [0.1, 0.01, ... 10^-7]
        "l1": hp.qnormal("2layers_l1",150, 65, 1), # Discretized normal dist, mu =150, stdev=65
        "l2": hp.qnormal("2layers_l2",75, 32, 1), #  mu =75, stdev=32
        "iters": hp.qnormal("2layers_max_iters", 300, 25, 1)
    },
    {
        "type": "three_layer",
        "k": hp.quniform("3layers_k",1,7,1),
        "l1": hp.qnormal("3layers_l1",150, 65, 1), # Discretized normal dist, mu =150, stdev=65
        "l2": hp.qnormal("3layers_l2",75, 32, 1), # mu =75, stdev=32
        "l3": hp.qnormal("3layers_l3",40, 20, 1), # Discretized normal dist, mu =75, stdev=32
        "iters": hp.qnormal("3layers_max_iters", 300, 35, 1)
    }
])

# Define the optimization function (Costly as it implies fitting a MLP classifier via CV)
def objective(architecture):
    """ The objective function to optimize is the 5-fold cross-validation fitting of a 2 hidden layer NN classifier.
        The hyperoptimizer operates as a minimizer, so the returned loss will be the negative of the avg of 
        mean accuracy.
    """
    # Determine classifer hyperparams from architecture dictionary given
    if "l3" in architecture:
        shape = (2, int(architecture["l1"]), int(architecture["l2"]), int(architecture["l3"]))
    else:
        shape = (2, int(architecture["l1"]), int(architecture["l2"]))
    k = architecture["k"]
    iters = architecture["iters"]
    
    # Make a new MLP Classifier
    cfier = MLPClassifier(max_iter=iters, hidden_layer_sizes=shape, solver= "lbfgs", alpha = pow(10, -k),
                        random_state=25)
    
    # Cross Validatae
    results = cross_validate(cfier, X=train_data, y=train_lbls, cv=5, n_jobs=-1)
    # Results must contain a "loss" value for hyperopt to minimize, and the status
    results["loss"] = -1 * mean(results["test_score"])
    del results["test_score"]
    # For brevity report means
    results["fit_time"] = mean(results["fit_time"])
    results["score_time"] = mean(results["score_time"])
    results["status"] = STATUS_OK

    
    return results
    
# Create a trials Hyperopt Object which stores data regarding the results of our hyperparameter search
trials = Trials()

# Run the hyperparam search. We'll let the hyperopt lib suggest the tpe to use
best_performer = fmin(fn=objective, space=search_space, algo=tpe.suggest, trials=trials, max_evals=100, 
                        rstate= np.random.RandomState(29))<jupyter_output>100%|██████████| 100/100 [00:34<00:00,  2.90trial/s, best loss: -0.6729838709677419]<jupyter_text>Hyperparameter optimization performance<jupyter_code># Analyze Hyperopt Progress
layers = best_performer["layers"] + 2
alpha = pow(10, -1 * best_performer[f"{layers}layers_k"])
iters = best_performer[f"{layers}layers_max_iters"]
l1_neurons = best_performer[f"{layers}layers_l1"]
l2_neurons = best_performer[f"{layers}layers_l2"]

print(f"Best performing architecture")
print(f"\tlayers: {layers}")
print(f"\talpha: {alpha}")
print(f"\tTraining Iterations: {iters}")
print(f"\titerations: {iters}")
print(f"\tl1 neurons: {l1_neurons}")
print(f"\tl2 neurons: {l2_neurons}")

if 3 == layers:
    l3_neurons = best_performer[f"{layers}layers_l3"]
    print(f"\tl3 neurons: {l3_neurons}")

# Plot mean accuracy versus iteration, and train time versus iteration
mean_acc = [abs(res["loss"]) for res in trials.results]
fit_time = [abs(res["fit_time"]) for res in trials.results]

fig, axs = plt.subplots(2, 1, constrained_layout=True)
mean_acc_nafiltered = pd.Series(mean_acc).fillna(value=0) # Replace NaNs with 0's
axs[0].plot(mean_acc_nafiltered, c="cyan")
axs[0].set_title(f"164 PC features: mean acc vs iteration")
axs[0].set_xlabel("Iteration")
axs[0].set_ylabel("Mean Accuracy (%)")
axs[1].plot(fit_time, c="red")
axs[1].set_title(f"164 PC features: mean fit time vs iteration")
axs[1].set_xlabel("Iteration")
axs[1].set_ylabel("Mean fit time (s)")

fig.set_figwidth(12)
fig.set_figheight(10)<jupyter_output>Best performing architecture
	layers: 3
	alpha: 0.01
	Training Iterations: 295.0
	iterations: 295.0
	l1 neurons: 164.0
	l2 neurons: 101.0
	l3 neurons: 37.0<jupyter_text>Training and Testing Optimal Architecture Performance<jupyter_code>from print_conf_mat import *

if 3 == layers:
    shape = (3,int(l1_neurons),int(l2_neurons),int(l3_neurons))
else:
    shape = (2,int(l1_neurons),int(l2_neurons))

cfier = MLPClassifier(max_iter=iters, hidden_layer_sizes=shape, solver="lbfgs", alpha=alpha,
                      random_state=25)
cfier.fit(train_data, train_lbls)

predictions = cfier.predict(test_data)
c_mat = confusion_matrix(test_lbls, predictions)
report = classification_report(test_lbls, predictions)
print(f"Classification Report for 164 PC feature MLP classifier\n {report}")

fig = print_confusion_matrix(c_mat, ["AD", "ND"], figsize=(10,8), fontsize=20, 
                             title="164 Principle Component MLP Classifier")
# y-axis annotation alignment goofs up on some versions of matplitlib.  
# If this is the case, comment/uncomment the following line
fig.gca().set_ylim([0,2])<jupyter_output>Classification Report for 164 PC feature MLP classifier
               precision    recall  f1-score   support

           0       0.44      0.52      0.48        21
           1       0.29      0.22      0.25        18

    accuracy                           0.38        39
   macro avg       0.36      0.37      0.36        39
weighted avg       0.37      0.38      0.37        39
{'fraction_non_alphanumeric': 0.1167298777918247, 'fraction_numerical': 0.048321393454136816, 'mean_word_length': 4.0070323488045005, 'pattern_counts': {'":': 11, '<': 15, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '581341', 'license': 'MIT', 'n_tokens_mistral': 2582, 'n_tokens_neox': 2338, 'n_words': 673, 'path': 'Networks/MLP Classifier using Principle Components.ipynb', 'repo_name': 'mooremk95/AD-Transcriptome-Classifiers', 'size': '7119'}