<jupyter_start><jupyter_text>Using Researcher With Tensorflow MNIST`researcher` makes it easier for you to record and visualize and reproduce the results of your data science experiments. Below we'll demonstrate how this can be done by using researcher to compare the results of different experiments on the tensorflow MNIST dataset (based off the [tensorflow MNIST tutorial](https://www.tensorflow.org/datasets/keras_example)).<jupyter_code>import tensorflow as tf
import tensorflow_datasets as tfds
import researcher as rs<jupyter_output><empty_output><jupyter_text>Prepare the dataFirst we have to download the MNIST dataset and split it into train and test subsets.<jupyter_code>(ds_train, ds_test), ds_info = tfds.load(
    'mnist',
    split=['train', 'test'],
    shuffle_files=True,
    as_supervised=True,
    with_info=True,
)
def normalize_img(image, label):
    """Normalizes images: `uint8` -> `float32`."""
    return tf.cast(image, tf.float32) / 255., label

ds_train = ds_train.map(
    normalize_img, num_parallel_calls=tf.data.experimental.AUTOTUNE)
ds_train = ds_train.cache()
ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)
ds_train = ds_train.batch(128)
ds_train = ds_train.prefetch(tf.data.experimental.AUTOTUNE)
ds_test = ds_test.map(
    normalize_img, num_parallel_calls=tf.data.experimental.AUTOTUNE)
ds_test = ds_test.batch(128)
ds_test = ds_test.cache()
ds_test = ds_test.prefetch(tf.data.experimental.AUTOTUNE)<jupyter_output><empty_output><jupyter_text>Running ExperimentsNow we're ready to start training. It's not good enough to just create a model and jump right in though, good data science requires planning. First we need to think about what statement we're trying to prove, for instance, perhaps we suspect that [ReLU](https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/) activation functions perform better on MNIST than [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) functions.Next we need to determine how we would prove this statement one way or the other. A simple, scientific approach would be to run a series of experiemnts where we train a neural network to recognize MNIST digits, keeping all variables exactly the same except the activation function, and record the final test loss each time. If one activation function has a much lower test loss, then we'll know that that activation function is better for MINST (at least under these specific conditions).The first step is to define an experiment helper function that will help us re-run exactly the same experiment while varying only the activation function:<jupyter_code>def run_experiment(params):
    model = tf.keras.models.Sequential([
      tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
      tf.keras.layers.Dense(128,activation=params["activation_function"]), # activation function is chosen from params
      tf.keras.layers.Dense(10, activation='softmax')
    ])
    model.compile(
        loss='sparse_categorical_crossentropy',
        optimizer=tf.keras.optimizers.Adam(0.001),
        metrics=['accuracy'],
    )
    history = model.fit(
        ds_train,
        epochs=5,
        validation_data=ds_test,
    )
    
    return history<jupyter_output><empty_output><jupyter_text>This function takes a set of parameters, creates a neural network according to those parameters, trains it on MNIST, and finally returns the results of the training.Let's try it out.<jupyter_code>param_queue = [
    {"activation_function": "relu"},
    {"activation_function": "sigmoid"},
    
]

results = []

for params in param_queue:
    print("\nStarting Next Experiment")
    results.append(run_experiment(params))<jupyter_output>Starting Next Experiment
Epoch 1/5
469/469 [==============================] - 1s 2ms/step - loss: 0.3461 - accuracy: 0.9058 - val_loss: 0.1848 - val_accuracy: 0.9442
Epoch 2/5
469/469 [==============================] - 1s 1ms/step - loss: 0.1618 - accuracy: 0.9530 - val_loss: 0.1381 - val_accuracy: 0.9599
Epoch 3/5
469/469 [==============================] - 1s 2ms/step - loss: 0.1187 - accuracy: 0.9658 - val_loss: 0.1065 - val_accuracy: 0.9687
Epoch 4/5
469/469 [==============================] - 1s 1ms/step - loss: 0.0925 - accuracy: 0.9730 - val_loss: 0.0981 - val_accuracy: 0.9705
Epoch 5/5
469/469 [==============================] - 1s 2ms/step - loss: 0.0750 - accuracy: 0.9782 - val_loss: 0.0892 - val_accuracy: 0.9733

Starting Next Experiment
Epoch 1/5
469/469 [==============================] - 1s 2ms/step - loss: 0.5951 - accuracy: 0.8562 - val_loss: 0.3031 - val_accuracy: 0.9171
Epoch 2/5
469/469 [==============================] - 1s 2ms/step - loss: 0.2733 - accuracy: 0.9233 - v[...]<jupyter_text>Saving Experiments with `researcher`Now, just by eyeballing the printed output we can see that the ReLU function seems to perform better than the sigmoid function. To get the most out of this experiment though, we need to record the results. If in 6 months a colleague or client asks us which activation to use for an MNIST classifier, we don't want to say "use ReLU, because if I remember correctly, it seemed a bit better than sigmoid". We want to be able to show them the actual data we gathered, so that they know **exactly** how much better, and according to which metrics. We also want to give them the tools to easily re-run the experiment for themselves if they have any doubts.This is what researcher helps us do: for each experiment we'll save the parameters and the associated experiment outcome to an "experiment" JSON file, then in future we can easily re-analyse or re-create the results.<jupyter_code># create a directory for storing experiments
! mkdir experiments
experiment_dir = "experiments/"

for i in range(len(param_queue)):
    params = param_queue[i]
    collector = rs.ObservationCollector()
    
    collector.add_tensorflow_history(0, results[i])
    
    rs.record_experiment_with_collector(params, experiment_dir, collector)
! ls experiments<jupyter_output>no_title_91601d41e6ba315dd57bd9c2b1db378e.json
no_title_eebb49b9d1487396dd6c0e5271cc3083.json<jupyter_text>`researcher`automatically associates each experiment with a hash. We can easily load them again and visualize their outcomes.<jupyter_code>experiments = rs.all_experiments(experiment_dir)
experiments
experiments[0].data
rs.final_compare(experiments, ["loss", "accuracy", "val_loss", "val_accuracy"])
rs.plot_training(experiments, ["loss", "val_loss"], figsize=(10, 10))<jupyter_output><empty_output><jupyter_text>Adding More ExperimentsIf we want to do more research, we simply run and save more experiments. We can define a function to make this process easier.<jupyter_code>def run_and_save(params):
    results = run_experiment(params)
    
    collector = rs.ObservationCollector()
    
    collector.add_tensorflow_history(0, results)
        
    rs.record_experiment_with_collector(params, "./experiments/", collector)<jupyter_output><empty_output><jupyter_text>Now running and recording new experiments becomes very simple. We're also going to give our experiment parameters a `"title"` now, to help us identify that experiment in future.<jupyter_code>run_and_save({
    "title": "tanh-activation",
    "activation_function": "tanh"
})
rs.final_compare(experiments, ["loss", "accuracy", "val_loss", "val_accuracy"])
experiments = rs.all_experiments("experiments/")
rs.plot_training(experiments, ["loss", "val_loss"], figsize=(10, 10))<jupyter_output><empty_output><jupyter_text>The tanh function seems to perform better than the sigmoid function, but worse than ReLU. Now that we have this data recorded, we can come back and re-confirm or re-assess these conclusions any time we like.Since the parameters for each experiment are also recorded, we can easily re-run experiments later to confirm that the results haven't changed (for example, we might want to confirm that the results still hold on a newer version of Tensorflow), or on someone else's machine.<jupyter_code>experiments[-1].data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11337137186802282, 'fraction_numerical': 0.03485487472091293, 'mean_word_length': 5.053303303303303, 'pattern_counts': {'":': 4, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '233410', 'license': 'MIT', 'n_tokens_mistral': 2622, 'n_tokens_neox': 2376, 'n_words': 811, 'path': 'mnist_demo.ipynb', 'repo_name': 'Lewington-pitsos/researcher', 'size': '8062'}