<jupyter_start><jupyter_text>Making Re-Usable Code - An Experiment Writeup Example<NAME>, 2019-11-02Tutorial is hosted on [gitlab](https://github.com/UCL/scikit-surgerytf/blob/master/doc/notebooks/tutorial_reusable_code_example.ipynb), and displayed on [readthedocs](https://scikit-surgerytf.readthedocs.io/en/latest/notebooks/tutorial_reusable_code_example.html). (in fact, if you are reading this on readthedocs, the page itself is generated from that jupyter notebook) Pre-requisitesEnsure you have already:* Understood how to use the [PythonTemplate](https://weisslab.cs.ucl.ac.uk/WEISS/SoftwareRepositories/PythonTemplate), and know that we use tox to run ```pylint```, ```pytest``` and ```coverage```. * Done [SNAPPYTutorial 01](https://weisslab.cs.ucl.ac.uk/WEISS/SoftwareRepositories/SNAPPY/SNAPPYTutorial01)* Done [SNAPPYTutorial 02](https://weisslab.cs.ucl.ac.uk/WEISS/SoftwareRepositories/SNAPPY/SNAPPYTutorial02)These tutorials are important, as the same PythonTemplate and hence ```tox``` and ```virtualenv``` layout is used for this project.* In addition, this tutorial follows on from [this one](https://scikit-surgerytf.readthedocs.io/en/latest/notebooks/tutorial_reusable_code_design.html), hosted [here on gitlab](https://github.com/UCL/scikit-surgerytf/blob/master/doc/notebooks/tutorial_reusable_code_design.ipynb). IntroductionThe aim of this notebook is to serve as an example for weekly supervisions with your research supervisor. The researcher should:* be able to run code* be able to write notes and hence this Jupyter notebook is like the traditional lab book. This notebook demonstrates how you might write up a specific experiment. This need not be verbose. You are not writing a scientific paper. Bullet point lists should be sufficient. What about versioning?Question: Do you need to maintain new versions of this workbook for each supervision meeting?Answer: No. Just use version control. Commit code before each presentation to your supervisor. The current version just represents the current state-of-play. You can always go back to previous versions.  Background* Provide links to the most relevant papers.* Why are you doing this experiment?  Hypothesis* Write down any preconditions, assumptions, caveats.* Write down the hypothesis for a valid experiment.  Method* Outline how you will test the hypothesis.* Provide links to data-sources.  Experiments* Include anything else you can think of relevance, before diving into code.<jupyter_code># Jupyter notebook sets the cwd to the folder containing the notebook.
# So, you want to add the root of the project to the sys path, so modules load correctly.
import sys
sys.path.append("../../")
# Import other packages and modules
from sksurgerytf.models import fashion as f
# Run something. Here, we just demonstrate that we can run TensorFlow via Keras API.
# More specifically, we create an instance of an object in this project.
fmn = f.FashionMNIST()<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten_1 (Flatten)          (None, 784)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 128)               100480    
_________________________________________________________________
dense_3 (Dense)              (None, 10)                1290      
=================================================================
Total params: 101,770
Trainable params: 101,770
Non-trainable params: 0
_________________________________________________________________
Train on 60000 samples, validate on 10000 samples
60000/60000 [==============================] - 6s 93us/sample - loss: 0.4989 - accuracy: 0.8252 - val_loss: 0.4266 - val_accuracy: 0.8512
10000/1 - 0s - loss: 0.3458 - accuracy: 0.8512<jupyter_text>ResultsHere's some ideas of what to present, and examples of how to present them. Graphs of LossThe class used above writes tensorboard log files. That might be easier to analyse. Hyperparameter tuningI would make sure my class object could be created with all the right parameters in the constructor. Then I'd write a script, and run it on cluster machines, writing results to log files. Then i'd collect up the log files, extract the results, save some simple .csv table and import it here, using numpy or pandas. So, this workbook would just contain the results, and the analysis of those results. I wouldn't do much hyperparameter tuning from within a notebook.  Example ImagesAs per Andrew Ng's suggestions, pick 100 falsely classified images, display an image thumbnail, along with incorrect result, and the correct result. In this case, we just pick the first 5.<jupyter_code>indexes, images, predicted, labels = fmn.extract_failures(5)
print(indexes)
print(predicted)
print(labels)
%matplotlib inline

import matplotlib.pyplot as plt

number_of_images = len(indexes)

fig = plt.figure(figsize=(20, 150))

for counter in range(number_of_images):
    a = fig.add_subplot(1, number_of_images, counter+1)
    imgplot = plt.imshow(images[counter])      
    a.set_title("Classified as:" + str(predicted[counter]) + ", instead of:" + str(labels[counter]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10389367278172969, 'fraction_numerical': 0.023212280044926994, 'mean_word_length': 4.890848952590959, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '214956', 'license': 'Apache-2.0', 'n_tokens_mistral': 1537, 'n_tokens_neox': 1389, 'n_words': 567, 'path': 'docs/notebooks/tutorial_reusable_code_example.ipynb', 'repo_name': 'UCL/scikit-surgerytf', 'size': '5349'}