<jupyter_start><jupyter_text>Research Research class is intended for multiple running of the same pipelines with different parameters in order to get some statistics. Let's compare `VGG7` and `VGG16` performance on `MNIST` dataset with different layouts of convolutional blocks. For each combination of layout and model class, we will train model for 500 iterations and repeat that process 2 times.<jupyter_code>import sys
import dill
import tensorflow as tf

import matplotlib.pyplot as plt
%matplotlib inline

sys.path.append("../..")
from dataset import Pipeline, B, C, V
from dataset.opensets import MNIST
from dataset.models.tf import VGG7, VGG16
from dataset.research import Research, Option
BATCH_SIZE=64<jupyter_output><empty_output><jupyter_text>Define pipelines Define model config. All parameters that we want to vary we define as ``C('parameter_name')``. In our case it's a `'body/block/layout'`.<jupyter_code>model_config={
    'session/config': tf.ConfigProto(allow_soft_placement=True),
    'inputs': dict(images={'shape': (28, 28, 1)},
                   labels={'classes': 10, 'transform': 'ohe', 'name': 'targets'}),
    'input_block/inputs': 'images',
    'body/block/layout': C('layout'),
    'output/ops': 'accuracy',
    'device': C('device') # it's technical parameter for TFModel
}<jupyter_output><empty_output><jupyter_text>Define loading as a separate pipeline with lazy run. This is optional.<jupyter_code>mnist = MNIST()
train_root = mnist.train.p.run(BATCH_SIZE, shuffle=True, n_epochs=None, lazy=True)
test_root = mnist.test.p.run(BATCH_SIZE, shuffle=True, n_epochs=1, lazy=True)<jupyter_output>Downloading Downloadinghttp://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
 http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
ExtractingDownloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
ExtractingExtracting  /tmp/train-images-idx3-ubyte.gz
/tmp/train-labels-idx1-ubyte.gz
 /tmp/t10k-images-idx3-ubyte.gz
Extracting /tmp/t10k-labels-idx1-ubyte.gz<jupyter_text>Now define main parts of pipelines where we want to vary model:<jupyter_code>train_template = (Pipeline()
            .init_variable('loss', init_on_each_run=list)
            .init_variable('accuracy', init_on_each_run=list)
            .init_model('dynamic', C('model'), 'conv', config=model_config)
            .to_array()
            .train_model('conv', 
                         fetches=['loss', 'output_accuracy'], 
                         feed_dict={'images': B('images'), 'labels': B('labels')},
                         save_to=[V('loss'), V('accuracy')], mode='w')
)
test_template = (Pipeline()
            .init_variable('accuracy', init_on_each_run=list)
            .import_model('conv', C('import_from'))
            .to_array()
            .predict_model('conv', 
                         fetches=['output_accuracy'], 
                         feed_dict={'images': B('images'), 'labels': B('labels')},
                         save_to=[V('accuracy')], mode='a')
)
train_ppl = train_root + train_template
test_ppl = test_root + test_template<jupyter_output><empty_output><jupyter_text>All parameter combinations we define through the dict where a key is a parameter name and value is a list of possible parameter values. Create a grid of parameters in the following way:<jupyter_code>grid = Option('layout', ['cna', 'can']) * Option('model', [VGG7, VGG16])<jupyter_output><empty_output><jupyter_text>You can get all variants of config:<jupyter_code>configs = list(grid.gen_configs())
configs<jupyter_output><empty_output><jupyter_text>Each element is a ConfigAlias. It's a Config dict of parameter values and dict with aliases for parameter values:<jupyter_code>configs[0].config(), configs[0].alias()<jupyter_output><empty_output><jupyter_text>Note that in the first case `model` is a class but in the second `str`.  Create Research object Create an instance of `Research` class and add train and test pipelines and grid of parameters:<jupyter_code>research = (Research()
            .pipeline(train_ppl, variables='loss', name='train')
            .grid(grid))<jupyter_output><empty_output><jupyter_text>Parameter ``name`` defines pipeline name inside ``research``. At each iteration train pipeline will be executed with ``.next_batch()`` and all ``variables`` from the pipeline will be saved so that variables should be added with ``mode='w'``. Test pipeline will be executed with ``.run()`` at each 100 iterations because of parameters ``run=True``  and ``execute='%100'``. `execute` can be `int` (iteration), `str` (`%{step})`) or `list` of `int` and `str`. Pipeline variable ``accuracy`` will be saved after each execution. In order to add a mean value of accuracy on test dataset, you can define a function<jupyter_code>def get_accuracy(iteration, experiment, pipeline):
    import numpy as np
    pipeline = experiment[pipeline].pipeline
    acc = pipeline.get_variable('accuracy')
    return np.mean(acc)

research.function(get_accuracy, returns='accuracy', name='test_accuracy', execute='%100', pipeline='test')<jupyter_output><empty_output><jupyter_text>That function will get iterartion, experiment and kwargs (in that case it's `pipeline='test'`"). Experiment is an OrderedDict for all pipelines and functions that were added to Researchand are running in current job. Key is a name of ExecutableUnit (class for function and pipeline), value is ExecutableUnit.Each pipeline and function added to Research is saved as an ExecutableUnit. Each ExecutableUnithas the following attributes:    function : callable        is None if ExecutableUnit is a pipeline    pipeline : Pipeline        is None if ExecutableUnit is a function    root_pipeline : Pipeline        is None if ExecutableUnit is a function or pipeline is not divided into root and branch    result : dict        current results of the ExecutableUnit. Keys are names of variables (for pipeline)        or returns (for function) values are lists of variable values    path : str        path to the folder where results will be dumped    exec_for : int, list of ints or None    dump_for : int, list of ints or None    to_run : bool    variables : list        variables (for pipeline) or returns (for function)    on_root : bool    args : list    kwargs : dict()  Run research And now you can start research. At each iteration units will be exuted in the same order as they were added into Research (if unit must be executed).<jupyter_code># research.run(n_reps=10, n_iters=1000, name='my_research', progress_bar=True)<jupyter_output><empty_output><jupyter_text>... but don't hurry up if you have a lot of gpus because you can do research much more faster, just define `workers=4` and `gpu = [0, 1, 2, 3]` as a list of available devices. In that case you can run 4 jobs in parallel!<jupyter_code># research.run(n_reps=10, n_iters=1000, workers=4, gpu=[0,1,2,3], name='my_research', progress_bar=True)<jupyter_output><empty_output><jupyter_text>Moreover, if you have heavy loading you can do it just one time for few pipelines with models. In that case you can define research in the following way:<jupyter_code>research = (Research()
    .pipeline(root=train_root, branch=train_template, variables='loss', name='train')
    .pipeline(root=test_root, branch=test_template,
              variables='accuracy', name='test', run=True, execute='%100', import_from='train')
    .grid(grid)
    .function(get_accuracy, returns='accuracy', name='test_accuracy', execute='%100', pipeline='test')
)<jupyter_output><empty_output><jupyter_text>In order to achieve parallelization in branches we added `device` into model_config. And now you can define the number of branches in each worker:<jupyter_code>research.run(n_reps=2, n_iters=1000, workers=2, branches=2, gpu=[0,1,2,3], name='my_research', progress_bar=True)<jupyter_output>Research my_research is starting...<jupyter_text>At last iteration all results will be saved into `{research_name}/results/{config_alias}/{repetition_index}/{unitname}_{iteration}` add a dict pickled by dill.<jupyter_code>import dill

with open('my_research/results/layout_can-model_VGG16/0/train_1000', 'rb') as f:
    result_train = dill.load(f)

with open('my_research/results/layout_can-model_VGG16/0/test_accuracy_1000', 'rb') as f:
    result_test = dill.load(f)

print(result_train.keys()) 
print(result_test.keys())
import matplotlib.pyplot as plt

plt.plot(result_train['iteration'], result_train['loss'])
plt.plot(result_test['iteration'], result_test['accuracy'])
plt.ylim((0,1))
plt.xlim((0,1000))<jupyter_output><empty_output><jupyter_text>Dumping of results and logging By default if unit has `varaibles` or `returns` then results will be dumped at last iteration. But there is unit parameter `dump` that allows to save result not only in the end. It defines as `execute` parameter. For example, dump train results each 200 iterations. Besides, each research has log file. In order to add information about unit execution and dumping into log, define `logging=True`.<jupyter_code>research = (Research()
    .pipeline(root=train_root, branch=train_template, variables='loss', name='train', dump='%200')
    .pipeline(root=test_root, branch=test_template,
              variables='accuracy', name='test', run=True, execute='%100', import_from='train', logging=True)
    .grid(grid)
    .function(get_accuracy, returns='accuracy', name='test_accuracy', execute='%100', pipeline='test')
)
research.run(n_reps=2, n_iters=1000, workers=2, branches=2, gpu=[0,1,2,3], name='my_research', progress_bar=True)<jupyter_output>../../dataset/research/research.py:326: UserWarning: Research with name my_research already exists. That research will be renamed to my_research_1
  "Research with name {} already exists. That research will be renamed to {}".format(name, dirname)<jupyter_text>Functions on root All functions and pipelines if `branches > 0` executed in parallel threads so sometime it can be a problem. In order to allow run function in main thread there exists parameter `on_root`. Function that will be added with `on_root=True` will get `iteration`, `experiments` and `kwargs`. `experiments` is a list of experiments that was defined above (`OrderedDict` of `ExecutableUnits`). Simple example of usage:<jupyter_code>def on_root(iteration, experiments):
    print("On root", iteration)
research = (Research()
    .function(on_root, on_root=True, execute=10, logging=True)
    .pipeline(root=train_root, branch=train_template, variables='loss', name='train')
    .pipeline(root=test_root, branch=test_template,
              variables='accuracy', name='test', run=True, execute='%100', import_from='train', logging=True)
    .grid(grid)
    .function(get_accuracy, returns='accuracy', name='test_accuracy', execute='%100', pipeline='test')
)<jupyter_output><empty_output><jupyter_text>That function will be executed just one time on 10 iteration and will be executed one time for all branches in task.<jupyter_code>research.run(n_reps=1, n_iters=100, workers=2, branches=2, gpu=[0,1,2,3], name='my_research', progress_bar=True)<jupyter_output>../../dataset/research/research.py:326: UserWarning: Research with name my_research already exists. That research will be renamed to my_research_2
  "Research with name {} already exists. That research will be renamed to {}".format(name, dirname)
{'fraction_non_alphanumeric': 0.11516533637400228, 'fraction_numerical': 0.014296991491974389, 'mean_word_length': 5.026427061310782, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '267610', 'license': 'Apache-2.0', 'n_tokens_mistral': 3570, 'n_tokens_neox': 3414, 'n_words': 1075, 'path': 'examples/tutorials/08_research.ipynb', 'repo_name': 'mikhailkin/dataset', 'size': '11401'}