<jupyter_start><jupyter_text>Introduction to Research Module This tutorial introduces `Research` functionality of `BatchFlow`.[`Research`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Research) class allows you to easily evaluate multiple parallel experiments with different parameters combinations.It includes instruments to* describe complex domains to generate parameters configurations,* create flexible experiment plan as a sequence of callables, generators and pipelines,* parallelize experiments by processes and GPUs,* share heavy processing between experiments,* save and load results of experiments in a unified form. We start with some useful imports and constant definitions<jupyter_code>import sys
sys.path.append('../../..')

from batchflow.research import Research, Option, Domain, EC, O, EP<jupyter_output><empty_output><jupyter_text>The simplest example In the simplest case `Research` can run [one callable](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Research.add_callable)  and save results (output). Of course, such example is meaningless but we will complicate it gradually to demonstrate how to work with `Research`.Experiment in research is a sequence of calls of some methods that can be chained.<jupyter_code>def power(a=2, b=3):
    return a ** b

research = Research().add_callable(power, save_to='power')
research.run(dump_results=False)<jupyter_output>100%|██████████| 1/1 [00:00<00:00,  5.01it/s]<jupyter_text>By default, `Research` creates folder and stores its results there but we specify `dump_results=False` to store results in RAM (see [tutorial 3](https://github.com/analysiscenter/batchflow/blob/research/examples/tutorials/research/03_results_processing.ipynb) for more details).The results can be seen in a special table even during the research execution (see `detach` parameter of [`run`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Research.run)). They are stored in `research.results` which can be transformed to `pandas.DataFrame` by calling `research.results.df` property:<jupyter_code>research.results.df<jupyter_output><empty_output><jupyter_text>In that case experiment has one iteration of function `power` execution and saving the results under the name `power` (why do we need iteration we will descibe below). In the column `id` we can see unique experiment id. The same callable with the same parameters can be added in several ways:<jupyter_code>research = Research().add_callable(power, save_to='power', a=3, b=2)
research = Research().add_callable(power, save_to='power', args=[3, 2])
research = Research().power(3, 2, save_to='power')
research = Research().power(a=3, b=2, save_to='power')

research.run(dump_results=False)

research.results.df<jupyter_output>100%|██████████| 1/1 [00:00<00:00,  4.28it/s]<jupyter_text>Now we get new id for the experiment but with the same first 8 digits. The first 8 digits is a hash of experiment config (here we have empty config), the rest is randomly sampled digits. To make all random generators in `Research` reproducible, define `seed` parameter of `run`.  Constructing domains Let's demonstrate how to create domains of parameters. [`Domain`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Domain) is a special class to describe parameters configurationsto try in experiment.<jupyter_code>domain = Domain(a=[2, 3], b=[3, 2])<jupyter_output><empty_output><jupyter_text>In that case we will get all possible combinations of `a` and `b` (totally four configs).<jupyter_code>research = Research(domain=domain).power(a=EC('a'), b=EC('b'), save_to='power')
research.run(dump_results=False)<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  7.87it/s]<jupyter_text>[`EC`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.EC) (abbreviation for "experiment config") is a named expression to refer to items of config which will be assigned to experiment. In general, named expression is a way to refer to objects that doesn't exist at the moment of the definition. Thus, `EC('key')` stands for experiment config item, `EC()` without args stands for the entire experiment config.The most common named expression is [`E`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.E) which allows to get [`Experiment`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Experiment) instance, thereby gaining access to all the attributes of the current experiment. For example, `EC()` is an alias for `E().config`. In results we can find two new columns: `a` and `b` for values of parameters in config.<jupyter_code>research.results.df<jupyter_output><empty_output><jupyter_text>The same domain can be created in several other ways:<jupyter_code>domain = Domain({'a': [2, 3], 'b': [3, 2]})
domain = Domain(a=[2, 3]) * Domain(b=[3, 2])
domain = Option('a', [2, 3]) * Option('b', [3, 2])<jupyter_output><empty_output><jupyter_text>To concat domains use `+`:<jupyter_code>domain = Domain(a=[2, 3]) + Domain(b=[3, 2])

research = Research(domain=domain).power(kwargs=EC(), save_to='power')
research.run(dump_results=False)

research.results.df<jupyter_output>0it [00:01, ?it/s]/4 [00:00<00:00,  3.88it/s]
0it [00:01, ?it/s]/4 [00:00<00:00,  4.97it/s]
100%|██████████| 4/4 [00:00<00:00,  6.20it/s]<jupyter_text>Here we specify only one parameter in config, for the second the default value from function is used (`a=2`, `b=3`). Besides, we can get "scalar product" of domains with one parameter of the same length:<jupyter_code>domain = Domain(a=[4, 5]) @ Domain(b=[1, 2])

research = Research(domain=domain).power(kwargs=EC(), save_to='power')
research.run(dump_results=False)

research.results.df<jupyter_output>0it [00:11, ?it/s]
0it [00:20, ?it/s]
100%|██████████| 2/2 [00:20<00:00, 10.22s/it]<jupyter_text>Experiment description Let's look at another toy experiment with two callables.<jupyter_code>domain = Domain({'a': [2, 3], 'b': [3, 2]})

def inc(x):
    return x + 1

research = (Research(domain=domain)
            .power(a=EC('a'), b=EC('b'), save_to='power')
            .inc(O('power'), save_to='inc')
           )
research.run(dump_results=False)

research.results.df<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  5.91it/s]<jupyter_text>We use [`O`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.O) named expression to substitute output of the `power` function to `inc` function. We have several columns for each saved value but we can use [`to_df`](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.ResearchResults.to_df) method with `pivot=False` to make two columns for all saved values: `'name'` and `'value'`. `to_df` with default parametrs returns the same as `df` property.<jupyter_code>research.results.to_df(pivot=False)<jupyter_output><empty_output><jupyter_text>Generators in research In addition to callables, we can add [generators](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Research.add_generator)  into `Research`. Now it will become clear why one experiment can have several iterations.<jupyter_code>def inc(x):
    for i in range(2):
        yield x + i

domain = Domain(a=[2, 3], b=[3, 2])
        
research = (Research(domain=domain)
            .add_callable(power, a=EC('a'), b=EC('b'))
            .add_generator(inc, save_to='inc', x=O('power'))
           )
research.run(dump_results=False, finalize=False)

research.results.df<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  5.04it/s]<jupyter_text>Here we have different increments for different iterations of experiment (`finalize` we will describe below). The number of iterations for each experiment is specified in `run` call. By defaults, it is equal to `1` if research contains only callables, and `None` if it has generators. `None` is interpreted as infinity and the experiment will continue until the generator in research is exhausted. We can define the same experiment plan in `Research` in the following way:<jupyter_code>research = (Research(domain=domain)
            .power(a=EC('a'), b=EC('b'))
            .inc(x=O('power'), save_to='inc', mode='generator')
           )

research.run(dump_results=False, finalize=False)

research.results.df<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  4.59it/s]<jupyter_text>The number of callables and generators in `Research` is not limited.<jupyter_code>research = (Research(domain=domain)
            .power(a=EC('a'), b=EC('b'))
            .inc(O('power'), save_to='inc', mode='generator')
            .add_callable('root', power, a=O('inc'), b=0.5, save_to='root', when='last')
           )
research.run(dump_results=False)

research.results.df<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  4.08it/s]<jupyter_text>Here we use default `finalize=True` because at the second iteration generator is exhausted and without that flag expeiment will stop so `root` will not be executed. With `finalize=False` we will get the following results:<jupyter_code>research = (Research(domain=domain)
            .power(a=EC('a'), b=EC('b'))
            .inc(O('power'), save_to='inc', mode='generator')
            .add_callable('root', power, a=O('inc'), b=0.5, save_to='root', when='last')
           )
research.run(dump_results=False, finalize=False)

research.results.df<jupyter_output>100%|██████████| 4/4 [00:01<00:00,  3.70it/s]<jupyter_text>Instances of some class in Research In order to define more complex experiments with interactions between units, we can add to experiments [instances](https://analysiscenter.github.io/batchflow/api/batchflow.research.htmlbatchflow.research.Research.add_instance)  of some classes. They will be initialized with config at the first iteration of experiment and its attributes can be used as callables and generators or in named expressions. To add its attribute as an executable unit, use `{instance_name}.{attr}`.<jupyter_code>class MyCalc:
    def __init__(self, b):
        self.b = b
    
    def power(self, a):
        return a ** self.b

research = (Research(domain=domain)
            .add_instance('calc', MyCalc, b=EC('b'))
            .add_callable('calc.power', a=EC('a'))
            .add_generator(inc, x=O('calc.power'), save_to='inc')
           )

research.run(n_iters=2, dump_results=False, finalize=False)

research.results.df<jupyter_output>100%|██████████| 8/8 [00:01<00:00,  6.66it/s]<jupyter_text>Parallel executions Experiments can be executed in parallel. Here is an example of heavy callable:<jupyter_code>def heavy_callable():
    i = int(2 * 10e7)
    for i in range(i):
        pass
%%timeit

heavy_callable()<jupyter_output>3.57 s ± 215 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Now let's execute it with default parameters.<jupyter_code>%%timeit

research = Research(n_reps=2).heavy_callable()
research.run(dump_results=False, bar=False)<jupyter_output>0it [01:18, ?it/s]
0it [01:17, ?it/s]
0it [01:16, ?it/s]
0it [01:15, ?it/s]
0it [01:14, ?it/s]<jupyter_text>To execute two experiments in parallel, let's define `workers=2`.<jupyter_code>%%timeit

research = Research(n_reps=2).heavy_callable()
research.run(dump_results=False, workers=2, bar=False)<jupyter_output>3.95 s ± 136 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Now we will have two parallel workers to run experiments within the research. And the full execution time is almost twice as fast.  Branches If there is a common part in experiments with different configurations, then it can be taken out into a separate unit and evaluated once for several experiments. To make your researches reproducible, use `random` attribute of the `Expeirment` as a [random generator](https://analysiscenter.github.io/batchflow/api/batchflow.utils_random.html). If the `seed` of `Research` will be fixed, then all samples in that way values will be generated in the same way.<jupyter_code>import numpy as np
from batchflow.research import E

def load_data(random):
    return random.normal(size=10)

def mean(array, use_numpy=True):
    if use_numpy:
        return f"{np.mean(array):.02f} (with numpy)"
    return f"{sum(array) / len(array):.02f} (without numpy)"

domain = Domain({'use_numpy': [False, True]})

research = (Research(domain=domain)
            .add_callable(load_data, random=E().random)
            .add_callable(mean, array=O('load_data'), use_numpy=EC('use_numpy'),
                          save_to='stat')
           )

research.run(n_iters=2, dump_results=False)<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  7.79it/s]<jupyter_text>The resulting dataframe will be the following:<jupyter_code>research.results.df<jupyter_output><empty_output><jupyter_text>As we can see, stats in value columns for different experiment are different. ![Title](img/without_branches.png)  Now let's add `root=True` to `load_data` callable and `branches=2` to `run`:<jupyter_code>research = (Research(domain=domain)
            .add_callable(load_data, random=E().random, root=True)
            .add_callable(mean, array=O('load_data'), use_numpy=EC('use_numpy'),
                          save_to='stat')
           )

research.run(n_iters=2, branches=2, dump_results=False)<jupyter_output>100%|██████████| 4/4 [00:00<00:00,  8.12it/s]<jupyter_text>In that case we will execute `load_data` once for two experiments and then its output will used by mean units in experiments (branches) which will be executed in parallel thread.<jupyter_code>research.results.df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.134965748433173, 'fraction_numerical': 0.02689112374289462, 'mean_word_length': 5.283424908424909, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 81, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '608123', 'license': 'Apache-2.0', 'n_tokens_mistral': 4751, 'n_tokens_neox': 4340, 'n_words': 1266, 'path': 'examples/tutorials/research/01_introduction.ipynb', 'repo_name': 'abrikoseg/batchflow', 'size': '13722'}