<jupyter_start><jupyter_text>ResultsIn this notebook we will learn the following about the Results class returned a Benchmark evaluation:1. **Result Creation**: How to create Results from a Benchmark and Learners.2. **Result Plotting**: How to create plots of the learners that were evaluted by a benchmark3. **Result Saving/Loading**: How to save and load results to and from disk4. **Result to Pandas**: How to turn Result into a collection of Pandas dataframes for more in-depth analysis Result Creation**Results** are created whenever Learners are evaluated on a Benchmark. Below is a simple example of this:<jupyter_code>from coba.simulations import ValidationSimulation
from coba.learners    import RandomLearner, VowpalLearner
from coba.benchmarks  import Benchmark

result = Benchmark([ValidationSimulation()]).evaluate([RandomLearner(), VowpalLearner()])<jupyter_output>2021-06-25 20:43:24 Processing chunk...
2021-06-25 20:43:24   * Creating source 0 from ValidationSimulation(cf=True,af=True,seed=1000)... (0.0 seconds)
2021-06-25 20:43:24   * Creating simulation 0 from source 0... (0.0 seconds)
2021-06-25 20:43:24   * Evaluating learner 1 on Simulation 0... (0.24 seconds)
2021-06-25 20:43:24   * Evaluating learner 0 on Simulation 0... (0.02 seconds)<jupyter_text>Result PlottingThe **Result** class is capable of creating two separate plots: 1) `plot_learners` and `plot_shuffles`.  Plot LearnersPerhaps the most straightforward question after running a Benchmark is asking which Learners performed best.To answer this question the **Result** has a `plot_learners` method.<jupyter_code>from coba.simulations import ValidationSimulation
from coba.learners    import RandomLearner, VowpalLearner
from coba.benchmarks  import Benchmark

result = Benchmark([ValidationSimulation()], shuffle=[1,2,3,4]).evaluate([RandomLearner(), VowpalLearner()])

result.plot_learners()<jupyter_output>2021-06-25 20:43:31 Processing chunk...
2021-06-25 20:43:31   * Creating source 0 from ValidationSimulation(cf=True,af=True,seed=1000)... (0.0 seconds)
2021-06-25 20:43:31   * Creating simulation 0 from source 0... (0.0 seconds)
2021-06-25 20:43:31   * Evaluating learner 1 on Simulation 0... (0.23 seconds)
2021-06-25 20:43:31   * Evaluating learner 0 on Simulation 0... (0.03 seconds)
2021-06-25 20:43:31   * Creating simulation 1 from source 0... (0.0 seconds)
2021-06-25 20:43:31   * Evaluating learner 1 on Simulation 1... (0.23 seconds)
2021-06-25 20:43:31   * Evaluating learner 0 on Simulation 1... (0.03 seconds)
2021-06-25 20:43:31   * Creating simulation 2 from source 0... (0.0 seconds)
2021-06-25 20:43:32   * Evaluating learner 1 on Simulation 2... (0.24 seconds)
2021-06-25 20:43:32   * Evaluating learner 0 on Simulation 2... (0.02 seconds)
2021-06-25 20:43:32   * Creating simulation 3 from source 0... (0.0 seconds)
2021-06-25 20:43:32   * Evaluating learner 1 on Simulation 3... (0[...]<jupyter_text>The `plot_learners` method has many arguments which allow customization of the plot.```python"""Args:    source_pattern: The pattern to match when determining which simulations to include in the plot. The "source"         matched against is either the "source" column in the simulations table or the first item in the list in         the simulation 'pipes' column. The simulations can be seen most easily by Result.simulations.to_pandas().    learner_pattern: The pattern to match against the 'full_name' column in learners to determine which learners        to include in the plot. In the case of multiple matches only the last match is kept. The learners table in        Result can be examined via result.learners.to_pandas().    span: In general this indicates how many previous evaluations to average together. In practice this works        identically to ewm span value in the Pandas API. Additionally, if span equals None then all previous         rewards are averaged together and that value is plotted. Compare this to span = 1 WHERE only the current         reward is plotted for each interaction.    start: Determines at which interaction the plot will start at. If start is greater than 1 we assume start is        an interaction index. If start is less than 1 we assume start is the percent of interactions to skip        before starting the plot.    err_every: Determines frequency of bars indicating the standard deviation of the population should be drawn.         Standard deviation gives a sense of how well the plotted average represents the underlying distribution.         Standard deviation is most valuable when plotting against multiple simulations. If plotting against a single         simulation standard error may be a more useful indicator of confidence. The value for sd_every should be        between 0 to 1 and will determine how frequently the standard deviation bars are drawn.    err_type: Determines what the error bars are. Valid types are `None`, 'se', and 'sd'. If err_type is None then         plot will use SEM when there is only one source simulation otherwise it will use SD. Otherwise plot will        display the standard error of the mean for 'se' and the standard deviation for 'sd'."""```For example, we can examine learner performance at the end of the Benchmark and compare smoothed reward to their instantaneous reward.<jupyter_code>result.plot_learners(start=0.8, end=1.0, span=1 , err_every=0                 )
result.plot_learners(start=0.8, end=1.0, span=40, err_every=0.1, err_type='se')<jupyter_output><empty_output><jupyter_text>Plot ShufflesIn addition to plotting average Learner performance we can also zoom in on a single learner and superimpose its average performance on top of each of the individual shuffles that we included in our Benchmark. In our current results we have 4 shuffle seeds so there are four different performance results which are averaged together.The `plot_shuffles` command has all the same arguments are `plot_learners`. Additionally, `plot_shuffles` assumes there's only one source of shuffles and one learner. For this reason you should pass in a source pattern and a learner pattern if there are multiple learners or simulation sources contained in the result.<jupyter_code>result.plot_shuffles(learner_pattern="vw")<jupyter_output><empty_output><jupyter_text>Result Saving/LoadingSaving and restoring a **Result** is the easiest way to archive and share results with others. Saving is done via Benchmark `evaluation`. The following code block will save the Result of the benchmark evaluation in result.log<jupyter_code>from coba.simulations import ValidationSimulation
from coba.learners    import RandomLearner, VowpalLearner
from coba.benchmarks  import Benchmark

simulations = [ValidationSimulation()          ]
learners    = [RandomLearner(), VowpalLearner()]

result = Benchmark(simulations, shuffle=[1,2,3]).evaluate(learners, "result.log")<jupyter_output>2021-06-25 20:43:49 Processing chunk...
2021-06-25 20:43:49   * Creating source 0 from ValidationSimulation(cf=True,af=True,seed=1000)... (0.0 seconds)
2021-06-25 20:43:49   * Creating simulation 0 from source 0... (0.0 seconds)
2021-06-25 20:43:50   * Evaluating learner 1 on Simulation 0... (0.23 seconds)
2021-06-25 20:43:50   * Evaluating learner 0 on Simulation 0... (0.03 seconds)
2021-06-25 20:43:50   * Creating simulation 1 from source 0... (0.0 seconds)
2021-06-25 20:43:50   * Evaluating learner 1 on Simulation 1... (0.23 seconds)
2021-06-25 20:43:50   * Evaluating learner 0 on Simulation 1... (0.02 seconds)
2021-06-25 20:43:50   * Creating simulation 2 from source 0... (0.0 seconds)
2021-06-25 20:43:50   * Evaluating learner 1 on Simulation 2... (0.23 seconds)
2021-06-25 20:43:50   * Evaluating learner 0 on Simulation 2... (0.03 seconds)<jupyter_text>After a result has been saved it can be restored using `Result.from_file`.<jupyter_code>from coba.benchmarks import Result
Result.from_file('result.log').plot_learners()<jupyter_output><empty_output><jupyter_text>Result To PandasFinally, if the plotting functionality just shared isn't enough it is possible to extract all the data stored in Result into dataframes.<jupyter_code>from coba.benchmarks import Result
Result.from_file('result.log').simulations.to_pandas()
from coba.benchmarks import Result
Result.from_file('result.log').learners.to_pandas()
from coba.benchmarks import Result
Result.from_file('result.log').interactions.to_pandas()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08377896613190731, 'fraction_numerical': 0.06702317290552584, 'mean_word_length': 4.5223097112860895, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '44121', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2790, 'n_tokens_neox': 2367, 'n_words': 995, 'path': 'examples/notebooks/Results.ipynb', 'repo_name': 'milena-mathew/coba-CCPO', 'size': '8415'}