<jupyter_start><jupyter_text>Benchmarking with sktimeThe benchmarking modules allows you to* orchestrate prediction experiments of one or more learning strategies on one or more datasets in an automated way,* evaluate and statistically compare predictive performances of different strategies over multiple datasets.[Github weblink](https://github.com/alan-turing-institute/sktime/blob/master/examples/benchmarking.ipynb)  Preliminaries<jupyter_code>import os

from sklearn.metrics import accuracy_score

from sktime.benchmarking.data import UEADataset, make_datasets
from sktime.benchmarking.evaluation import Evaluator
from sktime.benchmarking.metrics import PairwiseMetric
from sktime.benchmarking.orchestration import Orchestrator
from sktime.benchmarking.results import HDDResults
from sktime.classifiers.compose.ensemble import TimeSeriesForestClassifier
from sktime.highlevel.strategies import TSCStrategy
from sktime.highlevel.tasks import TSCTask
from sktime.model_selection import PresplitFilesCV<jupyter_output><empty_output><jupyter_text>Setup<jupyter_code># set paths to data and results folder
import sktime
REPOPATH = os.path.dirname(sktime.__file__)
DATAPATH = os.path.join(REPOPATH, "datasets/data/")
RESULTSPATH = "results"<jupyter_output><empty_output><jupyter_text>Create pointers to datasets on hard driveHere we use the `UEADataset` which follows the [UEA/UCR format](http://www.timeseriesclassification.com) and some of the time series classification datasets included in sktime.<jupyter_code># Create individual pointers to dataset on the disk
datasets = [
    UEADataset(path=os.path.join(DATAPATH, "GunPoint"), name="GunPoint"),
    UEADataset(path=os.path.join(DATAPATH, "ItalyPowerDemand"), name="ItalyPowerDemand"),
    UEADataset(path=os.path.join(DATAPATH, "ArrowHead"), name="ArrowHead")
]

# Alternatively, we can use a helper function to create them automatically
datasets = make_datasets(path=DATAPATH, dataset_cls=UEADataset, 
                         names=["GunPoint", "ItalyPowerDemand", "ArrowHead"])<jupyter_output><empty_output><jupyter_text>For each dataset, we also need to specify a learning taskIn this case, all tasks are the same, because the target variable has the same name for all datasets.<jupyter_code>tasks = [TSCTask(target="target") for _ in range(len(datasets))]<jupyter_output><empty_output><jupyter_text>Specify learning strategies<jupyter_code># Specify learning strategies
strategies = [
    TSCStrategy(TimeSeriesForestClassifier(n_estimators=10), name="tsf10"),
    TSCStrategy(TimeSeriesForestClassifier(n_estimators=20), name="tsf20")
]<jupyter_output><empty_output><jupyter_text>Set up a results objectThe results object encapsulates where and how benchmarking results are stored, here we choose to output them to the hard drive.<jupyter_code># Specify results object which manages the output of the benchmarking
results = HDDResults(path=RESULTSPATH)<jupyter_output><empty_output><jupyter_text>Run benchmarking<jupyter_code># run orchestrator
orchestrator = Orchestrator(datasets=datasets,
                            tasks=tasks,  
                            strategies=strategies, 
                            cv=PresplitFilesCV(), 
                            results=results)
 
orchestrator.fit_predict(save_fitted_strategies=False, overwrite_predictions=True)<jupyter_output><empty_output><jupyter_text>Evaluate and compare resultsHaving run the orchestrator, we can evaluate and compare the prediction strategies.<jupyter_code>evaluator = Evaluator(results)
metric = PairwiseMetric(func=accuracy_score, name="accuracy")
metrics_by_strategy = evaluator.evaluate(metric=metric)
metrics_by_strategy.head()<jupyter_output><empty_output><jupyter_text>The evaluator offers a number of additional methods for evaluating and comparing strategies, including statistical hypothesis tests and visualisation tools, for example:<jupyter_code>evaluator.rank()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0785008863003292, 'fraction_numerical': 0.002025829323879463, 'mean_word_length': 6.397003745318352, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '97872', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1179, 'n_tokens_neox': 1149, 'n_words': 291, 'path': 'examples/benchmarking.ipynb', 'repo_name': 'jasonlines/sktime-kernels', 'size': '3949'}