<jupyter_start><jupyter_text>Loading data into sktimeThis tutorial outlines time series related file formats and how to load data into sktime.Users can load or convert data into sktime compatible formats in two main ways:* pathway 1: direct loading from time series formats. Data can be loaded directly from a bespoke time series storage format, for instance `.ts` ([see Representing data with .ts files](ts_files)) or other supported file formats, such as Weka `ARFF` and `.tsv` (see [other existing data sources](other_file_types)). * pathway 2: loading in-memory, then conversion. sktime provides functions to convert between common in-memory representations, see `AA_datatypes_and_datasets` tutorial. Hence, data can be loaded via any loader utility (e.g., `pandas.read_csv`) first, then converted manually into one of the sktime compatible specifications, and then converted between specifications using the sktime `convert` or `convert_to` utility.The rest of this tutorial provides descriptions of pathway 1, i.e., how to load data from supported file formats. The .ts file formatOne common use case is to load locally stored data. To make this easy, the `.ts` file format has been created for representing problems in a standard format for use with sktime.  Representing data with .ts filesA .ts file include two main parts:* header information* dataThe header information is used to facilitate simple representation of the data through including metadata about the structure of the problem. The header contains the following:    @problemName     @timeStamps     @univariate     @classLabel      @dataThe data for the problem should begin after the @data tag. In the simplest case where @timestamps is false, values for a series are expressed in a comma-separated list and the index of each value is relative to its position in the list (0, 1, ..., m). An _instance_ may contain 1 to many dimensions, where instances are line-delimited and dimensions within an instance are colon (:) delimited. For example:    2,3,2,4:4,3,2,2    13,12,32,12:22,23,12,32    4,4,5,4:3,2,3,2This example data has 3 _instances_, corresponding to the three lines shown above. Each instance has 2 _dimensions_ with 4 observations per dimension. For example, the intitial instance's first dimension has the timepoint values of 2, 3, 2, 4 and the second dimension has the values 4, 3, 2, 2.Missing readings can be specified using ?. For example,     2,?,2,4:4,3,2,2    13,12,32,12:22,23,12,32    4,4,5,4:3,2,3,2    would indicate the second timepoint value of the initial instance's first dimension is missing. Alternatively, for sparse datasets, readings can be specified by setting @timestamps to true in the header and representing the data with tuples in the form of (timestamp, value) just for the obser. For example, the first instance in the example above could be specified in this representation as:    (0,2),(1,3)(2,2)(3,4):(0,4),(1,3),(2,2),(3,2)Equivalently, the sparser example    2,5,?,?,?,?,?,5,?,?,?,?,4could be represented with just the non-missing timestamps as:    (0,2),(1,5),(7,5),(12,4)When using the .ts file format to store data for timeseries classification problems, the class label for an instance should be specified in the last dimension and @classLabel should be set to true in the header information and be followed by the set of possible class values. For example, if a case consists of a single dimension and has a class value of 1 it would be specified as:     1,4,23,34:1 Loading from .ts file to pandas DataFrameA dataset can be loaded from a .ts file using the following method in sktime.datasets:    load_from_tsfile(full_file_path_and_name, replace_missing_vals_with='NaN')This can be demonstrated using the Arrow Head problem that is included in sktime under sktime/datasets/data<jupyter_code>import os

import sktime
from sktime.datasets import load_from_tsfile

DATA_PATH = os.path.join(os.path.dirname(sktime.__file__), "datasets/data")

train_x, train_y = load_from_tsfile(
    os.path.join(DATA_PATH, "ArrowHead/ArrowHead_TRAIN.ts")
)
test_x, test_y = load_from_tsfile(
    os.path.join(DATA_PATH, "ArrowHead/ArrowHead_TEST.ts")
)<jupyter_output><empty_output><jupyter_text>Train and test partitions of the ArrowHead problem have been loaded into nested dataframes (the `nested_univ` format for panel data) with an associated array of class values. As an example, below are the first 5 rows from the train_x and train_y:<jupyter_code>train_x.head()
train_y[0:5]<jupyter_output><empty_output><jupyter_text>The format of the loaded data can be controlled by the `return_data_type` argument.Allowed strings are identifier strings for sktime compatible `Panel` mtypes, as introduced in the "datatypes and datasets" tutorial (`AA_datatypes_and_datasets`).If provided, the loaded in-memory data container will comply with that type specification.<jupyter_code>train_x, train_y = load_from_tsfile(
    os.path.join(DATA_PATH, "ArrowHead/ArrowHead_TRAIN.ts"), return_data_type="numpy3d"
)
train_x<jupyter_output><empty_output><jupyter_text>Loading other file formatsResearchers who have made timeseries data available have used two other common formats, including:+ Weka ARFF files+ UCR .tsv files Loading from Weka ARFF filesIt is also possible to load data from Weka's attribute-relation file format (ARFF) files. Data for timeseries problems are made available in this format by researchers at the University of East Anglia (among others) at www.timeseriesclassification.com. The `load_from_arff_to_dataframe` method in `sktime.datasets` supports reading data for both univariate and multivariate timeseries problems. The univariate functionality is demonstrated below using data on the ArrowHead problem again (this time loading from ARFF file).<jupyter_code>from sktime.datasets import load_from_arff_to_dataframe

X, y = load_from_arff_to_dataframe(
    os.path.join(DATA_PATH, "ArrowHead/ArrowHead_TRAIN.arff")
)
X.head()<jupyter_output><empty_output><jupyter_text>The multivariate BasicMotions problem is used below to illustrate the ability to read multivariate timeseries data from ARFF files into the sktime format.<jupyter_code>X, y = load_from_arff_to_dataframe(
    os.path.join(DATA_PATH, "BasicMotions/BasicMotions_TRAIN.arff")
)
X.head()<jupyter_output><empty_output><jupyter_text>Loading from UCR .tsv Format FilesA further option is to load data into sktime from tab separated value (.tsv) files. Researchers at the University of Riverside, California make a variety of timeseries data available in this format at https://www.cs.ucr.edu/~eamonn/time_series_data_2018. The `load_from_ucr_tsv_to_dataframe` method in `sktime.datasets` supports reading  univariate problems. An example with ArrowHead is given below to demonstrate equivalence with loading from the .ts and ARFF file formats.<jupyter_code>from sktime.datasets import load_from_ucr_tsv_to_dataframe

X, y = load_from_ucr_tsv_to_dataframe(
    os.path.join(DATA_PATH, "ArrowHead/ArrowHead_TRAIN.tsv")
)
X.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07029253053109911, 'fraction_numerical': 0.01775063902300483, 'mean_word_length': 5.569962686567164, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '442570', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2082, 'n_tokens_neox': 1982, 'n_words': 854, 'path': 'examples/loading_data.ipynb', 'repo_name': 'Tomiiwa/sktime', 'size': '7042'}