<jupyter_start><jupyter_text>Univariate time series classification using Signatures/Log signaturesIn this notebook, we explore the classification results of signature/log signature features 114 datasets of the [UCR repository](https://arxiv.org/abs/1810.07758).<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
DATASETS = [
    "ACSF1", "Adiac", "AllGestureWiimoteX", "AllGestureWiimoteY",
    "AllGestureWiimoteZ", "ArrowHead", "Beef", "BeetleFly", "BirdChicken",
    "BME", "Car", "CBF", "Chinatown", "ChlorineConcentration",
    "CinCECGTorso", "Coffee", "Computers", "CricketX", "CricketY",
    "CricketZ", "Crop", "DiatomSizeReduction",
    "DistalPhalanxOutlineAgeGroup", "DistalPhalanxOutlineCorrect",
    "DistalPhalanxTW", "DodgerLoopDay", "DodgerLoopGame",
    "DodgerLoopWeekend", "Earthquakes", "ECG200", "ECG5000", "ECGFiveDays",
    "ElectricDevices", "EOGHorizontalSignal", "EOGVerticalSignal",
    "EthanolLevel", "FaceAll", "FaceFour", "FacesUCR", "FiftyWords",
    "Fish", "FordA", "FordB", "FreezerRegularTrain", "FreezerSmallTrain",
    "Fungi", "GestureMidAirD1", "GestureMidAirD2", "GestureMidAirD3",
    "GesturePebbleZ1", "GesturePebbleZ2", "GunPoint", "GunPointAgeSpan",
    "GunPointMaleVersusFemale", "GunPointOldVersusYoung", "Ham",
    "HandOutlines", "Haptics", "Herring", "HouseTwenty", "InlineSkate",
    "InsectEPGRegularTrain", "InsectEPGSmallTrain", "InsectWingbeatSound",
    "ItalyPowerDemand", "LargeKitchenAppliances", "Lightning2",
    "Lightning7", "Mallat", "Meat", "MedicalImages", "MelbournePedestrian",
    "MiddlePhalanxOutlineAgeGroup", "MiddlePhalanxOutlineCorrect",
    "MiddlePhalanxTW", "MixedShapesRegularTrain", "MixedShapesSmallTrain",
    "MoteStrain", "NonInvasiveFatalECGThorax1",
    "NonInvasiveFatalECGThorax2", "OliveOil", "OSULeaf",
    "PhalangesOutlinesCorrect", "Phoneme", "PickupGestureWiimoteZ",
    "PigAirwayPressure", "PigArtPressure", "PigCVP", "PLAID", "Plane",
    "PowerCons", "ProximalPhalanxOutlineAgeGroup",
    "ProximalPhalanxOutlineCorrect", "ProximalPhalanxTW",
    "RefrigerationDevices", "Rock", "ScreenType", "SemgHandGenderCh2",
    "SemgHandMovementCh2", "SemgHandSubjectCh2", "ShakeGestureWiimoteZ",
    "ShapeletSim", "ShapesAll", "SmallKitchenAppliances", "SmoothSubspace",
    "SonyAIBORobotSurface1", "SonyAIBORobotSurface2", "StarLightCurves",
    "Strawberry", "SwedishLeaf", "Symbols", "SyntheticControl",
    "ToeSegmentation1", "ToeSegmentation2", "Trace", "TwoLeadECG",
    "TwoPatterns", "UMD",
    "UWaveGestureLibraryAll", "UWaveGestureLibraryX",
    "UWaveGestureLibraryY", "UWaveGestureLibraryZ", "Wafer", "Wine",
    "WordSynonyms", "Worms", "WormsTwoClass", "Yoga",
]<jupyter_output><empty_output><jupyter_text>The results are already stored in the Github repository under `results/ucr`, we only need run load them into this notebook.<jupyter_code>def load_results(filename, results_dir='./results/ucr', datasets=DATASETS):
    r = []
    missing = []
    names = []
    for d in DATASETS:
        try:
            data = pd.read_pickle(f'{results_dir}/{d}/{filename}')
            names.append(d)
            r.append(data)
        except FileNotFoundError:
            missing.append(d)
    return (pd.concat(r, keys=names), missing)<jupyter_output><empty_output><jupyter_text>Let's load the results for **log signatures + XGBoost** and **signatures + logistic regression**.The results are obtained using **default** `sklearn.linear_model.LogisticRegression` and `xgboost.XGBClassifier` settings.<jupyter_code>lgr, lgrmissing = load_results('sig_LogisticRegression_2_3_4_5_6_7_8_9_10.pkl')
xgb, xgbmissing = load_results('logsig_XGBClassifier_2_3_4_5_6_7_8_9_10.pkl')<jupyter_output><empty_output><jupyter_text>Let's inspect results on a single dataset:<jupyter_code>lgr.loc['Herring']<jupyter_output><empty_output><jupyter_text>These are the classification scores of signatures + logistic regression on the Herring dataset. The index represents the truncation level of signature features.The elapsed training time is also included. We'll only consider the actual classifcation scores:<jupyter_code>lgr = lgr[['leadlag', 'timejoined', 'timeindexed']]
xgb = xgb[['leadlag', 'timejoined', 'timeindexed']]<jupyter_output><empty_output><jupyter_text>Comparison between the classifiersLet's get a single score for each classifier by taking the maximum over all truncation levels and path embedding methods:<jupyter_code>compare = pd.concat([
    lgr.max(level=0).max(axis=1),
    xgb.max(level=0).max(axis=1),
], axis=1, sort=True).dropna()
compare.columns = ["LogisticRegression", "XGBoost"]
compare.head()<jupyter_output><empty_output><jupyter_text>There are 114 datasets in this dataframe. We can compute the mean classification score over all the datasets for the two classifiers:<jupyter_code>compare.mean()<jupyter_output><empty_output><jupyter_text>We see that logistic regression on signature features has better scores than XGBoost on log signature features. To visualize this, we can plot a scatter graph:<jupyter_code>ax = compare.plot.scatter(x='LogisticRegression', y='XGBoost')
ax.plot([0.1,1],[0.1,1], c='darkorange') # add diagonal<jupyter_output><empty_output><jupyter_text>Each point in this figure represents the classification scores of a single dataset.That is, if we have a point $(x,y)$ in the figure, then $x$ is the classification score of logistic regression for that dataset while $y$ is the classification score of XGBoost for that dataset.Therefore, points that appear below the diagonal correspond to datasets on which logistic regression performs better; and vice versa.We see that most points appear below the diagonal, which seems to imply that **logistic regression + signature features** perform better.  Comparison with BenchmarksNext, we compare these results with benchmark results of severalstate-of-the-art time series classification algorithms provided by theUCR repository.First, we download the benchmark results from www.timeseriesclassification.com.<jupyter_code>def get_benchmarks(url='http://www.timeseriesclassification.com/singleTrainTest.csv'):
    df = pd.read_csv(url)
    df = df.set_index('Unnamed: 0', drop=True)
    df.index.name = 'Dataset'
    # Fix a few mismatches between dataset names
    df = df.rename(index={
        'CinCECGtorso': 'CinCECGTorso',
        'StarlightCurves': 'StarLightCurves'
    })
    return df

bm = get_benchmarks()
bm.head()<jupyter_output><empty_output><jupyter_text>We see that the benchmarks contains results for 37 different algorithms.Let's join these with our results:<jupyter_code>df = pd.concat([bm, compare], axis=1, sort=True).dropna(thresh=3) # drop row if less than 3 non-NA values<jupyter_output><empty_output><jupyter_text>The benchmarks only contain results for 85 out of 128 datasets.This is a subset of our results on 114 datasets.Let's look at how often each algorithm performs the best on the datasetsto obtain a rough ranking between the algorithms.<jupyter_code>df.idxmax(axis=1).value_counts()<jupyter_output><empty_output><jupyter_text>We see that [HIVE-COTE](https://dl.acm.org/citation.cfm?id=3182382) performs the best on 20 datasetswhereas our LogisticRegression + signatures performs the best on 5 datasets.There is no dataset on which log signatures + XGBoost performs the best.Let's visually compare HIVE-COTE with our classifiers through a scatter plot.<jupyter_code>ax = df.plot.scatter(x='LogisticRegression', y='HIVE-COTE')
ax.plot([0.1,1],[0.1,1], c='darkorange') # add diagonal

ax = df.plot.scatter(x='XGBoost', y='HIVE-COTE')
ax.plot([0.1,1],[0.1,1], c='darkorange') # add diagonal<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11973478939157567, 'fraction_numerical': 0.01391055642225689, 'mean_word_length': 6.397115384615384, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '642267', 'license': 'MIT', 'n_tokens_mistral': 2704, 'n_tokens_neox': 2512, 'n_words': 616, 'path': 'result_analysis.ipynb', 'repo_name': 'zhy0/sig-tsc', 'size': '7692'}