<jupyter_start><jupyter_text>created by <NAME> (https://github.com/williamsdoug) and <NAME> (https://github.com/timeseriesAI/tsai). contact: <EMAIL>  Purpose 😇 The purpose of this notebook is to demonstrate both multi-class and multi-label classification using `tsai`. - Multi-class classification: While the output can take on multiple possible values, for any given sample the output can take on only a single value. In other words, the label values are mutually exclusive. Implication: - CategoricalCrossEntropy (`nn.CrossEntropyLoss` in Pytorch, `CrossEntropyLossFlat` in fastai) is used as the loss function during training - Softmax is used to determine prediction since only one label value can be true, the predicted label is the value label value with the greatest probability.  - Softmax reduces the potential for spurious label predictions since only the label with the highest probability is selected - In both Pytorch and fastai the loss combines a Softmax layer and the CrossEntropyLoss in one single class, so Softmax shouldn't be added to the model. - Multi-label classification: This is the more general case where an individual sample can have one or more labels, relaxing the mutual label exclusivity constraint. Implications: - BinaryCrossEntropy (`nn.BCEWithLogitsLoss` in Pytorch, `BCEWithLogitsLossFlat` in fastai) is used as the loss function during training - Sigmoid is used to determine prediction since multiple labels may be true. In both Pytorch and fastai the loss combines a Sigmoid layer and the BCELoss in one single class, so Sigmoid shouldn't be added to the model. - Relative to multi-class classification, multi-label classification may be more prone to spurious false-positive labels.  Import libraries 📚<jupyter_code># ## NOTE: UNCOMMENT AND RUN THIS CELL IF YOU NEED TO INSTALL/ UPGRADE TSAI
# stable = False # True: latest version from github, False: stable version in pip
# if stable: 
#     !pip install -Uqq tsai
# else:      
#     !pip install -Uqq git+https://github.com/timeseriesAI/tsai.git

# ## NOTE: REMEMBER TO RESTART YOUR RUNTIME ONCE THE INSTALLATION IS FINISHED
from tsai.all import *
print('tsai       :', tsai.__version__)
print('fastai     :', fastai.__version__)
print('fastcore   :', fastcore.__version__)
print('torch      :', torch.__version__)<jupyter_output>tsai       : 0.2.17
fastai     : 2.3.0
fastcore   : 1.3.19
torch      : 1.7.1<jupyter_text>Prepare data 🔢For this example we will be using the UCR ECG5000 heartbeat dataset with is based on the [Physionet BIDMC Congestive Heart Failure Database](https://physionet.org/content/chfdb/1.0.0/]), specifically recording chf07.For the purposes of this example the UCR labels will be mapped to more descriptive labels:- 1 - Normal ('Nor')- 2 - R-on-T premature ventricular contraction ('RoT')- 3 - Premature ventricular contraction ('PVC')- 4 - Supraventricular premature or ectopic beat, atrial or nodal ('SPC')- 5 - Unclassifiable beat ('Unk')<jupyter_code>class_map = {
    '1':'Nor',  # N:1  - Normal
    '2':'RoT',  # r:2  - R-on-T premature ventricular contraction
    '3':'PVC',  # V:3  - Premature ventricular contraction
    '4':'SPC',  # S:4  - Supraventricular premature or ectopic beat (atrial or nodal)
    '5':'Unk',  # Q:5  - Unclassifiable beat
    }
class_map
# dataset id
dsid = 'ECG5000' 
X, y, splits = get_UCR_data(dsid, split_data=False)
labeler = ReLabeler(class_map)
new_y = labeler(y) # map to more descriptive labels
X.shape, new_y.shape, splits, new_y
label_counts = collections.Counter(new_y)
print('Counts by label:', dict(label_counts))
print(f'Naive Accuracy: {100*max(label_counts.values())/sum(label_counts.values()):0.2f}%')<jupyter_output>Counts by label: {'Nor': 2919, 'RoT': 1767, 'PVC': 96, 'SPC': 194, 'Unk': 24}
Naive Accuracy: 58.38%<jupyter_text>Note: naive accuracy is calculated by assuming all samples are predicted as 'Nor' (most frequent label).  Multi-class classification  Prepare dataloaders 💿<jupyter_code>tfms  = [None, TSClassification()] # TSClassification == Categorize
batch_tfms = TSStandardize()
dls = get_ts_dls(X, new_y, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=[64, 128])
dls.dataset<jupyter_output><empty_output><jupyter_text>Visualize data<jupyter_code>dls.show_batch(sharey=True)<jupyter_output><empty_output><jupyter_text>Build learner 🏗 There are at least 2 equivalent ways to build a time series learner:<jupyter_code>model = build_ts_model(InceptionTimePlus, dls=dls)
learn = Learner(dls, model, metrics=accuracy)
learn = ts_learner(dls, metrics=accuracy) # == ts_learner(dls, arch=InceptionTimePlus, metrics=accuracy) since InceptionTimePlus is the default arch<jupyter_output><empty_output><jupyter_text>LR find 🔎<jupyter_code>learn.lr_find(suggestions=False)<jupyter_output><empty_output><jupyter_text>Train 🏃🏽‍♀️<jupyter_code>learn = ts_learner(dls, metrics=accuracy, cbs=ShowGraph())
learn.fit_one_cycle(10, lr_max=1e-3)<jupyter_output><empty_output><jupyter_text>Visualize results 👁<jupyter_code>learn.show_results(sharey=True)
learn.show_probas()
interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
interp.most_confused(min_val=3)<jupyter_output><empty_output><jupyter_text>Multi-label Classification  Augment labels to demonstrate multi-label- Create additional label premature ('Pre')- Include with any sample where labels 'RoT','PVC','SPC' are already presentNote:  While in this example the new Pre label is a composite of existing labels, more typically multi-label classification problems include orthogonal label groups.  For example in ECG classification one might have labels related to timing (e.g.: premature), QRS shape (e.g.: block) and other factors (e.g.: ST elevation or depression)<jupyter_code>class_map = {
    '1':['Nor'],          # N:1  - Normal
    '2':['RoT', 'Pre'],   # r:2  - R-on-T premature ventricular contraction
    '3':['PVC', 'Pre'] ,  # V:3  - Premature ventricular contraction
    '4':['SPC', 'Pre'],   # S:4  - Supraventricular premature or ectopic beat (atrial or nodal)
    '5':['Unk'],          # Q:5  - Unclassifiable beat
    }
class_map
labeler = ReLabeler(class_map)
y_multi = labeler(y)
y_multi
label_counts = collections.Counter([a for r in y_multi for a in r])
print('Counts by label:', dict(label_counts))<jupyter_output>Counts by label: {'Nor': 2919, 'RoT': 1767, 'Pre': 2057, 'PVC': 96, 'SPC': 194, 'Unk': 24}<jupyter_text>Prepare Dataloaders- Replace earlier ```tfms  = [None, [TSClassification()]]```  with ```tfms  = [None, TSMultiLabelClassification()]```- TSMultiLabelClassification() is equivalent to [MultiCategorize(), OneHotEncode()] combined in a single transform.- When creating dataloaders in multi-label problems, always leave inplace=True (default value) to avoid issues.<jupyter_code>tfms  = [None, TSMultiLabelClassification()] # TSMultiLabelClassification() == [MultiCategorize(), OneHotEncode()]
batch_tfms = TSStandardize()
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=[64, 128])
dls.dataset
dls.show_batch(sharey=True)<jupyter_output><empty_output><jupyter_text>Build Learner and Train- Use ```metrics=accuracy_multi``` in place of ```metrics=accuracy``` used in earlier multi-class example- accuracy_multi can be calculated by sample (default) or by predicted label. When using accuracy_multi by_sample=True, all predicted labels for each sample must be correct to be counted as a correct sample. This is sometimes difficult (when having too many labels). There are cases when labels are partially correct. If we want to account for these, we'll set by_sample to False.<jupyter_code>def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True, by_sample=False):
    "Computes accuracy when `inp` and `targ` are the same size."
    if sigmoid: inp = inp.sigmoid()
    correct = (inp>thresh)==targ.bool()
    if by_sample:
        return (correct.float().mean(-1) == 1).float().mean()
    else:
        inp,targ = flatten_check(inp,targ)
        return correct.float().mean()
learn = ts_learner(dls, InceptionTime, metrics=accuracy_multi)
learn.lr_find(suggestions=False)
learn = ts_learner(dls, InceptionTimePlus, metrics=[partial(accuracy_multi, by_sample=True), partial(accuracy_multi, by_sample=False)], cbs=ShowGraph())
learn.fit_one_cycle(10, lr_max=1e-3)<jupyter_output><empty_output><jupyter_text>Additional Multi-label Metrics A naive classifier that always predicts no true labels would achieve 76% accuracy, so the classifier should do much better.  This also demonstrates the weakness of relying overly on the accuracy metric (with by_sample=False) since due to the prevalance of false outputs.<jupyter_code>label_counts = collections.Counter([a for r in y_multi for a in r])
print(f'Naive Accuracy: {100*(1-sum(label_counts.values())/(len(y_multi)*len(label_counts))):0.2f}%')<jupyter_output>Naive Accuracy: 76.48%<jupyter_text>Define MetricsWe have included a number of multilabel metrics in `tsai` based on definitions in [Wikipedia](https://en.wikipedia.org/wiki/Precision_and_recallF-measure)<jupyter_code>def precision_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Computes precision when `inp` and `targ` are the same size."
    
    inp,targ = flatten_check(inp,targ)
    if sigmoid: inp = inp.sigmoid()
    pred = inp>thresh
    
    correct = pred==targ.bool()
    TP = torch.logical_and(correct,  (targ==1).bool()).sum()
    FP = torch.logical_and(~correct, (targ==0).bool()).sum()

    precision = TP/(TP+FP)
    return precision

def recall_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Computes recall when `inp` and `targ` are the same size."
    
    inp,targ = flatten_check(inp,targ)
    if sigmoid: inp = inp.sigmoid()
    pred = inp>thresh
    
    correct = pred==targ.bool()
    TP = torch.logical_and(correct,  (targ==1).bool()).sum()
    FN = torch.logical_and(~correct, (targ==1).bool()).sum()

    recall = TP/(TP+FN)
    return recall

def specificity_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Computes specificity (true negative rate) when `inp` and `targ` are the same size."
    
    inp,targ = flatten_check(inp,targ)
    if sigmoid: inp = inp.sigmoid()
    pred = inp>thresh
    
    correct = pred==targ.bool()
    TN = torch.logical_and(correct,  (targ==0).bool()).sum()
    FP = torch.logical_and(~correct, (targ==0).bool()).sum()

    specificity = TN/(TN+FP)
    return specificity

def balanced_accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
    "Computes balanced accuracy when `inp` and `targ` are the same size."
    
    inp,targ = flatten_check(inp,targ)
    if sigmoid: inp = inp.sigmoid()
    pred = inp>thresh
    
    correct = pred==targ.bool()
    TP = torch.logical_and(correct,  (targ==1).bool()).sum()
    TN = torch.logical_and(correct,  (targ==0).bool()).sum()
    FN = torch.logical_and(~correct, (targ==1).bool()).sum()
    FP = torch.logical_and(~correct, (targ==0).bool()).sum()

    TPR = TP/(TP+FN)
    TNR = TN/(TN+FP)
    balanced_accuracy = (TPR+TNR)/2
    return balanced_accuracy

def Fbeta_multi(inp, targ, beta=1.0, thresh=0.5, sigmoid=True):
    "Computes Fbeta when `inp` and `targ` are the same size."
    
    inp,targ = flatten_check(inp,targ)
    if sigmoid: inp = inp.sigmoid()
    pred = inp>thresh
    
    correct = pred==targ.bool()
    TP = torch.logical_and(correct,  (targ==1).bool()).sum()
    TN = torch.logical_and(correct,  (targ==0).bool()).sum()
    FN = torch.logical_and(~correct, (targ==1).bool()).sum()
    FP = torch.logical_and(~correct, (targ==0).bool()).sum()

    precision = TP/(TP+FP)
    recall = TP/(TP+FN)
    beta2 = beta*beta
    
    if precision+recall > 0:
        Fbeta = (1+beta2)*precision*recall/(beta2*precision+recall)
    else:
        Fbeta = 0
    return Fbeta

def F1_multi(*args, **kwargs):
    return Fbeta_multi(*args, **kwargs)  # beta defaults to 1.0<jupyter_output><empty_output><jupyter_text>Train Classifier Including More Metrics<jupyter_code>metrics =[accuracy_multi, balanced_accuracy_multi, precision_multi, recall_multi, specificity_multi, F1_multi] 
learn = ts_learner(dls, InceptionTime, metrics=metrics, cbs=ShowGraph())
learn.fit_one_cycle(10, lr_max=1e-3)<jupyter_output><empty_output><jupyter_text>Optionally Use Class (Positive) Weights- Include Per-label positive weights bias the loss function to give greater importance to samples where label is present with loss function- `tsai` automatically calculates class weights. It's a dataloaders attribute called cws. **You should use dls.train.cws to avoid any leakage.**  Prepare DataLoaders<jupyter_code>tfms  = [None, TSMultiLabelClassification()] # TSMultiLabelClassification() == [MultiCategorize(), OneHotEncode()]
batch_tfms = TSStandardize()
dls = get_ts_dls(X, y_multi, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=[64, 128])
dls.vocab, dls.train.cws<jupyter_output><empty_output><jupyter_text>Build Learner and Train- Include class weights with ```loss_func```<jupyter_code>metrics = [accuracy_multi, balanced_accuracy_multi, precision_multi, recall_multi,  specificity_multi, F1_multi]
learn = ts_learner(dls, InceptionTime, metrics=metrics, loss_func=BCEWithLogitsLossFlat(pos_weight=dls.train.cws), cbs=ShowGraph())
learn.lr_find(suggestions=True, start_lr=1e-06, end_lr=1e2)
learn.fit_one_cycle(20, lr_max=1e-3)<jupyter_output><empty_output><jupyter_text>Try Again with Reduced Weights (reduce by sqrt)<jupyter_code>metrics = [accuracy_multi, balanced_accuracy_multi, precision_multi, recall_multi,  specificity_multi, F1_multi]
learn = ts_learner(dls, InceptionTime, metrics=metrics, loss_func=BCEWithLogitsLossFlat(pos_weight=dls.train.cws.sqrt()), cbs=ShowGraph())
learn.lr_find(suggestions=True, start_lr=1e-06, end_lr=1e2)
learn.fit_one_cycle(10, lr_max=1e-3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11959915765013435, 'fraction_numerical': 0.014305424442669377, 'mean_word_length': 5.26, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 75, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '152541', 'license': 'Apache-2.0', 'n_tokens_mistral': 4917, 'n_tokens_neox': 4656, 'n_words': 1260, 'path': 'tutorial_nbs/01a_MultiClass_MultiLabel_TSClassification.ipynb', 'repo_name': 'xanderdunn/tsai', 'size': '13801'}