<jupyter_start><jupyter_text>Whale Sound ClassificationIn this notebook we will discuss ways to classify signals using annotated data. Objectives:* train a machine learning classifier with the scikit-learn package* learn how to evaluate machine learning models  In the kaggle competition we had the privilage to have annotated data. So let's use those labels. We are in fact trying to solve a classification problem: we want to build a classifier which correctly identifies whale calls.  Data Loading---<jupyter_code># ignore warnings
import warnings
warnings.filterwarnings("ignore")
# importing multiple visualization libraries
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import mlab
import pylab as pl
#import seaborn
# importing libraries to manipulate the data files
import os
from glob import glob
# import numpy
import numpy as np
# loading the data
X = np.load('X.npy')
y = np.load('y.npy')<jupyter_output><empty_output><jupyter_text>Data Splitting---We will organize the data for traning. We will select a testing data set which we will not touch, until we are happy with our algorithm and we want to evaluate the error.  The rest will be used for training. During training we can use that dataset as much as we want to improve our algorithms. We can further subset it into a training set and validate set, and use the validate set to evaluate different hyperparameters and models (or use formal cross-vslidation). ![](img/TrainValidateTest.png) Since we have already left out a big part of the dataset (10000:30000), we will split $X$ and $y$ in two parts: a training set and a validation set.<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.20, random_state=2018)<jupyter_output><empty_output><jupyter_text>Model Fitting--- [//]:![text](https://gist.githubusercontent.com/amueller/4642976/raw/e48eff2df7790583f1f3212095d24639738a1b4a/drop_shadows.svg?sanitize=true)<jupyter_code>from  IPython.display import Image
Image("img/MLmap.png" )<jupyter_output><empty_output><jupyter_text>Source: http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html  `scikit learn` has many built-in classification algorithms. It is a good strategy to first try a linear classifier and create a baseline, and then try more complex methods. Here are some good candidates:* [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html),  Support Vector Classifier ([SVC](http://scikit-learn.org/stable/modules/svm.html))* [Random Forests](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), [Gradient Boosting](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) - nonlinear classifiers (ensemble methods)<jupyter_code>%%time
# Fitting a Logistic Regression
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X_train, y_train)<jupyter_output>CPU times: user 46.2 s, sys: 388 ms, total: 46.6 s
Wall time: 46.7 s<jupyter_text>Evaluation and Model Selection--- **Accuracy** Ok, we fitted a classifier, but how should we evaluate performance? First let's look at the accuracy on the train dataset: it better be good!!!<jupyter_code># prediction on the training dataset
train_accuracy = 1 - np.sum(np.abs(clf.predict(X_train) - y_train))/len(y_train)
print('Accuracy on the train dataset is '+ str(train_accuracy))<jupyter_output>Accuracy on the train dataset is 0.8398549818727341<jupyter_text>But that does not matter, what we want to know is how the method performs on the validation dataset, whose labels we have not seen in training.<jupyter_code># prediction on the validation dataset
val_accuracy = 1 - np.sum(np.abs(clf.predict(X_val)-y_val))/len(y_val)
print('Accuracy on the validation dataset is '+ str(val_accuracy))<jupyter_output>Accuracy on the validation dataset is 0.784<jupyter_text>Ok, it is lower, as expected, but still decent. **Receiver Operating Characteristic (ROC) Curves** Note: the Logistic Regression is a probalistic algorithm and in fact can output a score for the chance of belonging to a class.<jupyter_code># predicting class
y_pred = clf.predict(X_val)

# predicting score for each class 
y_score = clf.predict_proba(X_val)[:,1]<jupyter_output><empty_output><jupyter_text>Warning: `clf.predict` by default uses 0.5 as a decision threshold: `y_score>0.5`: right whale upcall`y_score<0.5`: no right whale upcall    But we can adjust this threshold to improve the performance.  To study this performance we can use the [Receiver Operating Characteristic](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and the Area Under the Curve (ROC AUC).<jupyter_code>from sklearn.metrics import roc_auc_score, roc_curve
roc_auc = roc_auc_score(y_val, y_score)
fpr, tpr, _ = roc_curve(y_val, y_score)

plt.figure()
lw = 2
plt.plot(fpr, tpr, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate\n predicted upcall when no upcall')
plt.ylabel('True Positive Rate\n predicted upcall when upcall')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.show()<jupyter_output><empty_output><jupyter_text>threshold == 0: we will identify all whale upcalls, but also all the no upcalls will be identified as upcalls.threshold == 1: we will miss all the whale upcalls, but won't have wrongly identified no upcalls.The area under the curve (AUC) gives us a measure for the performance under different thresholds. It is good when close to 1. But is this enough? *Question:* what will be the accuracy if we always claim there is no whale call? *Hint:* what is the percentage of snippets with whale calls? So we can achieve pretty decent accuracy with a crappy classifier, which never detects 0 of all upcalls.We definity need to look at other metrics. **Confusion Matrix** It is useful to look at all types of errors the algorithm makes. For, that we compute the confusion matrix.<jupyter_code>from sklearn.metrics import confusion_matrix
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    #plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_val, y_pred)
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
class_names = ['no_right_upcall','right_upcall']
plot_confusion_matrix(cnf_matrix, classes=class_names, normalize = True,
                      title='Confusion matrix, with normalization')<jupyter_output>Normalized confusion matrix
[[0.93 0.07]
 [0.66 0.34]]<jupyter_text>Based on the confusion matrix we see that we miss a lot of upcalls. We can investigate this deeper using metrics which are more suitable for such unbalanced problems. **Precision and Recall** Precision: $\frac{\textrm{correctly predicted upcalls}}{\textrm{predicted upcalls}}$<jupyter_code># calculate precision with formula
predicted_upcalls = (y_pred==1)
correctly_predicted_upcalls = y_val[predicted_upcalls]

precision = sum(correctly_predicted_upcalls)/sum(predicted_upcalls)
print('Precision: {0:0.2f}'.format(precision))
# calculate precision with scikit-learn function
from sklearn.metrics import precision_score, recall_score

precision_score(y_val, y_pred)
from sklearn.metrics import average_precision_score
average_precision = average_precision_score(y_val, y_pred)

print('Average precision-recall score: {0:0.2f}'.format(
      average_precision))<jupyter_output>Average precision-recall score: 0.38<jupyter_text>Recall: $\frac{\textrm{correctly predicted upcalls}}{\textrm{true upcalls}}$<jupyter_code>recall_score(y_val, y_pred)
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt

precision, recall, _ = precision_recall_curve(y_val, y_score)
plt.step(recall, precision, color='b', alpha=0.2,
         where='post')
plt.fill_between(recall, precision, step='post', alpha=0.2,
                 color='b')

plt.xlabel('Recall')
plt.ylabel('Precision')
plt.ylim([0.0, 1.05])
plt.xlim([0.0, 1.0])
plt.title('2-class Precision-Recall curve: AP={0:0.2f}'.format(
          average_precision))<jupyter_output><empty_output><jupyter_text>[F1 score](https://en.wikipedia.org/wiki/F1_score) is a measure which combines both precision and recall.$\textrm{F1} = 2 \frac{\textrm{precision} \times \textrm{recall}}{\textrm{precision}+\textrm{recall}}$<jupyter_code>from sklearn.metrics import f1_score
print('F1 score: {0:0.2f}'.format(f1_score(y_val, y_pred)))<jupyter_output>F1 score: 0.44<jupyter_text>From the precision and recall curve we can see that we can achieve higher f1-score if we increase the recall, before the precision starts to drastically decrease, we can achieve higher f1 score. **How can we improve**?* perform [cross validation](http://scikit-learn.org/stable/modules/cross_validation.html)![](https://scikit-learn.org/stable/_images/grid_search_cross_validation.png)* try other classifiers: [Random Forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), [gradient boosting](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html), [SVC](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) and compare their ROC/PR curves* balance the training set, not the validation one* stratify the samples, so that you have similar proportion in the subsamples* apply dimensionality reduction first and then classify* account for time shifting* ??? Too slow? Have more cores? Checkout [dask-ml](https://dask-ml.readthedocs.io/en/latest/) for parallelizing some machine learning functions.   References:---* https://github.com/jaimeps/whale-sound-classification * https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Scikit_Learn_Cheat_Sheet_Python.pdf  Exercises:--- **Exercise 1:** Find a different prediction threshold for which you can improve the f1-score. Can you find an optimal one? **Exercise 2:** 1. Fit the data with a RandomForest Classifier2. Calculate the ROC curve3. Plot together with the ROC curve for Logistic Regression: which classifier is better?<jupyter_code>from sklearn.ensemble import RandomForestClassifier
...<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10165195350056813, 'fraction_numerical': 0.016082510270081285, 'mean_word_length': 5.262725779967159, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 49, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '150040', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3677, 'n_tokens_neox': 3462, 'n_words': 1106, 'path': '3-classification.ipynb', 'repo_name': 'oceanhackweek/ohw19-tutorial-machine-learning', 'size': '11441'}