<jupyter_start><jupyter_text>Anomaly detection with autoencodersIn this blog post we will use autoencoders to detect anomalies in ECG5000 dataset. In writing this blog post I have assumed that the reader has some basic understanding of neural networks and autoencoders as a specific type of neural network. Jeremy Jordan has a nice 10mins-read blog post on autoencoders, you can find it [here](https://www.jeremyjordan.me/autoencoders/).  The ECG5000 dataset which we will use contains 50,000 Electrocardiograms (ECG). Each cardiogram has 140 data points.  Luckily for us, the data has been labeled into a normal and abnormal rhythm by medical experts. Our goal is to use autoencoders to see if they can mimic the knowledge of a medical doctor and identify abnormal Electrocardiograms.Our approach will be to (1) train the autoencoder on the normal data and (2) use our trained model to reconstruct the entire dataset. We hypothesize that abnormal Electrocardiograms will have a higher reconstruction error. Recall that an autoencoder takes the input data and projects it onto a lower-dimensional space that captures only the signals in the data. The data can then be reconstructed from the lower-dimensional space. Note here that if a data point is noisy its reconstruction error (the distance between the actual point and the reconstructed one) will be large. It is this simple principle that we use to identity anomalies.  Lets load the ECG data<jupyter_code># libraries 
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tensorflow as tf


from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score, recall_score
from tensorflow.keras import layers, losses
from tensorflow.keras.datasets import fashion_mnist
from tensorflow.keras.models import Model
import plotly.express as px 
import plotly.graph_objects as go 
import plotly.offline as py_offline
# Download the dataset
url = 'http://storage.googleapis.com/download.tensorflow.org/data/ecg.csv'

dataframe = pd.read_csv(url,header=None)
raw_data = dataframe.values
# first five rows and columns
print(dataframe.iloc[:,0:5].head().to_markdown())<jupyter_output>|    |         0 |         1 |        2 |        3 |        4 |
|---:|----------:|----------:|---------:|---------:|---------:|
|  0 | -0.112522 | -2.8272   | -3.7739  | -4.34975 | -4.37604 |
|  1 | -1.10088  | -3.99684  | -4.28584 | -4.50658 | -4.02238 |
|  2 | -0.567088 | -2.59345  | -3.87423 | -4.58409 | -4.18745 |
|  3 |  0.490473 | -1.91441  | -3.61636 | -4.31882 | -4.26802 |
|  4 |  0.800232 | -0.874252 | -2.38476 | -3.97329 | -4.33822 |<jupyter_text>The last column contains the labels. The other data points are the electrocardiogram data. We will also create a train and test a dataset, as well as their labels, this is what data scientists do to overcome a serious problem in data science namely `over-fitting`.<jupyter_code># The last element contains the labels
labels = raw_data[:, -1]

# The other data points are the electrocadriogram data
data = raw_data[:, 0:-1]

train_data, test_data, train_labels, test_labels = train_test_split(
    data, labels, test_size=0.2, random_state=21
)<jupyter_output><empty_output><jupyter_text>We will also normalize the data to lie between [0,1]. Note here that we do normalization because to speed up the learning and convergence of the optimizer.<jupyter_code>min_val = tf.reduce_min(train_data)
max_val = tf.reduce_max(train_data)

train_data = (train_data - min_val) / (max_val - min_val)
test_data = (test_data - min_val) / (max_val - min_val)

train_data = tf.cast(train_data, tf.float32)
test_data = tf.cast(test_data, tf.float32)<jupyter_output><empty_output><jupyter_text>As mentioned earlier, we will only train the autoencoder on the data with normal rhythms. Electrocardiograms with normal rhythm are labeled with 1. We will separate the normal rhythm from the abnormal ones in the following chunk of code.<jupyter_code>train_labels = train_labels.astype(bool)
test_labels = test_labels.astype(bool)

normal_train_data = train_data[train_labels]
normal_test_data = test_data[test_labels]


abnormal_train_data = train_data[~train_labels]
abnormal_test_data = test_data[~test_labels]<jupyter_output><empty_output><jupyter_text>Let's visualize a normal and an abnormal  ECG.<jupyter_code>fig = go.Figure()
trace1 = go.Scatter(x=np.arange(140), y=normal_train_data[0],name='Normal')
trace2 = go.Scatter(x=np.arange(140), y=abnormal_train_data[0],name='Abnormal')
data = [trace1,trace2]
# py_offline.plot(data, filename='basic-line', include_plotlyjs=False, output_type='div')
py_offline.iplot(data)<jupyter_output><empty_output><jupyter_text>Observe that there is a huge discrepancy between the normal and abnormal graphs for large values on the x-axis. We will now build an autoencoder that will encode the normal data.Build the model. Here we will use `Kera`'s sequential API, with three dense layers for the encoder and three dense layers for the decoder. We will create an `AnormlyDectector` class that inherits from the `Model` class.<jupyter_code>class AnomalyDetector(Model):
  def __init__(self):
    super(AnomalyDetector, self).__init__()
    self.encoder = tf.keras.Sequential([
      layers.Dense(32, activation="relu"),
      layers.Dense(16, activation="relu"),
      layers.Dense(8, activation="relu")])

    self.decoder = tf.keras.Sequential([
      layers.Dense(16, activation="relu"),
      layers.Dense(32, activation="relu"),
      layers.Dense(140, activation="sigmoid")])

  def call(self, x):
    encoded = self.encoder(x)
    decoded = self.decoder(encoded)
    return decoded

autoencoder = AnomalyDetector()<jupyter_output><empty_output><jupyter_text>Note that the call method in the `AnormalyDetector` class combines the `encoder` and `decoder` and returns the `decoder` object. Let's `compile`, compilation here will mean we will update our `autoencoder` object with an `optimizer` and a `loss` function. We are using the mean absolute error loss defined as:$$\text{mae} = \frac{1}{n}\sum_{i=1}^n{|y_i-\hat{y}_i|}.$$Where in this simple formular, we have $n$ data points $ i = 1,2,...,n$, $y_i $ refers to the actual (true/observed) data point and $\hat{y}_i$ is its estimate. In our use case, $y_i$ is the actual ECG and $\hat{y}_i$ will be its reconstructed version.<jupyter_code>autoencoder.compile(optimizer='adam', loss='mae')<jupyter_output><empty_output><jupyter_text>We now train the `autoencoder` by calling its fit method using only the normal rhythm ECG.<jupyter_code>history = autoencoder.fit(normal_train_data, normal_train_data, 
          epochs=40, 
          batch_size=1024,
          validation_data=(test_data, test_data),
          shuffle=True)<jupyter_output>Epoch 1/40
3/3 [==============================] - 0s 41ms/step - loss: 0.0609 - val_loss: 0.0548
Epoch 2/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0574 - val_loss: 0.0532
Epoch 3/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0560 - val_loss: 0.0524
Epoch 4/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0548 - val_loss: 0.0515
Epoch 5/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0532 - val_loss: 0.0507
Epoch 6/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0510 - val_loss: 0.0503
Epoch 7/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0491 - val_loss: 0.0496
Epoch 8/40
3/3 [==============================] - 0s 8ms/step - loss: 0.0470 - val_loss: 0.0478
Epoch 9/40
3/3 [==============================] - 0s 10ms/step - loss: 0.0447 - val_loss: 0.0461
Epoch 10/40
3/3 [==============================] - 0s 9ms/step - loss: 0.0426 - val_loss: 0.0449
Epoch 11/40
3/3 [====================[...]<jupyter_text>Note that although, the training is done on the normal rythm ECG, the validation is done on the entire test dataset.<jupyter_code>trace1 = go.Scatter(y=history.history["loss"],name='Training loss')
trace2 = go.Scatter(y=history.history["val_loss"],name="Validation Loss")

data = [trace1,trace2]
# py_offline.plot(data, filename='basic-line', include_plotlyjs=False, output_type='div')
py_offline.iplot(data)<jupyter_output><empty_output><jupyter_text>Reconstruction errorWe now have a model that can encode and decode ECG. Let's use the model to reconstruct a particular ECG and check the reconstruction error, i.e., the difference between the actual ECG and its reconstruction.<jupyter_code>encoded_data = autoencoder.encoder(normal_test_data).numpy()
decoded_data = autoencoder.decoder(encoded_data).numpy()


trace1 = go.Scatter(y=normal_test_data[0],name='Input data')
trace2 = go.Scatter(y=decoded_data[0],name='Reconstruction & error',
                         fill='tonexty')

data = [trace1,trace2]
# py_offline.plot(data, filename='basic-line', include_plotlyjs=False, output_type='div')
py_offline.iplot(data)<jupyter_output><empty_output><jupyter_text>Let's do the same as above for an abnormal rhythm ECG on the test dataset.<jupyter_code>encoded_data = autoencoder.encoder(abnormal_test_data).numpy()
decoded_data = autoencoder.decoder(encoded_data).numpy()


trace1 = go.Scatter(y=abnormal_test_data[0],name='Input data')
trace2 = go.Scatter(y=decoded_data[0],name='Reconstruction & error',
                         fill='tonexty')


data = [trace1,trace2]
# py_offline.plot(data, filename='basic-line', include_plotlyjs=False, output_type='div')
py_offline.iplot(data)<jupyter_output><empty_output><jupyter_text>With the naked eye, the two plots above seem to suggest that the reconstruction error for the abnormal rhythm ECG is larger. We will formalize our findings in the next section. Detecting anomaliesHere we will compute the reconstruction error for all the data points both normal and abnormal.  For the reconstruction error, we will use the mean absolute error.We will compute the reconstruction error of the training dataset and choose a threshold  (one standard deviation away from the mean) above which we will classify an ECG as abnormal.<jupyter_code>reconstructions = autoencoder.predict(normal_train_data)
train_loss = tf.keras.losses.mae(reconstructions, normal_train_data)

fig = go.Figure()
fig = fig.add_trace(go.Histogram(x=train_loss[None,:][0],name='Normal loss'))
fig.show()<jupyter_output><empty_output><jupyter_text>We now define the threshold.<jupyter_code>threshold = np.mean(train_loss) + np.std(train_loss)
print("Threshold: ", threshold)<jupyter_output>Threshold:  0.03278242<jupyter_text>On the test dataset, we will use the threshold above to determine anormalies. We will do this as follows:<jupyter_code>reconstructions_normal = autoencoder.predict(normal_test_data)
test_loss_normal = tf.keras.losses.mae(reconstructions_normal, normal_test_data)


reconstructions_abnormal = autoencoder.predict(abnormal_test_data)
test_loss_abnormal = tf.keras.losses.mae(reconstructions_abnormal, abnormal_test_data)

fig = go.Figure()
fig.add_trace(go.Histogram(x=test_loss_normal[None,:][0],name='Normal loss'))
fig.add_trace(go.Histogram(x=test_loss_abnormal[None,:][0],name='Abnormal loss',opacity=0.4))

# Overlay both histograms
fig.update_layout(barmode='overlay')
# Reduce opacity to see both histograms
fig.update_traces(opacity=0.75)




fig.add_shape(type="line",
    xref="x", yref="y",
    x0=threshold, y0=0, x1=threshold, y1=90,
    line=dict(
        color="red",
        width=3,
    ),
)
fig.show()<jupyter_output><empty_output><jupyter_text>The red vertical line is at the threshold. Anything above the red vertical line is considered as an anormaly.  How accurate is our model We will compute the accuracy, precision and recall of our model.<jupyter_code>def predict(model, data, threshold):
  reconstructions = model(data)
  loss = tf.keras.losses.mae(reconstructions, data)
  return tf.math.less(loss, threshold)

def print_stats(predictions, labels):
  print("Accuracy = {}".format(accuracy_score(labels, predictions)))
  print("Precision = {}".format(precision_score(labels, predictions)))
  print("Recall = {}".format(recall_score(labels, predictions)))



preds = predict(autoencoder, test_data, threshold)
print_stats(preds, test_labels)<jupyter_output>Accuracy = 0.944
Precision = 0.9921875
Recall = 0.9071428571428571
{'fraction_non_alphanumeric': 0.13250264679534163, 'fraction_numerical': 0.03925401091294079, 'mean_word_length': 5.088249876053545, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '397742', 'license': 'MIT', 'n_tokens_mistral': 4268, 'n_tokens_neox': 3875, 'n_words': 1160, 'path': 'notebooks/autoencoder/autoencoders.ipynb', 'repo_name': 'chambox/chambox.github.io', 'size': '12279'}