<jupyter_start><jupyter_text>ECG - ElectrocardiographyThis notebook illustrates how we can search a large time series of over 20 million data points using the mass2_batch algorithm. Electrocardiography measures heart beats through attaching sensors to the skin of an individual. This particular data set is sampled at 250 samples per second containing 22 hours and 23 minutes of data points; 110,087 heart beats. This individual is an 84 year old woman.The dataset original came from the UCR Suite example illustrating the LB_Keogh algorithm for DTW.https://www.cs.ucr.edu/~eamonn/UCRsuite.htmlThey found a series, used as the query, consisting of a a PVC - premature ventricular contraction on an educational poster.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import mass_ts as mts

%matplotlib inline
import tarfile

def load_file(archive, fp):
    """
    Utility function that reads a tar file directly into a numpy array.
    
    Parameters
    ----------
    archive : str
        The archive file to read.
    fp : str
        The file path of the file to read relative to the archive.
    
    
    Returns
    -------
    None if data reading failed or the numpy array of values.
    """
    data = None
    with tarfile.open(archive) as a:
        f = a.extractfile(dict(zip(a.getnames(),a.getmembers()))[fp])
        data = pd.read_csv(f, header=None, names=['reading', ])['reading'].values
    
    return data<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>%%time
ecg = load_file('ecg.tar.gz', 'ecg.txt')
ecg_query = load_file('ecg_query.tar.gz', 'ecg_query.txt')
print('ECG data length is {:,}'.format(len(ecg)))
print('Query length is {:,}'.format(len(ecg_query)))<jupyter_output>ECG data length is 20,140,000
Query length is 421<jupyter_text>VisualizeHere is a sample of what the data looks like.<jupyter_code>sample = ecg[0:10000]
plt.figure(figsize=(20,5))
plt.plot(range(len(sample)), sample)
plt.title('ECG Sample 0-10,000')
plt.show()
plt.figure(figsize=(10,5))
plt.plot(range(len(ecg_query)), ecg_query)
plt.title('PVC Query')
plt.show()<jupyter_output><empty_output><jupyter_text>Compute Top MatchesHere we compute the top 5 matches based on the query illustrating both single-threaded mode and multi-threaded mode.<jupyter_code># total number of matches we want returned
top_matches = 5

# length of the subsequence to search in batch processing.
# note that this has an impact on memory usage
batch_size = 10000<jupyter_output><empty_output><jupyter_text>Single-threaded<jupyter_code>%%timeit
best_indices, best_dists = mts.mass2_batch(ecg, ecg_query, batch_size=batch_size, top_matches=top_matches)<jupyter_output>56.7 s ± 5.43 s per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>Multi-threadedRun the same logic, but with 4 cpu threads.<jupyter_code># run same task with 4 cpu threads
n_jobs = 4
%%timeit
best_indices, best_dists = mts.mass2_batch(ecg, ecg_query, batch_size=batch_size, top_matches=top_matches, n_jobs=n_jobs)
best_indices, best_dists = mts.mass2_batch(ecg, ecg_query, batch_size=batch_size, top_matches=top_matches, n_jobs=n_jobs)<jupyter_output><empty_output><jupyter_text>The multi-threaded version is 2X faster than the single threaded version.  Visualize Top MatchesThe plot below shows the top 5 matches given our query. The first plot is the query we searched for.<jupyter_code># Plot the query with top matches
fig, axes = plt.subplots(top_matches + 1,1,sharex=True,figsize=(10,7))
axes[0].plot(np.arange(len(ecg_query)), ecg_query)

for ax, idx in zip(axes[1:], best_indices):
    ax.plot(np.arange(len(ecg_query)), ecg[idx:idx+len(ecg_query)])
    #ax.set_ylabel('Index {}'.format(idx), size=12)

plt.suptitle('Top 5 Matches')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10323939952594154, 'fraction_numerical': 0.021069265209375822, 'mean_word_length': 5.106109324758842, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '656020', 'license': 'Apache-2.0', 'n_tokens_mistral': 1342, 'n_tokens_neox': 1240, 'n_words': 390, 'path': 'ecg/ECG.ipynb', 'repo_name': 'Deepakgthomas/mass-ts-examples', 'size': '3797'}