<jupyter_start><jupyter_text>Author[<NAME>](https://koenaerts.ca/) @ [Mobia Technology Innovations](https://mobia.io)[myOpenHealth](https://mobia.io/healthcare/)  IntroThis notebook downloads the entire MIT-BIH Arrhythmia Database, which is publicly available here: https://physionet.org/physiobank/database/mitdb/It then converts all the WFDB files into CSV files with the same name (example from 100.dat, 100.hea, 100.atr the file 100.csv is generated).Each row in the CSV file corresponds to a single heartbeat and is generated as follows:* Split MIT-BIH record at the R-peaks into individual heartbeat records.* Each heartbeat record is appended with the first 40 readings of the next heartbeat record so that we include a full QRS Complex.* Resample each heartbeat record from 360Hz to 125Hz.* Normalize the mV readings to a 0-1 range.* Heartbeat records longer than 187 values are discarded.* Heartbeat records are padded with zeroes at the end until they contain exactly 187 values.* Heartbeat classifications from the annotations is reduced to just Normal and Abnormal and appended to the end of each heartbeat record (0 is normal, 1 is abnormal). Each row then contains exactly 188 values.* Heartbeat records without classifications are discarded.The purpose of these CSV files is so that they can be used in training your ECG model for classifying heartbeats as either Normal or Abnormal.  InitializeImport dependencies.Note that you will need to download and install the [mitdb](https://github.com/Nospoko/qrs-tutorial) library. The project contains convenience functions that make it easier to download and read [WFDB](https://physionet.org/physiotools/wfdb.shtml) compatible files. In addition, you will also need to install the [BioSPPy](https://github.com/PIA-Group/BioSPPy) library, which we use to find the R-peaks in the data.<jupyter_code>!pip install tqdm
!pip install wfdb
import os
import wfdb as wf
import numpy as np
import matplotlib.pyplot as plt
from scipy import signal
from datasets import mitdb as dm
from biosppy.signals import ecg<jupyter_output><empty_output><jupyter_text>Data ConversionRead the WFDB files and convert to CSV files. Data will be split into individual heartbeats, each row consisting of exactly 187 normalized and resampled values, plus the last value with be an integer representing the classification; 0 = Normal, 1 = Abnormal.<jupyter_code>records = dm.get_records()
print('Total files: ', len(records))

# Instead of using the annotations to find the beats, we will
# use R-peak detection instead. The reason for this is so that
# the same logic can be used to analyze new and un-annotated
# ECG data. We use the annotations here only to classify the
# beat as either Normal or Abnormal and to train the model.
# Reference:
# https://physionet.org/physiobank/database/html/mitdbdir/intro.htm
realbeats = ['N','L','R','B','A','a','J','S','V','r',
             'F','e','j','n','E','/','f','Q','?']

# Loop through each input file. Each file contains one
# record of ECG readings, sampled at 360 readings per
# second.
for path in records:
    pathpts = path.split('/')
    fn = pathpts[-1]
    print('Loading file:', path)

    # Read in the data
    record = wf.rdsamp(path)
    annotation = wf.rdann(path, 'atr')

    # Print some meta informations
    print('    Sampling frequency used for this record:', record[1].get('fs'))
    print('    Shape of loaded data array:', record[0].shape)
    print('    Number of loaded annotations:', len(annotation.num))
    
    # Get the ECG values from the file.
    data = record[0].transpose()

    # Generate the classifications based on the annotations.
    # 0.0 = undetermined
    # 1.0 = normal
    # 2.0 = abnormal
    cat = np.array(annotation.symbol)
    rate = np.zeros_like(cat, dtype='float')
    for catid, catval in enumerate(cat):
        if (catval == 'N'):
            rate[catid] = 1.0 # Normal
        elif (catval in realbeats):
            rate[catid] = 2.0 # Abnormal
    rates = np.zeros_like(data[0], dtype='float')
    rates[annotation.sample] = rate
    
    indices = np.arange(data[0].size, dtype='int')

    # Process each channel separately (2 per input file).
    for channelid, channel in enumerate(data):
        chname = record[1].get('sig_name')[channelid]
        print('    ECG channel type:', chname)
        
        # Find rpeaks in the ECG data. Most should match with
        # the annotations.
        out = ecg.ecg(signal=channel, sampling_rate=360, show=False)
        rpeaks = np.zeros_like(channel, dtype='float')
        rpeaks[out['rpeaks']] = 1.0
        
        beatstoremove = np.array([0])

        # Split into individual heartbeats. For each heartbeat
        # record, append classification (normal/abnormal).
        beats = np.split(channel, out['rpeaks'])
        for idx, idxval in enumerate(out['rpeaks']):
            firstround = idx == 0
            lastround = idx == len(beats) - 1

            # Skip first and last beat.
            if (firstround or lastround):
                continue

            # Get the classification value that is on
            # or near the position of the rpeak index.
            fromidx = 0 if idxval < 10 else idxval - 10
            toidx = idxval + 10
            catval = rates[fromidx:toidx].max()
            
            # Skip beat if there is no classification.
            if (catval == 0.0):
                beatstoremove = np.append(beatstoremove, idx)
                continue

            # Normal beat is now classified as 0.0 and abnormal is 1.0.
            catval = catval - 1.0

            # Append some extra readings from next beat.
            beats[idx] = np.append(beats[idx], beats[idx+1][:40])

            # Normalize the readings to a 0-1 range for ML purposes.
            beats[idx] = (beats[idx] - beats[idx].min()) / beats[idx].ptp()

            # Resample from 360Hz to 125Hz
            newsize = int((beats[idx].size * 125 / 360) + 0.5)
            beats[idx] = signal.resample(beats[idx], newsize)

            # Skipping records that are too long.
            if (beats[idx].size > 187):
                beatstoremove = np.append(beatstoremove, idx)
                continue

            # Pad with zeroes.
            zerocount = 187 - beats[idx].size
            beats[idx] = np.pad(beats[idx], (0, zerocount), 'constant', constant_values=(0.0, 0.0))

            # Append the classification to the beat data.
            beats[idx] = np.append(beats[idx], catval)

        beatstoremove = np.append(beatstoremove, len(beats)-1)

        # Remove first and last beats and the ones without classification.
        beats = np.delete(beats, beatstoremove)

        # Save to CSV file.
        savedata = np.array(list(beats[:]), dtype=np.float)
        outfn = 'data_ecg/'+fn+'_'+chname+'.csv'
        print('    Generating ', outfn)
        with open(outfn, "wb") as fin:
            np.savetxt(fin, savedata, delimiter=",", fmt='%f')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10088596741926265, 'fraction_numerical': 0.016147470705915976, 'mean_word_length': 2.868988391376451, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '327507', 'license': 'MIT', 'n_tokens_mistral': 2164, 'n_tokens_neox': 1987, 'n_words': 743, 'path': '02_import_mitdb_data.ipynb', 'repo_name': 'JiaminXu1019/ECG_ML', 'size': '7002'}