<jupyter_start><jupyter_text>Anomaly Detection in Electrocardigram Readings by <NAME> ![ecg](images/ecg.jpeg)  IntroductionSeveral of my potential desired employers are involved with signal processing. For one of my next projects, I am interested in doing image analysis for predictions for medical images or geospatial data, but since we haven't really looked at image processing yet, I chose a time-series signal -- specifically, electrocardiograms (ECG).Real-world anomaly detection for time series is still a challenging task. This is especially true for periodic or quasi-periodic time series since automated approaches have to learn long-term correlations before they are able to detect anomalies. ECG time series are a prominent, real-world example of quasi-periodic signals (1).The first labelled ecg dataset I explored already had single cardiac cycle data segmented. It would likely have been a simpler dataset to work with, but a lot of the extraction of the relevant signal portions from the raw signal was already done and that is a big part of the puzzle, so I wanted to stare with more real-world data and perform that step myself.I found a second dataset which had 9-120 second recordings which looked promising, but the data existed as thousands of individual matlab files, so ingesting them into a usable dataframe was posing a challenge. I took a little break from processing those files to search for an additional dataset that had the appropriate data in a slightly more user-friendly format and came across the following dataset which I will use moving forward.1. <NAME>, et. al. 2019 'Anomaly Detection in Electrocardiogram Readings with Stacked LSTM Networks', *Proceedings of the 19th Conference Information Technologies - Applications and Theory (ITAT 2019)*, p 17-25 ---  DatasetI will use the MIH-BIH Arrythmia dataset from https://physionet.org/content/mitdb/1.0.0/. This is a dataset with 48 half-hour two-channel ECG recordings measured at 360 Hz. The recordings have annotations from cardiologists for each heart beat. The symbols for the annotations can be found at https://archive.physionet.org/physiobank/annotations.shtml  GoalsPredict if a heartbeat from an ECG signal is normal or abnormal based on a six-second window centered on the peak of the hearbeat. ---  Data Preparation<jupyter_code>import pandas as pd

import numpy as np
import matplotlib.pyplot as plt
from os import listdir

from sklearn.model_selection import train_test_split

data_path = '/Users/jaredgodar/codeup-data-science/ecg_anomaly_detection/physionet.org/files/mitdb/1.0.0/'

# list of patients
pts = ['100','101','102','103','104','105','106','107',
       '108','109','111','112','113','114','115','116',
       '117','118','119','121','122','123','124','200',
       '201','202','203','205','207','208','209','210',
       '212','213','214','215','217','219','220','221',
       '222','223','228','230','231','232','233','234']

import wfdb #PyPi package for loading ecg and annotations
# Load annotations and look at distribution of heartbeat types

df = pd.DataFrame()

for pt in pts:
    file = data_path + pt
    annotation = wfdb.rdann(file, 'atr')
    sym = annotation.symbol
    
    values, counts = np.unique(sym, return_counts=True)
    df_sub = pd.DataFrame({'sym':values, 'val':counts, 'pt':[pt]*len(counts)})
    df = pd.concat([df, df_sub],axis = 0)
df.head()
df.groupby('sym').val.sum().sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>This dataset is extensively labelled with various abnormalities given their own names and labels. For the initial model, I will only use a boolean 0/1 to represent normal/abnormal. Later iterations may attempt to no only identify anomalies, but provide proposed labels naming the specific anomaly type, but that is beyond the scope of the current work.Some of the abnormal labels also apply to features that aren't part of the normal heartbeat / cardiac cycle. I will focus on the actual abnormal beat features, not these "nonbeat" features.<jupyter_code># list of nonbeat and abnormal
nonbeat = ['[','!',']','x','(',')','p','t','u','`',
           '\'','^','|','~','+','s','T','*','D','=','"','@','Q','?']
abnormal = ['L','R','V','/','A','f','F','j','a','E','J','e','S']
# break into normal, abnormal or nonbeat
df['cat'] = -1
df.loc[df.sym == 'N','cat'] = 0
df.loc[df.sym.isin(abnormal), 'cat'] = 1
df.groupby('cat').val.sum()<jupyter_output><empty_output><jupyter_text>This gives me an overall dataset of 75052 normal cardiac cycles, 34409 examples of abnormalities, and 3186 non-beat features which will be excluded from training and modeling. The following function will load a single patient's signals and annotations (annotation values are the indices of the signal array)<jupyter_code>def load_ecg(file):
    # load the ecg
    # example file: 'mit-bih-arrhythmia-database-1.0.0/101'
    
    # load the ecg
    record = wfdb.rdrecord(file)
    # load the annotation
    annotation = wfdb.rdann(file, 'atr')
    
    # extract the signal
    p_signal = record.p_signal
    
    # verify frequency is 360
    assert record.fs == 360, 'sample freq is not 360'
    
    # extract symbols and annotation index
    atr_sym = annotation.symbol
    atr_sample = annotation.sample
    
    return p_signal, atr_sym, atr_sample 
# Files are named by patient number, already listed in the pts array, in the path defined
file = data_path + pts[0]

p_signal, atr_sym, atr_sample = load_ecg(file)

values, counts = np.unique(sym, return_counts=True)
for v,c in zip(values, counts):
    print(v,c)<jupyter_output>+ 3
J 50
N 2700
V 3
~ 8<jupyter_text>It is important to note that the data do not contain 'normal' ECGs for 'normal' patients and 'abnornal' ecgs for 'abnormal' patients.For any given patient over the recording period, there will be many 'normal' beats and possibly 'abnormal' ones as well.<jupyter_code># Get index of any abnormal beats
ab_index = [b for a,b in zip(atr_sym,atr_sample) if a in abnormal][:10]
ab_index<jupyter_output><empty_output><jupyter_text>This is time-based data with a sample frequency of 360 Hz. I do not have a time-stamp in the dataframe, but we can make an x-array of the length of the signal values to serve as our time axis since the time sampling interval is regular each value is (1/360) seconds.<jupyter_code>x = np.arange(len(p_signal))<jupyter_output><empty_output><jupyter_text>Here, I will plot an abnormal beat from our abnormal index and show 3 seconds before and after the abnormality<jupyter_code>left = ab_index[1]-1080
right = ab_index[1]+1080

plt.plot(x[left:right],p_signal[left:right,0],'-',label='ecg',)
plt.plot(x[atr_sample],p_signal[atr_sample,0],'go',label ='normal')
plt.plot(x[ab_index],p_signal[ab_index,0],'ro',label='abnormal')

plt.xlim(left,right)
plt.ylim(p_signal[left:right].min()-0.05,p_signal[left:right,0].max()+0.05)
plt.xlabel('time index')
plt.ylabel('ECG signal')
plt.legend(bbox_to_anchor = (1.04,1), loc = 'upper left')
plt.show()<jupyter_output><empty_output><jupyter_text>Dataset GenerationHere, I will make a dataframe containing every heartbeat in the dataset centered on the beat with the 3 seconds before and after, like the example above.<jupyter_code>def make_dataset(pts, num_sec, fs, abnormal):
    '''
    function for making dataset ignoring non-beats
    
    input:
        pts - list of patients
        num_sec = number of seconds to include before and after the beat
        fs = frequency
    
    output: 
        X_all = signal (nbeats , num_sec * fs columns)
        Y_all = binary is abnormal (nbeats, 1)
        sym_all = beat annotation symbol (nbeats,1)
    '''

    #initialize numpy arrays
    num_cols = 2*num_sec * fs
    X_all = np.zeros((1,num_cols))
    Y_all = np.zeros((1,1))
    sym_all = []
    
    #list to keep track of number of beats across patients
    max_rows = []
        
    for pt in pts:
        file = data_path + pt
        
        p_signal, atr_sym, atr_sample = load_ecg(file)
        
        # grab the first signal
        p_signal = p_signal[:,0]
        
        # make df to exclude the nonbeats
        df_ann = pd.DataFrame({'atr_sym':atr_sym,
                              'atr_sample':atr_sample})
        df_ann = df_ann.loc[df_ann.atr_sym.isin(abnormal + ['N'])]
        
        X,Y,sym = build_XY(p_signal,df_ann, num_cols, abnormal)
        sym_all = sym_all+sym
        max_rows.append(X.shape[0])
        X_all = np.append(X_all,X,axis = 0)
        Y_all = np.append(Y_all,Y,axis = 0)
    # drop the first zero row
    X_all = X_all[1:,:]
    Y_all = Y_all[1:,:]
    
    # check sizes make sense
    assert np.sum(max_rows) == X_all.shape[0], 'number of X, max_rows rows messed up'
    assert Y_all.shape[0] == X_all.shape[0], 'number of X, Y rows messed up'
    assert Y_all.shape[0] == len(sym_all), 'number of Y, sym rows messed up'

    return X_all, Y_all, sym_all
def build_XY(p_signal, df_ann, num_cols, abnormal):
    '''
    this function builds the X,Y matrices for each beat
     it also returns the original symbols for Y
    '''
    
    num_rows = len(df_ann)

    X = np.zeros((num_rows, num_cols))
    Y = np.zeros((num_rows,1))
    sym = []
    
    # keep track of rows
    max_row = 0

    for atr_sample, atr_sym in zip(df_ann.atr_sample.values,df_ann.atr_sym.values):

        left = max([0,(atr_sample - num_sec*fs) ])
        right = min([len(p_signal),(atr_sample + num_sec*fs) ])
        x = p_signal[left: right]
        if len(x) == num_cols:
            X[max_row,:] = x
            Y[max_row,:] = int(atr_sym in abnormal)
            sym.append(atr_sym)
            max_row += 1
    X = X[:max_row,:]
    Y = Y[:max_row,:]
    return X,Y,sym
    
# define parameters

num_sec = 3
fs = 360
X_all, Y_all, sym_all = make_dataset(pts, num_sec, fs, abnormal)<jupyter_output><empty_output><jupyter_text>The previous cell took minutes to run, so I will convert X_all, Y_all, and sym_all to dataframes and cache them as CSV files (which I wont upload to Github this time as they are too large)<jupyter_code>X_all_df=pd.DataFrame(X_all)
Y_all_df=pd.DataFrame(Y_all)
sym_all_df=pd.DataFrame(sym_all)
X_all_df.head()
X_all_df.shape
Y_all_df.head()
Y_all_df.shape
# X_all_df.to_csv('X_all.csv')
# Y_all_df.to_csv('Y_all.csv')
# sym_all_df.to_csv('sym_all.csv')<jupyter_output><empty_output><jupyter_text>Split Data<jupyter_code>from sklearn.model_selection import train_test_split

def split_my_data(X_all, Y_all, sym_all, pct=0.10):
    '''
    This splits a dataframe into train, validate, and test sets. 
    df = dataframe to split
    pct = size of the test set, 1/2 of size of the validate set
    Returns three dataframes (train, validate, test)
    '''
    X_train_validate, X_test, Y_train_validate, Y_test, sym_train_validate, sym_test = train_test_split(
        X_all, Y_all, sym_all, test_size=pct, random_state=123)
    X_train, X_validate, Y_train, Y_validate, sym_train, sym_validate = train_test_split(
        X_train_validate, Y_train_validate, sym_train_validate, test_size=pct*2, random_state=123)
    return X_train, X_validate, X_test, Y_train, Y_validate, Y_test, sym_train, sym_validate, sym_test
X_all
X_train, X_validate, X_test, Y_train, Y_validate, Y_test, sym_train, sym_validate, sym_test = split_my_data(X_all, Y_all, sym_all)
print(X_train.shape)
print(X_validate.shape)
print(X_test.shape)
X_train<jupyter_output><empty_output><jupyter_text>Some of the keras / tensor flow modeling things require np arrays, other aspects of exploring and plotting are easier with dataaframes, so maintaining both.<jupyter_code>X_train_df=pd.DataFrame(X_train)
X_validate_df=pd.DataFrame(X_validate)
X_test_df=pd.DataFrame(X_test)

Y_train_df=pd.DataFrame(Y_train)
Y_validate_df=pd.DataFrame(Y_validate)
Y_test_df=pd.DataFrame(Y_test)

sym_train_df=pd.DataFrame(sym_train)
sym_validate_df=pd.DataFrame(sym_validate)
sym_test_df=pd.DataFrame(sym_test)
X_train

print(Y_train.shape)
print(Y_validate.shape)
print(Y_test.shape)
#X_train.to_csv('X_train.csv')
#X_validate.to_csv('X_validate.csv')
#X_test.to_csv('X_test.csv')
#Y_train.to_csv('Y_train.csv')
#Y_validate.to_csv('Y_validate.csv')
#Y_test.to_csv('Y_test.csv')
#sym_train.to_csv('sym_train.csv')
#sym_validate.to_csv('sym_validate.csv')
#sym_test.to_csv('sym_test.csv')
Y_all_df.head()
sym_all_df.head()<jupyter_output><empty_output><jupyter_text>Concatenate the Y_train target back on the X_train datasets so I can segregate normal / abnormal for exploration<jupyter_code>X_explore_df = X_train_df
X_explore_df['abnormal']=Y_train_df

X_explore_df.sample(25)
X_explore_normal = X_explore_df[X_explore_df.abnormal==0]
X_explore_normal.shape
X_explore_abnormal = X_explore_df[X_explore_df.abnormal==1]
X_explore_abnormal.shape<jupyter_output><empty_output><jupyter_text>Look at individual traces Three sample normal<jupyter_code>X_explore_normal.shape
X_explore_normal.sample(20)
plt.plot(np.arange(2161), X_explore_normal.iloc[0]);
plt.plot(np.arange(2161), X_explore_normal.iloc[1]);
plt.plot(np.arange(2161), X_explore_normal.iloc[2]);<jupyter_output><empty_output><jupyter_text>3 sample abnormal<jupyter_code>plt.plot(np.arange(2161), X_explore_abnormal.iloc[0]);
plt.plot(np.arange(2161), X_explore_abnormal.iloc[1]);
plt.plot(np.arange(2161), X_explore_abnormal.iloc[2]);<jupyter_output><empty_output><jupyter_text>---  Literature / Curriculum / Research Review- Now that I have my data in a usable format, I need to do some research on time series and anomaly detection in order to get into the modeling phase.**Topics to explore:**- [] Anomaly Detection- [] LSTM Networks - this concept comes up a lot in handling these data- [] Autoencoders- [] CNN? Neural networks are also commonly used for processing these types of data. Will start with other techniques and may get here in a subsequent iteration  Anomaly Detection- Task of determining when something has gone astray from the norm.   - Neural networks are used for unsupervised learning (I do have labels, so I should take advantage of those)  - Basically just learns the normal behavior and anything that isn't that gets labeled as an anomaly.  - i.e., autoencoder deep learning neural network  LSTM Networks- Long Short-Term Memory neural network cells in autoencoder. - Subset of recurrent neural networks (RNN).- Training data persists in network for use later     - Well-suited for temporal data that evolves over time- Common applications include:  - Speech recognition  - Text translation  - Analysis of sequential sensor readings for anomaly detection  Recurrent Neural Networks (2)- Networks with loops in them allowing information to persist.- Able to use "reasoning" about previous events to make decisions.![RNN](images/RNN1.png)**Recurring Neural Networks have loops**These loops can be thought of as multiple copies of the same network![RNN unwound](images/RNN2.png)2. https://colah.github.io/posts/2015-08-Understanding-LSTMs/  Problem of Long-Term Dependencies - Appeal of RNN is to connect previous information to the present task.- Sometimes we only need to look at recent information. i.e., language model trying to predict the next word based on previous ones. If trying to predict "the clouds are in the ___" we don't need any further context - small gap between relevant information and the place it is needed.    ![RNN-short-dependencies](images/RNN3.png)- Other times, more context is needed. "I grew up in France...I speak fluent ____" Recent information suggest the name of a language for the answer, but to get which language more context from earlier is needed. The gab between the relevant information and the point it is needed can be quite large. ![RNN-long-dependencies](images/RNN4.png)- RNNs struggle as that gap grows.- RNNs theoretically capable of hangling such long dependencies, but fail to do so in practice. LTSMs don't have this problem  LSTM Networks- Special kind of RNN capable of learning long-term dependencies- Explicitly designed to avoid long-term dependency problem. Remembering information for long periods of time is their default behavior, not something they struggle to learn.- Standard RNN:![RNN-standard-single-layer](images/RNN4.png)- LSTM module has 4 interacting layers![LSTM](images/LSTM1.png)![LSTM-layers](images/LSTM2.png) Core Concept- **Cell State:** Horizontal line running through the top of the diagram.- Kind of like a conveyor belt - straight down the entire chain with only minor interactions. Easy for information to flow along unchanged.- There is the ability to remove or add information to the cell state, but it is carefully regulated by gates which let ingotmation through  - Composed of sigmoid layers and pointwise multiplication:    - 0 - let nothing through    - 1 - let everything through  - LSTM has three of these gates to protect and control the cell state.  ![LSTM-gate](images/gate.png)  LSTM: Step by step1. Decide that information we are going to throw away from the cell state   - "Forget gate layer" outputs a number between 0 and 1 keep / get rid of2. Decide what new information to store    1. "Input gate layer" decides which values to update    2. Tanh layer creates vector of new candidate values that could be added to the state3. Update cell state    - Forget what decided to forget    - Add scaled candidate values4. Decide output based on cell statehttps://colah.github.io/posts/2015-08-Understanding-LSTMs/ ---  Review Codeup Curriculum on Anomaly Detection - Practice of identifying data points, items, observations, or events that do not conform to the expected pattern of a given group What are Anomalies?- **Anomaly**: Unusual, unexpected, surprising patterns in the observed rule. An anomalous data point is a deviation from a rule or from what is regarded as normal; an outlier. An anomaly is any event or measurement that is out of the ordinary regardless of whether it is exceptional or not.- **Outlier**: An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. In a sense, this definition leaves it up to the analyst (or a consensus process) to decide what will be considered abnormal. Before abnormal observations can be singled out, it is necessary to characterize normal observations.- These terms are frequently used interchangeably.- In some cases, statisticians sometimes use the term 'outlier' to mean "something I should remove from the dataset so that it doesn't skew my model I'm building".- In other cases, we may reserve the word 'anomaly' for data points we want to keep because they are valuable if caught or costly if missed. It's also possible to have an anomaly that is an inlier, meaning that the data is anomalous in nature but is close (in distance) to the mean, mode, or expected grouping(s). Examples  of Anomaly Detection in Data ScienceFraud detection for financial companiesMedical scans like XRays, ultrasounds, and MRIs seeking abnormalities in location, density, growth, etc..Detecting and preventing cyber-crimeMalware detectionIntelligence and Defense applicationsAnalytical Marketing looking for new trends in business or household purchases The best way to detect anomalies is to know your domainFor example, knowing that there is nothing colder than absolute zero degrees Kelvin informs how we approach an observation below 0 degrees Kelvin. It may mean that there is something wrong with the measuring tool or a clerical error like a typo (more likely).What determines if a data point is an outlier is a function of who determines if that data point is anomalous and why. One person's noise is another person's signal, depending on their goals and the costs and benefits to be gained/avoided from identifying that data point.To one analyst aiming for a clean model of 50/50 chance, observing a flipped coin that lands exactly on its side is an anomaly may be noise that they should ignore from their model. To another practitioner, the anomaly may be an observation of very high value that they want to catch, like early detection of a rare disease. Recall the e-Handbook of Statistical Methods definition an outlier: "an observation that lies an abnormal distance from other values in a random sample from a population. In a sense, this definition leaves it up to the analyst (or a consensus process) to decide what will be considered abnormal". It's important to define the criteria, the decision rule, for what makes an observation an inlier or an outlier.Usually, the deciding factor is cost/benefit analysis:Is our goal to hunt for anomalies at all costs because lives are on the line?Or do we do more good for people by producing a model that treats outliers as noise?What is the cost of a false positive or a false negative?What is the benefit of a true positive or a true negative? Outliers may contain important informationOutliers should be investigated carefully. Often they contain valuable information about the process under investigation or the data gathering and recording process. Before considering the possible elimination of these points from the data, one should try to understand why they appeared and whether it is likely similar values will continue to appear. Of course, outliers are often bad data points. Types of AnomaliesUse Cases: Response times (longer than usual), error rates (more than usual), network load, cyber intrusions, fraud.Point Anomalies:  A single instance of data is anomalous if it's too far off from the rest. For example, detecting data exfiltration based on gigabytes leaving the network, detecting credit card fraud based on "amount spent", etc.Contextual Anomalies: The abnormality is context specific. This type of anomaly is common in time-series data. For example, accessing confidential files during work hours is normal, but in the middle of the night is odd.Collective Anomalies:  A set of data instances collectively helps in detecting anomalies. For example, someone is trying to copy data form a remote machine to a local host unexpectedly, an anomaly that would be flagged as a potential cyber attack. The combination of the event from the remote machine and the event from the local host combine to detect the anomaly.Anomalies vs. Noise Removal vs. Novelty Detection: Novelty detection is concerned with identifying an unobserved pattern in new observations not included in training data — for instance, a sudden interest in a new channel on YouTube during Christmas. Noise Removal (NR) is the process of immunizing analysis from the occurrence of unwanted observations; in other words, removing noise from an otherwise meaningful signal.https://www.datascience.com/blog/python-anomaly-detection Specific Techniques for Identifying/Detecting AnomaliesStatistical MethodsFlag the data points that deviate from the expected, based on the statistical properties, such as mean, median, mode, and quantiles. You could define an anomalous data point as one that deviates by a certain standard deviation from the mean. You could use a simple or exponential moving average to smooth short-term fluctuations and highlight long-term ones.This method is challenging with really noisy data. Support Vector Machine &amp; Isolation Forest Anomaly DetectionThere are extensions to the supervised technique, such as OneClassSVM, that can be used to identify anomalies as an unsupervised problems (in which training data are not labeled).The algorithm learns a soft boundary in order to cluster the normal data instances using the training set, and then, using the testing instance, it tunes itself to identify the abnormalities that fall outside the learned region.Clustering-Based Anomaly DetectionAssumption: Data points that are similar tend to belong to similar groups or clusters, as determined by their distance from local centroids.K-means creates 'k' similar clusters of data points. Data instances that fall into abnormally small clusters could potentially be marked as anomalies.Using density based clustering, like DBSCAN, we can design the model such that the data points that do not fall into a cluster are the anomalies.Density-Based Anomaly DetectionAssumption: Normal data points occur around a dense neighborhood and abnormalities are far away.The nearest set of data points are evaluated using a score, using one of two algorithms:KNN: K-Nearest NeighborLOF: Local Outlier Factor, aka the relative density of data which is based on the reachability distance. RememberThere is not a magic algorithm that takes away our responsibility to think clearly and critically about why an observation is anomalous.There are no hard fast rules for whether or not an outlier should be ignored or investigated.The answer is "it depends", and the important part is identifying what that depends on, specifically. Further ResourcesVideo overview of anomaly detection from Andrew NgSKLearn Examples of Outlier DetectionAdditional Anomaly Detection ResourcesPython Toolbox for Scalable Outlier Detection --- Doesn't appear to be any LSTM / CNN resources in the curriculum or appendix, so - back to the web!  Time Series Prediction with LSTM Recurrent Neural Networks in Python with Kerashttps://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/ - Discover how to develop LSTM networks in Python using the Keras deep learning library to address a demonstration time-series prediction problem.-  ---<jupyter_code>#!keras --version
# from keras.layers import Bidirectional, LSTM

# from keras.models import Sequential
#from keras.layers import Dense, Flatten, Dropout
#from keras.utils import to_categorical
# reshape input to be [samples, time steps, features = 1]
#X_train_cnn = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))
#X_valid_cnn = np.reshape(X_valid, (X_valid.shape[0], X_valid.shape[1], 1))

#print(X_train_cnn.shape)
#print(X_valid_cnn.shape)
#model = Sequential()
#model.add(Bidirectional(LSTM(64, input_shape=(X_train_cnn.shape[1], X_train_cnn.shape[2]))))
#model.add(Dropout(rate = 0.25))
#model.add(Dense(1, activation = 'sigmoid'))
#model.compile(
#                loss = 'binary_crossentropy',
#                optimizer = 'adam',
#                metrics = ['accuracy'])<jupyter_output><empty_output><jupyter_text>--- So, something about keras was breaking my Jypyter kernel. And, the LSTM modeling was more involved than I anticipated it being, so... I am shifting gears and focus to do random forrest classification of these data using the labeled traces.Previously, I was maintaining parallel numpy arrays and dataframes for my data since different things I was doing eat either one (likely an unnecessary step as I could intervonvert from one to the other as needed simply, but oh well).Before proceeding with Random Forrest modeling, I will verify my X/Y train / validate/ test dataframes appear as they should<jupyter_code>X_train_df.sample(20)<jupyter_output><empty_output><jupyter_text>It does have the target, wich it should not<jupyter_code>Y_train_df
print(X_train_df.shape)
Y_train_df.shape<jupyter_output><empty_output><jupyter_text>It is no longer neccessary to have the np arrays, so I will just overwrite them with the dataframes to have X_train instead of X_train_df, since some of my other modeling and evaluation functions are already written with these variables.<jupyter_code>X_train = X_train_df
X_validate = X_validate_df
X_test = X_test_df
Y_train = Y_train_df
Y_validate = Y_validate_df
Y_test = Y_test_df
sym_train = sym_train_df
sym_validate = sym_validate_df
sym_test = sym_test_df
print(X_train.shape)
print(X_validate.shape)
print(X_test.shape)
print()
print(Y_train.shape)
print(Y_validate.shape)
print(Y_test.shape)
print()
print(sym_train.shape)
print(sym_validate.shape)
print(sym_test.shape)
X_validate.head()
X_train.head()
X_train.drop('abnormal', axis=1, inplace=True)
X_train.head()<jupyter_output><empty_output><jupyter_text>Okay... have my data split. On to modeling ---  Modeling As mentioned above, the state of the art for modeling these data is using neural networks, specifically the LSTM subset of RNNs. I will revisit this as a future direction. In the absence of neural networks, random forrest classifiers often perform the best of the simpler techniques. Will make and tune multiple random forrest models and try the lazy classifier as well<jupyter_code># from lazypredict.Supervised import LazyClassifier, LazyRegressor

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix


import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns


rf = RandomForestClassifier(bootstrap=True, 
                            class_weight=None, 
                            criterion='gini',
                            min_samples_leaf=3,
                            n_estimators=100,
                            max_depth=3, 
                            random_state=123)
rf.fit(X_train, Y_train)

RandomForestClassifier(max_depth=3, min_samples_leaf=3, random_state=123)

y_pred = rf.predict(X_train)

print('Accuracy of random forest classifier on training set: {:.2f}'
     .format(rf.score(X_train, Y_train)))
print(confusion_matrix(Y_train, y_pred))

print(classification_report(Y_train, y_pred))

print('Accuracy of random forest classifier on test set: {:.2f}'
     .format(rf.score(X_validate, Y_validate)))
y_validate_pred = rf.predict(X_validate)

print('Accuracy of random forest classifier on test set: {:.2f}'
     .format(rf.score(X_validate, Y_validate)))
print(classification_report(Y_validate, y_validate_pred))

tn_train_rf1, fp_train_rf1, fn_train_rf1, tp_train_rf1 = confusion_matrix(Y_train, y_pred).ravel()
print(tn_train_rf1)
print(fp_train_rf1)
print(fn_train_rf1)
print(tp_train_rf1)
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
clf = SVC(random_state=0)
clf.fit(X_train, y_train)
SVC(random_state=0)
cm = confusion_matrix(y_test, predictions, labels=clf.classes_)
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=clf.classes_)
disp.plot()

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07919157227304251, 'fraction_numerical': 0.013506819457745782, 'mean_word_length': 4.699604743083004, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 74, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '193664', 'license': 'MIT', 'n_tokens_mistral': 9199, 'n_tokens_neox': 8605, 'n_words': 3516, 'path': 'explore_model2.ipynb', 'repo_name': 'Jared-Godar/ecg_anomaly_detection', 'size': '30288'}