<jupyter_start><jupyter_text>Error Detection using LSTMsThis notebook demonstrates how we use record embeddings to perform error detection in tuples. We are using the **Hospital** dataset which has been artificially augmented to contain the following type of errors:1. The cell values have been shuffled across the columnar axis.2. 5% of tuples has a cell value assigned to NaN.3. 10% of the tuples has a cell value misspelt.By converting each tuple to a space separated string and feeding it to an LSTM, we see if the network can learn to detect errors in the tuples.<jupyter_code>import warnings
warnings.filterwarnings("ignore")

import pandas as pd
import numpy as np
import h5py
import pickle
from sklearn.model_selection import train_test_split

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras import models
from keras import layers
from keras import optimizers
from keras import activations
from keras import losses
from keras import metrics
from keras.utils import to_categorical

import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output>Using TensorFlow backend.<jupyter_text>Read the data<jupyter_code>df = pd.read_csv('final_4k_tuples.csv', encoding='utf8')
df.head(2)
len(df)
df['label'].value_counts()<jupyter_output><empty_output><jupyter_text>Some preprocessing to replace phoneNumber and providerNumber with a 0 instead of NaN<jupyter_code>df['PhoneNumber'].fillna(0, inplace=True)
df['ProviderNumber'].fillna(0, inplace=True)
df['ZipCode'].fillna(0, inplace=True)
df.head(2)<jupyter_output><empty_output><jupyter_text>Separate the labels from the training data<jupyter_code>df['label'] = df['label'].astype('category')
df["Label_Cat"] = df["label"].cat.codes
df.head()
df['Label_Cat'].value_counts()<jupyter_output><empty_output><jupyter_text>Get the mapping<jupyter_code>mapping = dict(enumerate(df['label'].cat.categories))
mapping
df.drop(columns=['label'], axis=1, inplace=True)
df.head(2)
labels = df['Label_Cat'].tolist()
labels = np.asarray(labels)
labels.shape
labels = to_categorical(labels)
labels.shape
df.drop(columns=['Label_Cat'], axis=1, inplace=True)
df.head(2)<jupyter_output><empty_output><jupyter_text>Drop the State as it does not add anything new<jupyter_code>df.drop(columns=['State'], axis=1, inplace=True)
df.head(2)<jupyter_output><empty_output><jupyter_text>Convert PhoneNumber and ProviderNumber to int<jupyter_code>df['ProviderNumber'] = df['ProviderNumber'].apply(lambda x: int(x))
df['PhoneNumber'] = df['PhoneNumber'].apply(lambda x: int(x))
df['ZipCode'] = df['ZipCode'].apply(lambda x: int(x))
df.head(2)<jupyter_output><empty_output><jupyter_text>Convert each row to a space separated string<jupyter_code>dfList = df.values.tolist()
rows = []
for row in dfList:
    combined = ','.join([str(x) for x in row])
    rows.append(combined)
len(rows)
rows[0]
max([len(r) for r in rows])<jupyter_output><empty_output><jupyter_text>Save the dataset<jupyter_code>df.to_csv("HospitalTrainingData.csv", encoding='utf8', index=False)<jupyter_output><empty_output><jupyter_text>Tokenize the data Since the max length of a row is 110, we can set the maxSequenceLength as 110.<jupyter_code>maxSequenceLength = 105
maxWords = 2500
tokenizer = Tokenizer(num_words=maxWords, split=',')
tokenizer.fit_on_texts(rows)
sequences = tokenizer.texts_to_sequences(rows)
word_index = tokenizer.word_index
len(word_index)
data = pad_sequences(sequences, maxlen=maxSequenceLength)
data.shape
labels.shape<jupyter_output><empty_output><jupyter_text>Train test split<jupyter_code>indices = np.arange(data.shape[0])
np.random.shuffle(indices)
data = data[indices]
labels = labels[indices]
X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=22, stratify=labels)
len(X_train), len(y_train)
len(X_test), len(y_test)<jupyter_output><empty_output><jupyter_text>Model**Architecture*** 1 Embedding layer* 1 Bidirectional LSTM layer* 1 output layer, _sigmoid_ activation* Optimizer: Adam (learning rate = 0.01)* Loss function: Binary crossentropy loss<jupyter_code>embeddingDim = 110
model = models.Sequential()
model.add(layers.Embedding(2500, embeddingDim, input_length=maxSequenceLength))
model.add(layers.Bidirectional(layers.LSTM(32)))
model.add(layers.Dense(4, activation=activations.softmax))
model.compile(optimizer=optimizers.RMSprop(lr=0.01), loss=losses.categorical_crossentropy, metrics=[metrics.categorical_accuracy])

history = model.fit(X_train, y_train,
                    epochs=70,
                    batch_size=32, 
                    validation_split=0.2)
acc = history.history['categorical_accuracy']
val_acc = history.history['val_categorical_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(1, len(acc) + 1)
plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel("Epochs")
plt.ylabel("Accuracy")
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.legend()
plt.show();<jupyter_output><empty_output><jupyter_text>Evaluation on test set<jupyter_code>results = model.evaluate(X_test, y_test)
loss = results[0]
accuracy = results[1]
loss
accuracy<jupyter_output><empty_output><jupyter_text>Save the model and tokenizer<jupyter_code>model.save("HospitalErrorDetectionMultiClassifier.h5")
with open('HospitalErrorDetectionMultiTokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL, fix_imports=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12130691692735489, 'fraction_numerical': 0.01251303441084463, 'mean_word_length': 7.732928679817906, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '32285', 'license': 'MIT', 'n_tokens_mistral': 2023, 'n_tokens_neox': 1934, 'n_words': 437, 'path': 'Error Detection using LSTMs-MultiClassifier for Errors.ipynb', 'repo_name': 'yogeshchellappa/recordEmbedding', 'size': '5754'}