<jupyter_start><jupyter_text>Autoencoders on Tabular Data* Normalizaing data: https://www.tensorflow.org/api_docs/python/tf/keras/utils/normalize* Keras functional API for more flexible models: https://keras.io/getting-started/functional-api-guide/* Batch Size should not matter too much, but larger batch sizes might require more training epochs: https://ieeexplore.ieee.org/document/8323035<jupyter_code>import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)
print(tf.__version__)<jupyter_output>1.12.0<jupyter_text>Loading and exploring our data setThis is a database of customers of an insurance company. Each data point is one customer. The group represents the number of accidents the customer has been involved with in the past* 0 - red: many accidents* 1 - green: few or no accidents* 2 - yellow: in the middle<jupyter_code>!curl -O https://raw.githubusercontent.com/DJCordhose/deep-learning-crash-course-notebooks/master/data/insurance-customers-1500.csv
import pandas as pd

df = pd.read_csv('./insurance-customers-1500.csv', sep=';')
df.describe()
df.head()
import seaborn as sns

sample_df = df.sample(n=100, random_state=42)
sns.pairplot(sample_df, 
             hue="group", palette={0: '#AA4444', 1: '#006000', 2: '#EEEE44'},
#              kind='reg',
             size=5,
             diag_kind='kde',
             vars=['age', 'speed', 'miles'])<jupyter_output><empty_output><jupyter_text>The embedding network  Network Architecture<jupyter_code>from tensorflow import keras
from tensorflow.keras.layers import Input, Flatten, GlobalAveragePooling1D, Dense
from tensorflow.keras.models import Sequential, Model

encoding_dim = 32

# https://keras.io/getting-started/functional-api-guide/
input_data = Input(shape=(4,))
encoded = Dense(units=encoding_dim, activation='relu', name="encoder")(input_data)
decoded = Dense(units=4, activation='linear', name="decoder")(encoded)

autoencoder = Model(inputs=input_data, outputs=decoded)
autoencoder.compile(optimizer='adam', loss='mse')
autoencoder.summary();<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 4)                 0         
_________________________________________________________________
encoder (Dense)              (None, 32)                160       
_________________________________________________________________
decoder (Dense)              (None, 4)                 132       
=================================================================
Total params: 292
Trainable params: 292
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Training<jupyter_code>X = df
BATCH_SIZE = 1 # larger batch size might force more epochs
%time history = autoencoder.fit(X, X, epochs=10, batch_size=BATCH_SIZE, verbose=1)<jupyter_output>Epoch 1/10
1500/1500 [==============================] - 6s 4ms/step - loss: 89.6819
Epoch 2/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.6331
Epoch 3/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.1449
Epoch 4/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0546
Epoch 5/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0201
Epoch 6/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0117
Epoch 7/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0431
Epoch 8/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0082
Epoch 9/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0259
Epoch 10/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.0232
CPU times: user 1min 4s, sys: 9.02 s, total: 1min 13s
Wall time: 52.6 s<jupyter_text>Evaluation<jupyter_code>import matplotlib.pyplot as plt

plt.yscale('log')
plt.plot(history.history['loss'])
loss = autoencoder.evaluate(X, X, batch_size=BATCH_SIZE)
loss
samples = df.sample(10).reset_index(drop=True)
samples
predictions = pd.DataFrame(autoencoder.predict(samples), columns=["speed",	"age",	"miles",	"group"])
predictions
samples.subtract(predictions)<jupyter_output><empty_output><jupyter_text>Looks quite nice, but how impressed are you by abstracting 4 variables into 32 / Let us try dimension of 2<jupyter_code>from tensorflow.keras.initializers import glorot_normal

encoding_dim = 2
seed = 13 # make training results more deterministic

input_data = Input(shape=(4,))
# notice tanh
encoded = Dense(units=encoding_dim, activation='tanh', name="encoder", kernel_initializer=glorot_normal(seed=seed))(input_data)
decoded = Dense(units=4, activation='linear', name="decoder", kernel_initializer=glorot_normal(seed=seed))(encoded)

autoencoder = Model(inputs=input_data, outputs=decoded)
# this does not get much better than 210 as a loss (bad), so we can just as well get there a bit faster (10 instead of 50 epochs with standard lr)
adam = keras.optimizers.Adam(lr=0.01)
# adam = keras.optimizers.Adam()
autoencoder.compile(optimizer=adam, loss='mse')

X = df
BATCH_SIZE = 1
%time history = autoencoder.fit(X, X, epochs=10, batch_size=BATCH_SIZE, verbose=1)

plt.yscale('log')
plt.plot(history.history['loss'])
autoencoder.evaluate(X, X, batch_size=BATCH_SIZE)
# just average over all of them :D
predictions = pd.DataFrame(autoencoder.predict(samples), columns=["speed",	"age",	"miles",	"group"])
predictions
samples.describe()<jupyter_output><empty_output><jupyter_text>relu makes it train<jupyter_code>from tensorflow.keras.initializers import glorot_normal

encoding_dim = 2
seed = 13 # if it trains still depens on initialization

input_data = Input(shape=(4,))
# notice relu
encoded = Dense(units=encoding_dim, activation='relu', name="encoder", kernel_initializer=glorot_normal(seed=seed))(input_data)
decoded = Dense(units=4, activation='linear', name="decoder", kernel_initializer=glorot_normal(seed=seed))(encoded)

autoencoder = Model(inputs=input_data, outputs=decoded)
# adam = keras.optimizers.Adam(lr=0.01)
adam = keras.optimizers.Adam()
autoencoder.compile(optimizer=adam, loss='mse')

X = df
BATCH_SIZE = 1
%time history = autoencoder.fit(X, X, epochs=10, batch_size=BATCH_SIZE, verbose=1)

plt.yscale('log')
plt.plot(history.history['loss'])
samples<jupyter_output><empty_output><jupyter_text>not too bad, but all groups close to 1, all off as do not add much to loss<jupyter_code>predictions = pd.DataFrame(autoencoder.predict(samples), columns=["speed",	"age",	"miles",	"group"])
predictions
samples.subtract(predictions)<jupyter_output><empty_output><jupyter_text>Let's try normalizing the data first (caution: losses do not compare any more)<jupyter_code>df = pd.read_csv('./insurance-customers-1500.csv', sep=';')
normalized_df = (df - df.mean()) / df.std()
normalized_df.head()
X = normalized_df

encoding_dim = 2
seed = 13 # make results determinisitic

input_data = Input(shape=(4,))
encoded = Dense(units=encoding_dim, activation='relu', name="encoder", kernel_initializer=glorot_normal(seed=seed))(input_data)
decoded = Dense(units=4, activation='linear', name="decoder", kernel_initializer=glorot_normal(seed=seed))(encoded)

autoencoder = Model(inputs=input_data, outputs=decoded)
# adam = keras.optimizers.Adam(lr=0.01)
adam = keras.optimizers.Adam()
autoencoder.compile(optimizer=adam, loss='mse')

BATCH_SIZE = 1
%time history = autoencoder.fit(X, X, epochs=10, batch_size=BATCH_SIZE, verbose=1)

plt.yscale('log')
plt.plot(history.history['loss'])<jupyter_output>Epoch 1/10
1500/1500 [==============================] - 5s 4ms/step - loss: 0.6657
Epoch 2/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3895
Epoch 3/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3520
Epoch 4/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3430
Epoch 5/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3391
Epoch 6/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3377
Epoch 7/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3364
Epoch 8/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3356
Epoch 9/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3347
Epoch 10/10
1500/1500 [==============================] - 5s 3ms/step - loss: 0.3336
CPU times: user 1min 3s, sys: 8.42 s, total: 1min 12s
Wall time: 51.8 s<jupyter_text>This makes it very hard to estimate how well we did our job, but we can still plot the embeddingCould be interpreted as a fancy version of dimensionality reduction<jupyter_code>encoder = Model(inputs=input_data, outputs=encoded)
latent_representation = encoder.predict(X)
latent_representation.shape<jupyter_output><empty_output><jupyter_text>Look, ma, clusters<jupyter_code>latent_x = latent_representation[:, 0]
latent_y = latent_representation[:, 1]
plt.scatter(latent_x, latent_y, alpha=0.5)<jupyter_output><empty_output><jupyter_text>Clusters have actual meaning<jupyter_code>from matplotlib.colors import ListedColormap

# * 0 - red: many accidents
# * 1 - green: few or no accidents
# * 2 - yellow: in the middle
colors = X['group']
color_map = ListedColormap(['#AA4444', '#006000', '#EEEE44'])

plt.scatter(latent_x, latent_y, alpha=0.5, s=100, marker='o', edgecolors='w', cmap=color_map, c=colors)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.19405668358714043, 'fraction_numerical': 0.056472081218274114, 'mean_word_length': 5.8578680203045685, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '243742', 'license': 'MIT', 'n_tokens_mistral': 3503, 'n_tokens_neox': 2994, 'n_words': 726, 'path': 'notebooks/2019_tf/autoencoders_tabular.ipynb', 'repo_name': 'subodhchhabra/ai', 'size': '9456'}