<jupyter_start><jupyter_text>This is a simple autoencoder example just to play around with how to ingest some of the data. This approach has several clearly obvious weaknesses, including the fact that it doesn't deal with the fact that different samples have different sizes.<jupyter_code>%load_ext autoreload
%autoreload 2
import tensorflow as tf
import numpy as np
from arc_code.data import get_data<jupyter_output><empty_output><jupyter_text>We start with a really simple autoencoder (taken from https://towardsdatascience.com/implementing-an-autoencoder-in-tensorflow-2-0-5e86126e9f7)<jupyter_code>class SimpleEncoder(tf.keras.layers.Layer):
    def __init__(self, intermediate_dim):
        super(SimpleEncoder, self).__init__()
        self.hidden_layer = tf.keras.layers.Dense(
          units=intermediate_dim,
          activation=tf.nn.relu,
          kernel_initializer='he_uniform'
        )
        self.output_layer = tf.keras.layers.Dense(
          units=intermediate_dim,
          activation=tf.nn.sigmoid
        )

    def call(self, input_features):
        activation = self.hidden_layer(input_features)
        return self.output_layer(activation)
class SimpleDecoder(tf.keras.layers.Layer):
    def __init__(self, intermediate_dim, original_dim):
        super(SimpleDecoder, self).__init__()
        self.hidden_layer = tf.keras.layers.Dense(
          units=intermediate_dim,
          activation=tf.nn.relu,
          kernel_initializer='he_uniform'
        )
        self.output_layer = tf.keras.layers.Dense(
          units=original_dim,
          activation=tf.nn.sigmoid
        )

    def call(self, code):
        activation = self.hidden_layer(code)
        return self.output_layer(activation)
class SimpleAutoencoder(tf.keras.Model):
    def __init__(self, intermediate_dim, original_dim):
        super(SimpleAutoencoder, self).__init__()
        self.encoder = SimpleEncoder(intermediate_dim=intermediate_dim)
        self.decoder = SimpleDecoder(intermediate_dim=intermediate_dim, original_dim=original_dim)
  
    def call(self, input_features):
        code = self.encoder(input_features)
        reconstructed = self.decoder(code)
        return reconstructed
def loss(model, original):
    reconstruction_error = tf.reduce_mean(tf.square(tf.subtract(model(original), original)))
    return reconstruction_error
def train(loss, model, opt, original):
    with tf.GradientTape() as tape:
        gradients = tape.gradient(loss(model, original), model.trainable_variables)
        gradient_variables = zip(gradients, model.trainable_variables)
        opt.apply_gradients(gradient_variables)<jupyter_output><empty_output><jupyter_text>Let's get all the data<jupyter_code>X_train, Y_train, X_test, Y_test = get_data('../data', training_set = True, grouped_by_problem = False)
unique_shapes = {}
for entry in X_train:
    if entry.shape not in unique_shapes:
        unique_shapes[entry.shape] = [entry]
    else:
        unique_shapes[entry.shape].append(entry)
# get the shape with the most elements
max_len = -1
max_len_shape = -1
for shape,entries in unique_shapes.items():
    if len(entries) > max_len:
        max_len = len(entries)
        max_len_shape = shape
print("Most common shape is {} with {} examples".format(max_len_shape, max_len))
X_train_samesize = np.zeros((max_len, max_len_shape[0], max_len_shape[1]), dtype = np.float32)
Y_train_samesize = []
counter = 0
for i in range(len(X_train)):
    if X_train[i].shape == max_len_shape:
        X_train_samesize[counter] = X_train[i]
        Y_train_samesize.append(Y_train[i])
        counter += 1
# flatten training features
#X_train_samesize = X_train_samesize.reshape(X_train_samesize.shape[0],
#                                              X_train_samesize.shape[1] * X_train_samesize.shape[2])
learning_rate = 1e-3
epochs = 125
batch_size = 5
autoencoder = SimpleAutoencoder(intermediate_dim=5, original_dim=10)
opt = tf.optimizers.Adam(learning_rate=learning_rate)

training_dataset = tf.data.Dataset.from_tensor_slices(X_train_samesize)
training_dataset = training_dataset.shuffle(X_train_samesize.shape[0])
training_dataset = training_dataset.prefetch(batch_size * 4)
all_loss = []
for epoch in range(epochs):
    for step, batch_features in enumerate(training_dataset):
        train(loss, autoencoder, opt, batch_features)
        loss_values = loss(autoencoder, batch_features)
        all_loss.append(loss_values.numpy())
        #original = tf.reshape(batch_features, (batch_features.shape[0], 10, 10, 1))
        #reconstructed = tf.reshape(autoencoder(tf.constant(batch_features)), (batch_features.shape[0], 10, 10, 1))
    ave_loss = np.average(all_loss)
    print("Epoch {} had average loss {}".format(epoch, ave_loss))<jupyter_output>Epoch 0 had average loss 4.212340354919434
Epoch 1 had average loss 4.178206920623779
Epoch 2 had average loss 4.142806529998779
Epoch 3 had average loss 4.110197067260742
Epoch 4 had average loss 4.0782012939453125
Epoch 5 had average loss 4.047335147857666
Epoch 6 had average loss 4.010562419891357
Epoch 7 had average loss 3.9731171131134033
Epoch 8 had average loss 3.9398117065429688
Epoch 9 had average loss 3.9111602306365967
Epoch 10 had average loss 3.8865966796875
Epoch 11 had average loss 3.8653275966644287
Epoch 12 had average loss 3.846630096435547
Epoch 13 had average loss 3.8299922943115234
Epoch 14 had average loss 3.8150482177734375
Epoch 15 had average loss 3.801501512527466
Epoch 16 had average loss 3.7891225814819336
Epoch 17 had average loss 3.7777130603790283
Epoch 18 had average loss 3.7671713829040527
Epoch 19 had average loss 3.757396936416626
Epoch 20 had average loss 3.7482993602752686
Epoch 21 had average loss 3.739809036254883
Epoch 22 had average loss 3.73184[...]
{'fraction_non_alphanumeric': 0.08693388859968766, 'fraction_numerical': 0.07773728960610793, 'mean_word_length': 4.164874551971326, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '583818', 'license': 'MIT', 'n_tokens_mistral': 2147, 'n_tokens_neox': 1847, 'n_words': 434, 'path': 'notebooks/Simple AutoEncoder.ipynb', 'repo_name': 'Pikus16/Abstraction-ReasoningChallenge', 'size': '5763'}