<jupyter_start><jupyter_text>The analytical functions to regress and the activations that will be used<jupyter_code>activations = ["relu", "tanh"]
analytic_functions = ["identity", "tanh", "sin", "square"]<jupyter_output><empty_output><jupyter_text>Configuration for the extrapolation experiments<jupyter_code>NUM_RUNS = 21
EPOCHS = 500
ARCHITECTURE = (1, 512, 1)
fig, ax = plt.subplots(nrows=2, ncols=4)
fig.set_figheight(8)
fig.set_figwidth(20)

for i, activation in enumerate(activations):
    for j, function in enumerate(analytic_functions):

        print(f"Regressing {function} with {activation} activation")

        function_inputs, function_outputs = gen_func_dataset(lower_limit=-20, upper_limit=20, func=function)
        sample_inputs1, sample_outputs1 = sample_func_dataset(dataset_inputs=function_inputs, dataset_outputs=function_outputs, lower_lim=-5, upper_lim=-1, num_points=50)
        sample_inputs2, sample_outputs2 = sample_func_dataset(dataset_inputs=function_inputs, dataset_outputs=function_outputs, lower_lim=1, upper_lim=5, num_points=50)

        sample_inputs = torch.cat((sample_inputs1, sample_inputs2))
        sample_outputs = torch.cat((sample_outputs1, sample_outputs2))

        preds = []

        for run in range(NUM_RUNS):
            model = SimpleMLP(ARCHITECTURE, activation)

            model, _ = train_extrapolation(model, sample_inputs, sample_outputs, epochs=EPOCHS, verbose=False)
            with torch.no_grad():
                model.cpu()
                pred = model(function_inputs)
            preds.append(pred)

        preds = torch.stack(preds)

        preds_mean = torch.mean(preds, axis=0)
        preds_std = torch.std(preds, axis=0)
        preds_conf_interval = 1.64 * preds_std/math.sqrt(NUM_RUNS)

        ax[i, j].plot(function_inputs, function_outputs, label="Ground truth")
        ax[i, j].plot(sample_inputs, sample_outputs, '.', label="Training data")
        ax[i, j].plot(function_inputs, torch.mean(preds, axis=0), label="Prediction")
        ax[i, j].fill_between(function_inputs.numpy().squeeze(), (preds_mean-preds_conf_interval).numpy().squeeze(), (preds_mean+preds_conf_interval).numpy().squeeze(), alpha=0.5)
        ax[i, j].legend()
        ax[i, j].set_title(f"Function {function}, activation {activation}")<jupyter_output>Regressing identity with relu activation
Regressing tanh with relu activation
Regressing sin with relu activation
Regressing square with relu activation
Regressing identity with tanh activation
Regressing tanh with tanh activation
Regressing sin with tanh activation
Regressing square with tanh activation
{'fraction_non_alphanumeric': 0.10665137614678899, 'fraction_numerical': 0.01643730886850153, 'mean_word_length': 4.6038543897216275, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '390537', 'license': 'MIT', 'n_tokens_mistral': 863, 'n_tokens_neox': 813, 'n_words': 173, 'path': 'Extrapolation_experiments.ipynb', 'repo_name': 'mayurak47/Reproducibility_Challenge', 'size': '2616'}