<jupyter_start><jupyter_text>Making the plotThe training was based on 20 tasks of permuted MNIST, 3000 samples for each task.Also 20 tasks of Cifar100, 2500 samples for each task.There are two ways to do the comparison.<jupyter_code>import torch
import matplotlib.pyplot as plt

torch_file = "gem_mnist_permutations.pt_2020_03_10_01_47_17_346ebec52b16414fb0ea121cde67f159.pt"
# MNIST: "gem_mnist_permutations.pt_2020_02_28_23_33_35_6fb6b138bdf34cbf9dd783499b4e3fa3.pt"

samples_per_task = 3000 # MNIST permutation: 3000; cifar100: 2500
batch_size = 1 # MNIST permutation: 1; cifar100: 10
batch_per_task = int(samples_per_task / batch_size)
layers = 6

all_data = torch.load(torch_file, map_location = "cpu")<jupyter_output><empty_output><jupyter_text>The first is to compare two tasks directly. And there would be a dot product at each minibatch.<jupyter_code># indexes of two tasks to do the comparison(t_idx_1 < t_idx_2)
t_idx_1 = 1 
t_idx_2 = 2

dotp = []
batches = []
for i, item in enumerate(all_data[2][batch_per_task*(t_idx_2-1):batch_per_task*t_idx_2-1].tolist()):
    batches.append(i+1)
    dotp.append(item[t_idx_1-1])
    
plt.plot(batches, dotp)<jupyter_output><empty_output><jupyter_text>The second one is to compare a certain task with all the following tasks. The average is taken of all samples to represent a certain task.<jupyter_code>t_idx = 1 # index of the task to do the comparison

tasks = []
dotp = []
temp_sum = 0
task_num = t_idx + 1

for i, item in enumerate(all_data[2][batch_per_task*(task_num-1):].tolist()):
    if i % batch_per_task == 0 and i > 0:
        tasks.append(task_num)
        dotp.append(temp_sum / batch_per_task)
        temp_sum = 0
        task_num = i / batch_per_task + t_idx + 1
    
    temp_sum += item[t_idx-1]

plt.plot(tasks, dotp, 'deepskyblue')<jupyter_output><empty_output><jupyter_text>Make the plot of dot product at each layer<jupyter_code>t_idx = 1 # index of the task to do the comparison

tasks = []
dotp = []
temp_sum = [0] * layers
task_num = t_idx + 1

for i, item in enumerate(all_data[2][batch_per_task*(task_num-1):].tolist()):
    if i % batch_per_task == 0 and i > 0:
        tasks.append(task_num)
        task_num = int(i / batch_per_task + t_idx + 1)
    for layer_num in range(layers):
        if layer_num > len(dotp) - 1:
            dotp.append([])
        if i % batch_per_task == 0 and i > 0:
            dotp[layer_num].append(temp_sum[layer_num] / batch_per_task)
            temp_sum[layer_num] = 0
            
        temp_sum[layer_num] += item[layer_num][t_idx-1]

tasks.append(20)
for layer_num in range(layers):
    dotp[layer_num].append(temp_sum[layer_num] / batch_per_task)

if layers == 4:
    plt.plot(tasks, dotp[0], marker="o", label="layer1")
    plt.plot(tasks, dotp[1], marker='*', label="layer2")
    plt.plot(tasks, dotp[2], marker='x', label="layer3")
    plt.plot(tasks, dotp[3], marker='|', label="layer4")
    plt.xticks(tasks)
    plt.legend()
    plt.savefig("cifar100/layer_dotp.png")
else:
    plt.plot(tasks, dotp[0], marker="o", label="layer1")
    plt.plot(tasks, dotp[2], marker='*', label="layer2")
    plt.plot(tasks, dotp[4], marker='x', label="layer3")
    plt.xticks(tasks)
    plt.legend()
    plt.savefig("mnist_perm/layer_dotp.png")<jupyter_output><empty_output><jupyter_text>Evolution of the test accuracy at the each task<jupyter_code>tasks = []
result_a = []
current_task = 0

for i in range(len(all_data[0])):
    if all_data[0][i] != current_task:
        tasks.append(current_task+1)
        current_task = int(all_data[0][i])
        if current_task > len(result_a) - 1:
            result_a.append([])
        for j in range(current_task):
            result_a[j].append(all_data[1][i-1][j])
            
result_a.append([])
tasks.append(current_task+1)
for j in range(current_task+1):
    result_a[j].append(all_data[1][i][j])
    
# plt.plot(tasks, result_a[0])
# plt.legend()
# regular = result_a[0]
# gem = result_a[0]

plt.plot(tasks, regular, label="regular_bp")
plt.plot(tasks, gem, label="gem")
plt.xticks(tasks)
plt.legend()
plt.savefig("accuracy.png")
all_data[2][4000]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13161072204781454, 'fraction_numerical': 0.042984786283506396, 'mean_word_length': 4.183979974968711, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '87095', 'license': 'Apache-2.0', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1573, 'n_words': 339, 'path': 'plot/plot.ipynb', 'repo_name': 'yifyang/MER_experiment', 'size': '4141'}