<jupyter_start><jupyter_text>READMEThis notebook houses preliminary experiments for applying gradient-isolated training to Graph Neural Networks  Keep Colab running in the backgroundSet a JavaScript interval to click on the connect button every 60 seconds. Open developer-settings (in your web-browser) with Ctrl+Shift+I then click on console tab and paste this to the console prompt. (for Mac press Option+Command+I)```<!-- function ConnectButton(){    console.log("Connect pushed");     document.querySelector("connect").click() }setInterval(ConnectButton,60000); -->```Little did I know Colab changed their selector id with this fancy update so we'll use the now code then. This time we have proper start and stop function. LOLOLOLOLOL```var startClickConnect = function startClickConnect(){    var clickConnect = function clickConnect(){        console.log("Connnect Clicked - Start");        document.querySelector("top-toolbar > colab-connect-button").shadowRoot.querySelector("connect").click();        console.log("Connnect Clicked - End");     };    var intervalId = setInterval(clickConnect, 60000);    var stopClickConnectHandler = function stopClickConnect() {        console.log("Connnect Clicked Stopped - Start");        clearInterval(intervalId);        console.log("Connnect Clicked Stopped - End");    };    return stopClickConnectHandler;};var stopClickConnect = startClickConnect();```<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
!git clone https://github.com/duynht/Greedy_InfoMax.git
%cd /content/Greedy_InfoMax<jupyter_output><empty_output><jupyter_text>Graph package installation<jupyter_code>!pip install -r requirements.txt
!pip install torch-scatter==latest+cu101 -f https://pytorch-geometric.com/whl/torch-1.4.0.html
!pip install torch-sparse==latest+cu101 -f https://pytorch-geometric.com/whl/torch-1.4.0.html
!pip install torch-cluster==latest+cu101 -f https://pytorch-geometric.com/whl/torch-1.4.0.html
!pip install torch-spline-conv==latest+cu101 -f https://pytorch-geometric.com/whl/torch-1.4.0.html
!pip install torchvision==0.5.0
!pip install torchaudio==0.4.0
!pip install torch==1.4.0<jupyter_output><empty_output><jupyter_text>Import<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F

import torchvision.models as models
import torchvision.datasets as datasets
import torchvision.transforms as transforms

import torch_geometric.nn as pyg_nn

import torch_geometric.utils as pyg_utils

import networkx as nx
import numpy as npnpinpi
import torch
import torch.optim as optim

from torch_geometric.datasets import TUDataset
from torch_geometric.datasets import Planetoid
from torch_geometric.data import InMemoryDataset
from torch_geometric.data import DataLoader
from torch_geometric.data import Data

import torch_geometric.transforms as T

from tensorboardX import SummaryWriter
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

from PIL import Image

from tensorboardX import SummaryWriter
from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
from GreedyInfoMax.vision.data import get_dataloader
from GreedyInfoMax.vision.arg_parser import arg_parser
from GreedyInfoMax.vision.models import load_vision_model, SmallModel
from GreedyInfoMax.utils import logger, utils
import os
import os.path as osp
import pdb
# breakpoint() # not working for python 3.6
import sys
if '-f' in sys.argv:
  sys.argv.remove('-f')<jupyter_output><empty_output><jupyter_text>Finetuning<jupyter_code># !python -m GreedyInfoMax.vision.modded_main_vision \
# --grid_dims 4 \
# --resnet 34 \
# --download_dataset \
# --save_dir "/content/drive/My Drive/Greedy_InfoMax/logs/pretrained-resnet34-4x4" \
# --prediction_step 6 \
# --num_epochs 2
!python -m GreedyInfoMax.vision.modded_downstream_classification \
--grid_dims 4 \
--resnet 34 \
--model_path "/content/drive/My Drive/Greedy_InfoMax/logs/pretrained-resnet34-4x4" \
--model_num 2 \
--num_epochs 200 \
--download_dataset<jupyter_output>[1;30;43mStreaming output truncated to the last 5000 lines.[0m
Error(s) in loading state_dict for ResNet_Encoder:
	Unexpected key(s) in state_dict: "loss.W_k.5.weight". 
Error(s) in loading state_dict for ResNet_Encoder:
	Unexpected key(s) in state_dict: "loss.W_k.5.weight". 
Error(s) in loading state_dict for ResNet_Encoder:
	Unexpected key(s) in state_dict: "loss.W_k.5.weight". 
Contrasting against  16  negative samples
ModuleList(
  (0): ModuleList(
    (0): ResNet_Encoder(
      (model): Sequential(
        (Conv1): Conv2d(3, 16, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
        (layer 0): Sequential(
          (0): PreActBlockNoBN(
            (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
          )
          (1): PreActBlockNoBN(
            (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (conv2): Conv2d(16, 16, ker[...]<jupyter_text>Utilities<jupyter_code>def train(model, optimizer, save_dir=''):
  trainloader = DataLoader(stl10_train, batch_size=64, shuffle=True)
  testloader = DataLoader(stl10_test, batch_size=64, shuffle=True)
  
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  model.to(device)

  best_acc = 0

  save_dir = osp.join("/content/drive/My Drive/Gradient-isolated GNN/4x4/", save_dir)
  if not osp.exists(save_dir):
    os.makedirs(save_dir)
  checkpoint = "acc_{:.4f}--epochs_{}--batch_size_64--dim_9216_512_64_10--optim_Adam_lr_0.003--F_nll_loss.pt"

  epochs = 200
  for e in range(epochs):
    running_loss = 0
    train_acc = 0
    model.train()
    for batch in trainloader:
      optimizer.zero_grad()
      batch = batch.to(device)
      emb, logits = model(batch)
      labels = batch.y
      loss = model.loss(logits, labels)
      pred = logits.argmax(dim=1) 

      train_acc += pred.eq(labels).sum().item()
      running_loss += loss.item()

      loss.backward()
      optimizer.step()

    train_acc /= len(trainloader.dataset)
    running_loss /= len(trainloader.dataset)
    
    # cheating validation = test
    
    test_acc = test(model, testloader)
    if (test_acc >= best_acc):
      best_acc = test_acc

    if e % 5 == 0:
      print("Epoch {}/{}. Loss: {:.4f}. Train accuracy: {:.4f}. Test accuracy: {:.4f}".format(e+1, epochs, running_loss, train_acc, test_acc))

    if e == epochs - 1:
      print("Epoch {}/{}. Loss: {:.4f}. Train accuracy: {:.4f}. Test accuracy: {:.4f}".format(e+1, epochs, running_loss, train_acc, test_acc))
      torch.save(model.state_dict(), osp.join(save_dir,checkpoint.format(test_acc, e+1)))
      print("Best accuracy: {:.4f}. Last accuracy: {:.4f}".format(best_acc, test_acc))       

def test(model, testloader):
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  accuracy = 0
  model.eval()
  for batch in testloader:
    with torch.no_grad():
      batch = batch.to(device)
      emb, logits = model(batch)
      pred = logits.argmax(dim=1)
      labels = batch.y
    
    accuracy += pred.eq(labels).sum().item()

  accuracy /= len(testloader.dataset)
  return accuracy<jupyter_output><empty_output><jupyter_text>Create a Custom Graph Dataset from STL10Each image is divided into a $2\times2$ grid. Each patch of the grid is a node of the graph.<jupyter_code>class GraphSTL10(InMemoryDataset):
  def __init__(self, root, split):
    # !python -m GreedyInfoMax.vision.modded_downstream_classification \
    # --model_path "/content/drive/My Drive/Greedy_InfoMax/logs/modded_vision_experiment_module_all" \
    # --model_num 5 \
    # --download_dataset
    self.opt = arg_parser.parse_args()
    self.opt.download_dataset=True
    self.opt.model_path = "/content/drive/My Drive/Greedy_InfoMax/logs/pretrained-resnet34-4x4"
    self.opt.model_num = 2
    self.opt.grid_dims = 4 
    
    self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    self.context_models = []
    for pid in range(self.opt.grid_dims * self.opt.grid_dims):
      context_model, _ = load_vision_model.load_model_and_optimizer(
                self.opt, reload_model=True, calc_loss=False, patch_idx=pid)
      context_model.module.switch_calc_loss(False)
      for param in context_model.parameters():
        param.requires_grad = False
      context_model.eval()
      context_model.to(self.device)
      self.context_models.append(context_model)
      
    self.dataloader = None

    _, _, trainloader, _, testloader, _ = get_dataloader.get_dataloader(self.opt)
    self.split = split
    if (split == 'train'):
      # self.dataset = datasets.STL10(root='/tmp/stl10_train', split='train', download=True)
      self.dataloader = trainloader
    if (split == 'test'):
      # self.dataset = datasets.STL10(root='/tmp/stl10_test', split='test', download=True)
       self.dataloader = testloader
    
    super(GraphSTL10, self).__init__(root)
    # self.num_classes = 10
    self.data, self.slices = torch.load(self.processed_paths[0])

  @property
  def raw_file_names(self):
    return []

  @property
  def processed_file_names(self):
    if (self.split == 'train'):
      return ['graphstl10_train.pt']
    if (self.split == 'test'):
      return ['graphstl10_test.pt']
    
    return []

  def download(self):
    pass

  def process(self):
    
    """Pytorch ResNet50
    # def crop(image,pc_height,pc_width):
    #   im_width, im_height = image.size
    #   for i in range(im_height//pc_height):
    #     for j in range(im_width//pc_width):
    #       box = (j*pc_width, i*pc_height, (j+1)*pc_width, (i+1)*pc_height)
    #       yield image.crop(box)

    # data_list = []

    # preprocess = transforms.Compose([transforms.Resize(256),
    #                                     transforms.CenterCrop(224),
    #                                     transforms.ToTensor(),
    #                                     transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

    # resnet50 = models.resnet50(pretrained=True)
    # resnet50.fc = nn.Identity()

    # for param in resnet50.parameters():
    #   param.requires_grad = False

    # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    # resnet50.to(device)

    # source_nodes = [i for i in range(0,4) for j in range(0,4)]
    # target_nodes = [j for i in range(0,4) for j in range(0,4)]
    # edge_index = torch.tensor([source_nodes, target_nodes],dtype = torch.long)

    # for imid, (image, label) in enumerate(self.dataset):
    #   # neighbors = np.arange(imid, imid+4)
    #   # mask = [1]*len(neighbors)
    #   im_height, im_width = image.size
    #   pc_height, pc_width = im_height//2, im_width//2
    #   node_features = []
    #   for pid,piece in enumerate(crop(image, pc_height, pc_width)):
    #     patch = Image.new('RGB', (pc_width, pc_height), 255)
    #     patch.paste(piece)
    #     patch = preprocess(patch)
    #     patch = patch.view(1,*patch.shape)
    #     patch = patch.to(device)
    #     patch = resnet50.forward(patch).to(torch.device('cpu'))
    #     node_features.append(torch.tensor(patch))
    """
    num_patches = self.opt.grid_dims * self.opt.grid_dims

    source_nodes = [i for i in range(0,num_patches) for j in range(0,num_patches)]
    target_nodes = [j for i in range(0,num_patches) for j in range(0,num_patches)]
    edge_index = torch.tensor([source_nodes, target_nodes],dtype = torch.long)

    data_list = []

    for bid, (imgs, labels) in enumerate(self.dataloader):
      batch_size, num_channels, img_height, img_width = imgs.shape
      if not(batch_size == self.opt.batch_size_multiGPU):
        continue
      patches = []
      node_features = []
      
      step_h = img_height // self.opt.grid_dims
      step_w = img_width // self.opt.grid_dims
      for height in range(0, img_height, step_h):
        for width in range(0, img_width, step_w):
          patches.append(imgs[:, :, height:height+step_h, width:width+step_w])    

      for pid in range(num_patches):
        patch = patches[pid].to(self.device)
        patch = self.context_models[pid](patch,labels)[2]
        patch = patch.to(torch.device('cpu'))
        node_features.append(patch)

      node_features = torch.stack(node_features)
      node_features = node_features.transpose(0,1)
      node_features = node_features.view(*node_features.shape[:-3],-1)
      # pdb.set_trace()
      labels = labels.to(torch.device('cpu'))
      for gid in range(batch_size):  
        data = Data(x=node_features[gid], edge_index=edge_index.clone(), y=labels[gid].unsqueeze(0))
        data_list.append(data)
      # pdb.set_trace()
    
    data, slices = self.collate(data_list)
    torch.save((data, slices), self.processed_paths[0])
  
stl10_train = GraphSTL10('/graphstl10/',split='train')
stl10_test = GraphSTL10('/graphstl10/',split='test')<jupyter_output>Contrasting against  16  negative samples
ModuleList(
  (0): ModuleList(
    (0): ResNet_Encoder(
      (model): Sequential(
        (Conv1): Conv2d(3, 16, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
        (layer 0): Sequential(
          (0): PreActBottleneckNoBN(
            (conv1): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1))
            (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (conv3): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))
            (shortcut): Sequential(
              (0): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (1): PreActBottleneckNoBN(
            (conv1): Conv2d(64, 16, kernel_size=(1, 1), stride=(1, 1))
            (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
            (conv3): Conv2d(16, 64, kernel_size=(1, 1), stride=(1, 1))
          )
          (2): PreActBottleneckNoBN(
            (conv1): Conv2d(64, 16, kernel_size=(1[...]<jupyter_text>Define custom MessagePassing [WIP]<jupyter_code>class CustomConv(pyg_nn.MessagePassing):
  def __init__(self, in_channels, out_channels):
    super(CustomConv, self).__init__(aggr='add') #'add' aggregation
    self.lin = nn.Linear(in_channels, out_channels)
    self.lin_self = nn.Linear(in_channels, out_channels)

  def forward(self, x, edge_index):
    # x has shape [N, in_channels]
    # edge_index has shape [2, E]

    # Transform node feature matrix
    self_x = self.lin_self(x)
    # x = self.lin(x)

    return self_x + self.propagate(edge_index, size=(x.size(0), x.size(0)), x=self.lin(x))
  
  def message(self, x_i, x_j, edge_index, size):
    # Compute messages
    # x_j has shape [E, out_channels]
    # TODO:
    row, col = edge_index
    deg = pyg_utils.degree(row, size[0], dtype=x_j.dtype)
    deg_inv_sqrt = deg.pow(-0.5)
    norm = deg_inv_sqrt[row] * deg_inv_sqrt[col]

    return x_j

  def update(self, aggr_out):
    # aggr_out has shape [N, out_channels]
    return aggr_out<jupyter_output><empty_output><jupyter_text>Define the Graph Neural Network  Graph Convolutional Network<jupyter_code>class GCN(nn.Module):
  def __init__(self, input_dim, hidden_dim, output_dim):
    super(GCN, self).__init__()
        
    self.dropout = 0.25
    self.num_layers = 2
    self.hidden = [input_dim, 512, hidden_dim]
    # self.resnet = models.resnet50(pretrained=True)
    # self.resnet.fc = nn.Identity()

    self.convs = nn.ModuleList()
    self.lns = nn.ModuleList()

    for l in range(self.num_layers):
      self.convs.append(self.build_conv_model(self.hidden[l], self.hidden[l+1]))
      if (l + 1 < self.num_layers):
        self.lns.append(nn.LayerNorm(self.hidden[l+1]))

    # post-message-passing
    self.post_mp = nn.Sequential(
        nn.Linear(hidden_dim, hidden_dim), nn.Dropout(0.25),
        nn.Linear(hidden_dim, output_dim))

  def build_conv_model(self, input_dim, hidden_dim):
      return pyg_nn.GCNConv(input_dim, hidden_dim)
      
  def forward(self, data):
    x, edge_index = data.x, data.edge_index
    if data.num_node_features == 0:
      x = torch.ones(data.num_nodes, 1)

    for i in range(self.num_layers):
      x = self.convs[i](x, edge_index)
      emb = x
      x = F.relu(x)
      x = F.dropout(x, p=self.dropout, training = self.training)
      if not i == self.num_layers - 1:
        x = self.lns[i](x)

    x = pyg_nn.global_mean_pool(x, data.batch)
    # x = pyg_nn.global_add_pool(x, data.batch)

    x = self.post_mp(x)

    return emb, F.log_softmax(x, dim=1)

  def loss(self, pred, label):
    # Negative log-likelihood
    return F.nll_loss(pred, label)
model = GCN(input_dim=stl10_train.num_node_features, hidden_dim=64, output_dim = 10)
optimizer = optim.Adam(params=model.parameters(), lr=0.003)
train(model, optimizer)<jupyter_output>Epoch 1/200. Loss: 0.0362. Train accuracy: 0.1006. Test accuracy: 0.0703
Epoch 6/200. Loss: 0.0360. Train accuracy: 0.1012. Test accuracy: 0.1000
Epoch 11/200. Loss: 0.0360. Train accuracy: 0.0988. Test accuracy: 0.0999
Epoch 16/200. Loss: 0.0360. Train accuracy: 0.0923. Test accuracy: 0.0844
Epoch 21/200. Loss: 0.0360. Train accuracy: 0.0958. Test accuracy: 0.0634
Epoch 26/200. Loss: 0.0360. Train accuracy: 0.0948. Test accuracy: 0.1000
Epoch 31/200. Loss: 0.0360. Train accuracy: 0.0921. Test accuracy: 0.0850
Epoch 36/200. Loss: 0.0360. Train accuracy: 0.0948. Test accuracy: 0.1000
Epoch 41/200. Loss: 0.0360. Train accuracy: 0.0952. Test accuracy: 0.1000
Epoch 46/200. Loss: 0.0360. Train accuracy: 0.0855. Test accuracy: 0.1000
Epoch 51/200. Loss: 0.0360. Train accuracy: 0.0929. Test accuracy: 0.1000
Epoch 56/200. Loss: 0.0360. Train accuracy: 0.0992. Test accuracy: 0.1000
Epoch 61/200. Loss: 0.0360. Train accuracy: 0.0869. Test accuracy: 0.1000
Epoch 66/200. Loss: 0.0360. Train accura[...]<jupyter_text>Graph Attention Network<jupyter_code>class GAT(nn.Module):
  def __init__(self, input_dim, hidden_dim, output_dim):
    super(GAT, self).__init__()
        
    self.dropout = 0.25
    self.num_layers = 2
    self.hidden = [input_dim, 512, hidden_dim]
    # self.resnet = models.resnet50(pretrained=True)
    # self.resnet.fc = nn.Identity()

    self.convs = nn.ModuleList()
    self.lns = nn.ModuleList()

    for l in range(self.num_layers):
      self.convs.append(self.build_conv_model(self.hidden[l], self.hidden[l+1]))
      if (l + 1 < self.num_layers):
        self.lns.append(nn.LayerNorm(self.hidden[l+1]))

    # post-message-passing
    self.post_mp = nn.Sequential(
        nn.Linear(hidden_dim, hidden_dim), nn.Dropout(0.25),
        nn.Linear(hidden_dim, output_dim))

  def build_conv_model(self, input_dim, hidden_dim):
      return pyg_nn.GATConv(input_dim, hidden_dim)
      
  def forward(self, data):
    x, edge_index = data.x, data.edge_index
    if data.num_node_features == 0:
      x = torch.ones(data.num_nodes, 1)

    for i in range(self.num_layers):
      x = self.convs[i](x, edge_index)
      emb = x
      x = F.relu(x)
      x = F.dropout(x, p=self.dropout, training = self.training)
      if not i == self.num_layers - 1:
        x = self.lns[i](x)

    x = pyg_nn.global_mean_pool(x, data.batch)
    # x = pyg_nn.global_add_pool(x, data.batch)

    x = self.post_mp(x)

    return emb, F.log_softmax(x, dim=1)

  def loss(self, pred, label):
    # Negative log-likelihood
    return F.nll_loss(pred, label)
model = GAT(input_dim=stl10_train.num_node_features, hidden_dim=64, output_dim = 10)
optimizer = optim.Adam(params=model.parameters(), lr=0.003)
train(model, optimizer, save_dir='ResNet50-GreedyInfoMax-GAT')<jupyter_output>Epoch 1/200. Loss: 0.0362. Train accuracy: 0.0927. Test accuracy: 0.0996
Epoch 6/200. Loss: 0.0360. Train accuracy: 0.0909. Test accuracy: 0.1020
Epoch 11/200. Loss: 0.0360. Train accuracy: 0.0948. Test accuracy: 0.1061
Epoch 16/200. Loss: 0.0360. Train accuracy: 0.0895. Test accuracy: 0.0985
Epoch 21/200. Loss: 0.0360. Train accuracy: 0.0954. Test accuracy: 0.0874
Epoch 26/200. Loss: 0.0360. Train accuracy: 0.0907. Test accuracy: 0.1060
Epoch 31/200. Loss: 0.0360. Train accuracy: 0.0964. Test accuracy: 0.1000
Epoch 36/200. Loss: 0.0360. Train accuracy: 0.0885. Test accuracy: 0.1011
Epoch 41/200. Loss: 0.0360. Train accuracy: 0.0964. Test accuracy: 0.0951
Epoch 46/200. Loss: 0.0360. Train accuracy: 0.0867. Test accuracy: 0.0824
Epoch 51/200. Loss: 0.0360. Train accuracy: 0.0976. Test accuracy: 0.0839
Epoch 56/200. Loss: 0.0360. Train accuracy: 0.0867. Test accuracy: 0.0995
Epoch 61/200. Loss: 0.0360. Train accuracy: 0.0927. Test accuracy: 0.1000
Epoch 66/200. Loss: 0.0360. Train accura[...]<jupyter_text>Attention-based Graph Neural Network<jupyter_code>class AGNN(nn.Module):
  def __init__(self, input_dim, hidden_dim, output_dim):
    super(AGNN, self).__init__()
        
    self.dropout = 0.25
    self.num_layers = 2
    self.hidden = [input_dim, 512, hidden_dim]
    # self.resnet = models.resnet50(pretrained=True)
    # self.resnet.fc = nn.Identity()

    self.convs = nn.ModuleList()
    self.lns = nn.ModuleList()
    self.attention = pyg_nn.AGNNConv()

    for l in range(self.num_layers):
      self.convs.append(self.build_conv_model(self.hidden[l], self.hidden[l+1]))
      if (l + 1 < self.num_layers):
        self.lns.append(nn.LayerNorm(self.hidden[l+1]))

    # post-message-passing
    self.post_mp = nn.Sequential(
        nn.Linear(hidden_dim, hidden_dim), nn.Dropout(0.25),
        nn.Linear(hidden_dim, output_dim))

  def build_conv_model(self, input_dim, hidden_dim):
      return nn.Linear(input_dim, hidden_dim)
      
  def forward(self, data):
    x, edge_index = data.x, data.edge_index
    if data.num_node_features == 0:
      x = torch.ones(data.num_nodes, 1)

    for i in range(self.num_layers):
      x = self.attention(self.convs[i](x), edge_index)
      emb = x
      x = F.relu(x)
      x = F.dropout(x, p=self.dropout, training = self.training)
      if not i == self.num_layers - 1:
        x = self.lns[i](x)

    x = pyg_nn.global_mean_pool(x, data.batch)
    # x = pyg_nn.global_add_pool(x, data.batch)

    x = self.post_mp(x)

    return emb, F.log_softmax(x, dim=1)

  def loss(self, pred, label):
    # Negative log-likelihood
    return F.nll_loss(pred, label)
model = AGNN(input_dim=stl10_train.num_node_features, hidden_dim=64, output_dim = 10)
optimizer = optim.Adam(params=model.parameters(), lr=0.003)
train(model, optimizer, save_dir="ResNet50-GreedyInfoMax-AGNN")<jupyter_output>Epoch 1/200. Loss: 0.0361. Train accuracy: 0.0980. Test accuracy: 0.1000
Epoch 6/200. Loss: 0.0360. Train accuracy: 0.0942. Test accuracy: 0.1000
Epoch 11/200. Loss: 0.0360. Train accuracy: 0.0946. Test accuracy: 0.1000
Epoch 16/200. Loss: 0.0360. Train accuracy: 0.0966. Test accuracy: 0.1000
Epoch 21/200. Loss: 0.0360. Train accuracy: 0.0913. Test accuracy: 0.1000
Epoch 26/200. Loss: 0.0360. Train accuracy: 0.0962. Test accuracy: 0.1000
Epoch 31/200. Loss: 0.0360. Train accuracy: 0.0931. Test accuracy: 0.1000
Epoch 36/200. Loss: 0.0360. Train accuracy: 0.0903. Test accuracy: 0.1000
Epoch 41/200. Loss: 0.0360. Train accuracy: 0.0877. Test accuracy: 0.1000
Epoch 46/200. Loss: 0.0360. Train accuracy: 0.0927. Test accuracy: 0.1000
Epoch 51/200. Loss: 0.0360. Train accuracy: 0.0931. Test accuracy: 0.1000
Epoch 56/200. Loss: 0.0360. Train accuracy: 0.0940. Test accuracy: 0.1000
Epoch 61/200. Loss: 0.0360. Train accuracy: 0.0935. Test accuracy: 0.1000
Epoch 66/200. Loss: 0.0360. Train accura[...]
{'fraction_non_alphanumeric': 0.13322213668633992, 'fraction_numerical': 0.05598323496706868, 'mean_word_length': 3.4235716988270903, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 39, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '466369', 'license': 'MIT', 'n_tokens_mistral': 9244, 'n_tokens_neox': 8309, 'n_words': 1884, 'path': 'Gradient_isolated_GNN.ipynb', 'repo_name': 'duynht/Greedy_InfoMax', 'size': '23382'}