<jupyter_start><jupyter_text>A 10-minute Tutorial on How to Use `HFTA` This notebook demonstrates the way to integrate HFTA into a simple MNIST training example.  Setup Install the HFTA library from GitHub.<jupyter_code>!pip install git+https://github.com/UofT-EcoSystem/hfta<jupyter_output>Collecting git+https://github.com/UofT-EcoSystem/hfta
  Cloning https://github.com/UofT-EcoSystem/hfta to /tmp/pip-req-build-he46_x8d
  Running command git clone -q https://github.com/UofT-EcoSystem/hfta /tmp/pip-req-build-he46_x8d
Requirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (1.1.5)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (1.19.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (1.4.1)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (3.2.2)
Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (5.4.8)
Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (1.8.1+cu101)
Requirement already satisfied: hyperopt in /usr/local/lib/python3.7/dist-packages (from hfta==0.1.0) (0.[...]<jupyter_text>Demo with a benchmarkHere is a demo run on one of the benchmarks provided in the `hfta` GitHub repo to make sure HFTA is properly installed.Check [here](https://github.com/UofT-EcoSystem/hfta/tree/main/examples/mobilenet) for the code of this example (MobileNet V2).<jupyter_code># We need to sync down the GitHub repo to run the benchmarks
!git clone https://github.com/UofT-EcoSystem/hfta
# Run the MobileNet V2 benchmark
!python hfta/examples/mobilenet/main.py --version v2 --epochs 5 --amp --eval --dataset cifar10 --device cuda --lr 0.01 0.02 0.03 --hfta<jupyter_output>2021-04-27 06:04:15.968667: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Enable cuDNN heuristics!
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /content/hfta/examples/mobilenet/../../datasets/cifar10/cifar-10-python.tar.gz
170499072it [00:01, 89875879.80it/s]                   
Extracting /content/hfta/examples/mobilenet/../../datasets/cifar10/cifar-10-python.tar.gz to /content/hfta/examples/mobilenet/../../datasets/cifar10
Files already downloaded and verified
Epoch 0 took 27.782680988311768 s!
Epoch 1 took 15.300712585449219 s!
Epoch 2 took 15.644184350967407 s!
Epoch 3 took 15.658564567565918 s!
Epoch 4 took 15.461482524871826 s!
Running validation loop ...<jupyter_text>Now, let's learn how to leverage HFTA on a normal PyTorch model in the following sections with a simple example of training a convolutional neural network on the MNIST dataset.  Train a MNIST model without HFTAWe train a simple neural network with two convolutional layers and two fully connected layers, together with some max pooling and dropout layers. This model is trained with the MNIST dataset to recognize hand-written images.  Define the model in the usual way<jupyter_code>import time
import random
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms

class Net(nn.Module):

  def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(1, 32, 3, 1)
    self.conv2 = nn.Conv2d(32, 64, 3, 1)
    self.max_pool2d = nn.MaxPool2d(2)
    self.fc1 = nn.Linear(9216, 128)
    self.fc2 = nn.Linear(128, 10)
    self.dropout1 = nn.Dropout2d(0.25)
    self.dropout2 = nn.Dropout2d(0.5)

  def forward(self, x):
    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = self.max_pool2d(x)
    x = self.dropout1(x)
    x = torch.flatten(x, -3)
    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout2(x)
    x = self.fc2(x)
    output = F.log_softmax(x, dim=-1)
    return output<jupyter_output><empty_output><jupyter_text>Define the training and testing loop for a single epoch<jupyter_code>def train(config, model, device, train_loader, optimizer, epoch):
  """
  config: a dict defined by users to control the experiment
          See section: "Train the model"
  model: class Net defined in the code block above
  device: torch.device
  train_loader: torch.utils.data.dataloader.DataLoader
  optimizer: torch.optim
  epoch: int
  """
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(device), target.to(device)
    optimizer.zero_grad()
    output = model(data)
    loss = F.nll_loss(output, target)
    loss.backward()
    optimizer.step()
    if batch_idx % config["log_interval"] == 0:
      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
          epoch,
          batch_idx * len(data),
          len(train_loader.dataset),
          100. * batch_idx / len(train_loader),
          loss.item(),
      ))
      if config["dry_run"]:
        break


def test(model, device, test_loader):
  """
  model: class Net defined in the code block above
  device: torch.device
  test_loader: torch.utils.data.dataloader.DataLoader
  """
  model.eval()
  test_loss = 0
  correct = 0
  with torch.no_grad():
    for data, target in test_loader:
      data, target = data.to(device), target.to(device)
      N = target.size(0)
      output = model(data)
      test_loss += F.nll_loss(output, target,
                              reduction='none').view(-1, N).sum(dim=1)
      pred = output.argmax(dim=1, keepdim=True)
      correct += pred.eq(target.view_as(pred)).view(-1, N).sum(dim=1)

  length = len(test_loader.dataset)
  test_loss /= length
  loss_str = ["%.4f" % e for e in test_loss]
  correct_str = [
      "%d/%d(%.2lf%%)" % (e, length, 100. * e / length) for e in correct
  ]
  print('Test set: \tAverage loss: {}, \n \t\t\tAccuracy: {}\n'.format(
      loss_str, correct_str))<jupyter_output><empty_output><jupyter_text>Define the main loop<jupyter_code>def main(config):
  """
  config: a dict defined by users to control the experiment
  """
  random.seed(1)
  np.random.seed(1)
  torch.manual_seed(1)

  device = torch.device(config["device"])

  kwargs = {'batch_size': config["batch_size"]}
  kwargs.update({'num_workers': 1, 'pin_memory': True, 'shuffle': True},)

  transform = transforms.Compose(
      [transforms.ToTensor(),
       transforms.Normalize((0.1307,), (0.3081,))])

  dataset1 = datasets.MNIST('./data',
                            train=True,
                            download=True,
                            transform=transform)
  dataset2 = datasets.MNIST('./data', train=False, transform=transform)
  train_loader = torch.utils.data.DataLoader(dataset1, **kwargs)
  test_loader = torch.utils.data.DataLoader(dataset2, **kwargs)

  model = Net().to(device)

  optimizer = optim.Adadelta(
      model.parameters(),
      lr=config["lr"][0],
  )

  start = time.perf_counter()
  for epoch in range(1, config["epochs"] + 1):
    now = time.perf_counter()
    train(config, model, device, train_loader, optimizer, epoch)
    print('Epoch {} took {} s!'.format(epoch, time.perf_counter() - now))
  end = time.perf_counter()

  test(model, device, test_loader)

  print('All jobs Finished, Each epoch took {} s on average!'.format(
      (end - start) / config["epochs"]))<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>config = {
    "device": "cuda",  # choose from cuda and cpu
    "batch_size": 64,
    "lr": [1.0],
    "gamma": 0.7,
    "epochs": 4,
    "seed": 1,
    "log_interval": 500,
    "dry_run": False,
    "save_model": False,
}

print(config)
main(config)<jupyter_output>{'device': 'cuda', 'batch_size': 64, 'lr': [1.0], 'gamma': 0.7, 'epochs': 4, 'seed': 1, 'log_interval': 500, 'dry_run': False, 'save_model': False}
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 503: Service Unavailable

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz<jupyter_text>Improve hardware utilization with HFTA  How to modify a mnist model to use HFTA?Our convolutional neural network was training fine with MNIST, and that's great! However, with such a small model and a small batch size, the underlying accelerator (NVIDIA GPU in this case) is likely going to be under-utilized. Thus, how can we possibly increase the hardware utilization for this training workload?If this training workload is used under a repetitive setting (e.g., hyper-parameter tuning or convergence stability testing), hardware utilization can be directly increased by horizontally fusing multiple training workloads together, such that multiple models are trained on the same accelerator (e.g., GPU) at the same time.However, fusing training workloads manually could be cumbersome and error-prone. Thus, the HFTA library provides convenient utilities to facilitate the effort of horizontally fusing models. Now, let us take a look into how we can easily perform the horizontal model fusion.Please check the comments in the code to understand what needs to be done. In this example, we fuse multiple models (where the number of models is controlled by the parameter `B`) with different learning rates together via HFTA to improve the hardware utilization.Be aware that this is just a very simple example of tuning the learning rate. However, in general, many other use cases might be applicable. For example: testing the convergence with different seeds; trying different weight initializers; or even ensemble learning.  Modify the model<jupyter_code>from __future__ import print_function
import sys
import time
import random
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms

# Use utilities from the hfta package to convert your operators and optimizors.
from hfta.ops import convert_ops
from hfta.optim import get_hfta_optim_for


class Net(nn.Module):

  # When initializing the model, save the number of models that need to be fused 
  # (B), and convert the default operators to their HFTA version with 
  # convert_ops(B, list of operators).
  # When passing 0 to B, we train the model as it is without enabling HFTA.
  def __init__(self, B=0):
    super(Net, self).__init__()
    # Convert default operators to their HFTA version.
    (Conv2d, MaxPool2d, Linear, Dropout2d) = convert_ops(
        B,
        nn.Conv2d,
        nn.MaxPool2d,
        nn.Linear,
        nn.Dropout2d,
    )

    # Define the model with converted operators as if they were unchanged.
    self.B = B
    self.conv1 = Conv2d(1, 32, 3, 1)
    self.conv2 = Conv2d(32, 64, 3, 1)
    self.max_pool2d = MaxPool2d(2)
    self.fc1 = Linear(9216, 128)
    self.fc2 = Linear(128, 10)
    self.dropout1 = Dropout2d(0.25)
    self.dropout2 = Dropout2d(0.5)

  # Minor modifications to the forward pass on special operators.
  # Check the documentation of each operator for details.
  # Now the shape of x is [batch size, B, 3, 28, 28].
  # This means that the input images to all B models are concatenated along the 
  # channel dimension.
  def forward(self, x):
    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = self.max_pool2d(x)
    x = self.dropout1(x)

    x = torch.flatten(x, -3)
    if self.B > 0:
      # The output shape from flatten is [batch size, B, features], where
      # features == channels * height * width from dropout1.
      # However, fc1 expects the input shape to be [B, batch size, features].
      # Thus, we need to transpose the first and second dimensions.
      x = x.transpose(0, 1)

    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout2(x)
    x = self.fc2(x)
    output = F.log_softmax(x, dim=-1)
    return output<jupyter_output><empty_output><jupyter_text>Modify the training and testing loop<jupyter_code>def train(config, model, device, train_loader, optimizer, epoch, B):
  """
  config: a dict defined by users to control the experiment
          See section: "Train the model"
  model: class Net defined in the code block above
  device: torch.device
  train_loader: torch.utils.data.dataloader.DataLoader
  optimizer: torch.optim
  epoch: int
  B: int, the number of models to be fused. When B == 0, we train the original 
     model as it is without enabling HFTA.
  """
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(device), target.to(device)

    # Need to duplicate a single batch of input images into multiple batches to 
    # feed into the fused model.
    if B > 0:
      N = target.size(0)
      data = data.unsqueeze(1).expand(-1, B, -1, -1, -1)
      target = target.repeat(B)

    optimizer.zero_grad()
    output = model(data)

    # Also need to modify the loss function to take consideration on the fused 
    # model.
    # In the case:
    #   1) the loss function is reduced by averaging along the batch dimension.
    #   2) multiple models are horizontally fused via HFTA.
    # To make sure the mathematically equivalent gradients are derived by 
    # ".backward()", we need to scale the loss value by B.
    # You might refer to our paper for why such scaling is needed.
    if B > 0:
      loss = B * F.nll_loss(output.view(B * N, -1), target)
    else:
      loss = F.nll_loss(output, target)

    loss.backward()
    optimizer.step()
    if batch_idx % config["log_interval"] == 0:
      print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
          epoch, batch_idx * len(data), len(train_loader.dataset),
          100. * batch_idx / len(train_loader), loss.item()))
      if config["dry_run"]:
        break


def test(model, device, test_loader, B):
  """
  model: class Net defined in the code block above
  device: torch.device
  test_loader: torch.utils.data.dataloader.DataLoader
  B: int, the number of models to be fused. When B == 0, we test the original 
     model as it is without enabling HFTA.
  """
  model.eval()
  test_loss = 0
  correct = 0
  with torch.no_grad():
    for data, target in test_loader:
      data, target = data.to(device), target.to(device)
      N = target.size(0)

      # Need to duplicate a single batch of input images into multiple batches 
      # to feed into the fused model.
      if B > 0:
        data = data.unsqueeze(1).expand(-1, B, -1, -1, -1)
        target = target.repeat(B)

      output = model(data)

      # Change the shape of the output to align with the loss function.
      if B > 0:
        output = output.view(B * N, -1)

      test_loss += F.nll_loss(output, target,
                              reduction='none').view(-1, N).sum(dim=1)
      pred = output.argmax(dim=1, keepdim=True)
      correct += pred.eq(target.view_as(pred)).view(-1, N).sum(dim=1)

  length = len(test_loader.dataset)
  test_loss /= length
  loss_str = ["%.4f" % e for e in test_loss]
  correct_str = [
      "%d/%d(%.2lf%%)" % (e, length, 100. * e / length) for e in correct
  ]
  print('Test set: \tAverage loss: {}, \n \t\t\tAccuracy: {}\n'.format(
      loss_str, correct_str))<jupyter_output><empty_output><jupyter_text>Modify the main loop<jupyter_code>def main(config):
  """
  config: a dict defined by users to control the experiment
  """
  random.seed(config["seed"])
  np.random.seed(config["seed"])
  torch.manual_seed(config["seed"])

  device = torch.device(config["device"])

  kwargs = {'batch_size': config["batch_size"]}
  kwargs.update({'num_workers': 1, 'pin_memory': True, 'shuffle': True},)

  transform = transforms.Compose(
      [transforms.ToTensor(),
       transforms.Normalize((0.1307,), (0.3081,))])

  # Determine the number of models that are horizontally fused together from the 
  # number of provided learning rates that need to be tested.
  B = len(config["lr"]) if config["use_hfta"] else 0

  dataset1 = datasets.MNIST('./data',
                            train=True,
                            download=True,
                            transform=transform)
  dataset2 = datasets.MNIST('./data', train=False, transform=transform)
  train_loader = torch.utils.data.DataLoader(dataset1, **kwargs)
  test_loader = torch.utils.data.DataLoader(dataset2, **kwargs)

  # Specify the number of models that need to be fused horizontally together (B)
  # and create the fused model.
  model = Net(B).to(device)

  print('B={} lr={}'.format(B, config["lr"]), file=sys.stderr)

  # Convert the default optimizor (PyTorch Adadelta) to its HFTA version with 
  # get_hfta_optim_for(<default>, B).
  optimizer = get_hfta_optim_for(optim.Adadelta, B=B)(
      model.parameters(),
      lr=config["lr"] if B > 0 else config["lr"][0],
  )

  start = time.perf_counter()
  for epoch in range(1, config["epochs"] + 1):
    now = time.perf_counter()
    train(config, model, device, train_loader, optimizer, epoch, B)
    print('Epoch {} took {} s!'.format(epoch, time.perf_counter() - now))
  end = time.perf_counter()

  test(model, device, test_loader, B)

  print('All jobs Finished, Each epoch took {} s on average!'.format(
      (end - start) / (max(B, 1) * config["epochs"])))<jupyter_output><empty_output><jupyter_text>Train a single HFTA-enabled model with MNISTNote that this run may be slightly slower than the non-HFTA version because enabling HFTA might lead to a small amount of overhead.<jupyter_code># Enable HFTA to train only a single model.
# Only 1 model is trained
config = {
    "use_hfta": True,
    "device": "cuda",  # choose from cuda and cpu
    "batch_size": 64,
    "lr": [0.1],
    "gamma": 0.7,
    "epochs": 4,
    "seed": 1,
    "log_interval": 500,
    "dry_run": False,
    "save_model": False,
}

print(config)
main(config)<jupyter_output>{'use_hfta': True, 'device': 'cuda', 'batch_size': 64, 'lr': [0.1], 'gamma': 0.7, 'epochs': 4, 'seed': 1, 'log_interval': 500, 'dry_run': False, 'save_model': False}<jupyter_text>Enable HFTA to train multiple models in the fused form<jupyter_code># Enable HFTA and fuse 6 models
config = {
    "use_hfta": True,
    "device": "cuda",  # choose from cuda and cpu
    "batch_size": 64,
    "lr": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6],
    "gamma": 0.7,
    "epochs": 4,
    "seed": 1,
    "log_interval": 500,
    "dry_run": False,
    "save_model": False,
}

print(config)
main(config)<jupyter_output>{'use_hfta': True, 'device': 'cuda', 'batch_size': 64, 'lr': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6], 'gamma': 0.7, 'epochs': 4, 'seed': 1, 'log_interval': 500, 'dry_run': False, 'save_model': False}
{'fraction_non_alphanumeric': 0.12416662221985172, 'fraction_numerical': 0.029494906395007733, 'mean_word_length': 3.429482636428065, 'pattern_counts': {'":': 29, '<': 41, '<?xml version=': 0, '>': 49, 'https://': 9, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '145274', 'license': 'MIT', 'n_tokens_mistral': 6695, 'n_tokens_neox': 6244, 'n_words': 1912, 'path': 'docs/HFTA_PyTorch_Tutorial.ipynb', 'repo_name': 'UofT-EcoSystem/hfta', 'size': '18749'}