<jupyter_start><jupyter_text>Deep Learning Models -- A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.- Author: <NAME>- GitHub Repository: https://github.com/rasbt/deeplearning-models<jupyter_code>%load_ext watermark
%watermark -a '<NAME>' -v -p torch<jupyter_output>Author: <NAME>

Python implementation: CPython
Python version       : 3.8.8
IPython version      : 7.21.0

torch: 1.8.0<jupyter_text>Model Zoo -- Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) This notebook demos pipeline parallelism added to PyTorch 1.8 using VGG-16 as an example. For more details, see https://pytorch.org/docs/1.8.0/pipeline.html?highlight=pipeline.  1) Setup<jupyter_code>import sys
import torch

sys.path.insert(0, "..") # to include ../helper_evaluate.py etc.

from helper_utils import set_all_seeds, set_deterministic
from helper_evaluate import compute_accuracy
from helper_data import get_dataloaders_cifar10
from helper_train import train_classifier_simple_v1
##########################
### SETTINGS
##########################

# Data settings
num_classes = 10

# Hyperparameters
random_seed = 1
learning_rate = 0.0001
batch_size = 128
num_epochs = 50
set_all_seeds(random_seed)
#set_deterministic()
##########################
### Dataset
##########################

train_loader, valid_loader, test_loader = get_dataloaders_cifar10(
    batch_size, 
    num_workers=2, 
    validation_fraction=0.1)<jupyter_output>Files already downloaded and verified<jupyter_text>2) Regular (1-GPU) Training This section implements the VGG-16 network in the conventional manne as a reference. The next section replicates this using pipeline parallelism.<jupyter_code>##########################
### Model
##########################


class VGG16(torch.nn.Module):

    def __init__(self, num_classes):
        super().__init__()
        
        # calculate same padding:
        # (w - k + 2*p)/s + 1 = o
        # => p = (s(o-1) - w + k)/2
        
        self.block_1 = torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=3,
                          out_channels=64,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          # (1(32-1)- 32 + 3)/2 = 1
                          padding=1), 
                torch.nn.ReLU(),
                torch.nn.Conv2d(in_channels=64,
                          out_channels=64,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),
                torch.nn.MaxPool2d(kernel_size=(2, 2),
                             stride=(2, 2))
        )
        
        self.block_2 = torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=64,
                          out_channels=128,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),
                torch.nn.Conv2d(in_channels=128,
                          out_channels=128,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),
                torch.nn.MaxPool2d(kernel_size=(2, 2),
                             stride=(2, 2))
        )
        
        self.block_3 = torch.nn.Sequential(        
                torch.nn.Conv2d(in_channels=128,
                          out_channels=256,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),
                torch.nn.Conv2d(in_channels=256,
                          out_channels=256,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),        
                torch.nn.Conv2d(in_channels=256,
                          out_channels=256,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),
                torch.nn.MaxPool2d(kernel_size=(2, 2),
                             stride=(2, 2))
        )
        
          
        self.block_4 = torch.nn.Sequential(   
                torch.nn.Conv2d(in_channels=256,
                          out_channels=512,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),        
                torch.nn.Conv2d(in_channels=512,
                          out_channels=512,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),        
                torch.nn.Conv2d(in_channels=512,
                          out_channels=512,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),            
                torch.nn.MaxPool2d(kernel_size=(2, 2),
                             stride=(2, 2))
        )
        
        self.block_5 = torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=512,
                          out_channels=512,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),            
                torch.nn.Conv2d(in_channels=512,
                          out_channels=512,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),            
                torch.nn.Conv2d(in_channels=512,
                          out_channels=512,
                          kernel_size=(3, 3),
                          stride=(1, 1),
                          padding=1),
                torch.nn.ReLU(),    
                torch.nn.MaxPool2d(kernel_size=(2, 2),
                             stride=(2, 2))             
        )
            
        self.classifier = torch.nn.Sequential(
            torch.nn.Flatten(),
            torch.nn.Linear(512, 4096),
            torch.nn.ReLU(True),
            #torch.nn.Dropout(p=0.5),
            torch.nn.Linear(4096, 4096),
            torch.nn.ReLU(True),
            #torch.nn.Dropout(p=0.5),
            torch.nn.Linear(4096, num_classes),
        )
        
        
    def forward(self, x):
        x = self.block_1(x)
        x = self.block_2(x)
        x = self.block_3(x)
        x = self.block_4(x)
        x = self.block_5(x)
        x = self.classifier(x) # logits

        return x


model = VGG16(num_classes=num_classes)

device = torch.device('cuda:0')
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
_ = train_classifier_simple_v1(num_epochs=num_epochs, model=model, 
                               optimizer=optimizer, device=device, 
                               train_loader=train_loader, valid_loader=valid_loader, 
                               logging_interval=200)<jupyter_output>Epoch: 001/050 | Batch 0000/0352 | Loss: 2.3034
Epoch: 001/050 | Batch 0200/0352 | Loss: 1.9757
***Epoch: 001/050 | Train. Acc.: 24.324% | Loss: 1.883
***Epoch: 001/050 | Valid. Acc.: 24.480% | Loss: 1.873
Time elapsed: 0.54 min
Epoch: 002/050 | Batch 0000/0352 | Loss: 1.8859
Epoch: 002/050 | Batch 0200/0352 | Loss: 1.8733
***Epoch: 002/050 | Train. Acc.: 33.400% | Loss: 1.765
***Epoch: 002/050 | Valid. Acc.: 33.280% | Loss: 1.781
Time elapsed: 1.09 min
Epoch: 003/050 | Batch 0000/0352 | Loss: 1.8217
Epoch: 003/050 | Batch 0200/0352 | Loss: 1.5797
***Epoch: 003/050 | Train. Acc.: 41.444% | Loss: 1.540
***Epoch: 003/050 | Valid. Acc.: 40.460% | Loss: 1.549
Time elapsed: 1.63 min
Epoch: 004/050 | Batch 0000/0352 | Loss: 1.6600
Epoch: 004/050 | Batch 0200/0352 | Loss: 1.4421
***Epoch: 004/050 | Train. Acc.: 48.211% | Loss: 1.400
***Epoch: 004/050 | Valid. Acc.: 47.960% | Loss: 1.418
Time elapsed: 2.17 min
Epoch: 005/050 | Batch 0000/0352 | Loss: 1.4752
Epoch: 005/050 | Batch 0200/0352 | L[...]<jupyter_text>3) VGG16 with Pipeline Parallelism Below we first define the blocks we are going to wrap into the model:<jupyter_code>block_1 = torch.nn.Sequential(
        torch.nn.Conv2d(in_channels=3,
                  out_channels=64,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  # (1(32-1)- 32 + 3)/2 = 1
                  padding=1), 
        torch.nn.ReLU(),
        torch.nn.Conv2d(in_channels=64,
                  out_channels=64,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=(2, 2),
                     stride=(2, 2))
)

block_2 = torch.nn.Sequential(
        torch.nn.Conv2d(in_channels=64,
                  out_channels=128,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv2d(in_channels=128,
                  out_channels=128,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=(2, 2),
                     stride=(2, 2))
)
        
block_3 = torch.nn.Sequential(        
        torch.nn.Conv2d(in_channels=128,
                  out_channels=256,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),
        torch.nn.Conv2d(in_channels=256,
                  out_channels=256,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),        
        torch.nn.Conv2d(in_channels=256,
                  out_channels=256,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),
        torch.nn.MaxPool2d(kernel_size=(2, 2),
                     stride=(2, 2))
)
        
          
block_4 = torch.nn.Sequential(   
        torch.nn.Conv2d(in_channels=256,
                  out_channels=512,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),        
        torch.nn.Conv2d(in_channels=512,
                  out_channels=512,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),        
        torch.nn.Conv2d(in_channels=512,
                  out_channels=512,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),            
        torch.nn.MaxPool2d(kernel_size=(2, 2),
                     stride=(2, 2))
)
        
block_5 = torch.nn.Sequential(
        torch.nn.Conv2d(in_channels=512,
                  out_channels=512,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),            
        torch.nn.Conv2d(in_channels=512,
                  out_channels=512,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),            
        torch.nn.Conv2d(in_channels=512,
                  out_channels=512,
                  kernel_size=(3, 3),
                  stride=(1, 1),
                  padding=1),
        torch.nn.ReLU(),    
        torch.nn.MaxPool2d(kernel_size=(2, 2),
                     stride=(2, 2))             
)
            
classifier = torch.nn.Sequential(
    torch.nn.Flatten(),
    torch.nn.Linear(512, 4096),
    torch.nn.ReLU(True),
    #torch.nn.Dropout(p=0.5),
    torch.nn.Linear(4096, 4096),
    torch.nn.ReLU(True),
    #torch.nn.Dropout(p=0.5),
    torch.nn.Linear(4096, num_classes),
)<jupyter_output><empty_output><jupyter_text>Before setting up the environment for the distributed run, we check if the distributed setting is supported on our machine. The following should return `True`:<jupyter_code>torch.distributed.is_available()<jupyter_output><empty_output><jupyter_text>Next, we set the following environment variables for your machine: For `MASTER_ADDR` just use the IP address of your machine. E.g., 123.45.67.89<jupyter_code>%env MASTER_ADDR=xxx.xx.xx.xx<jupyter_output><empty_output><jupyter_text>Choose a free port:<jupyter_code>%env MASTER_PORT=8891<jupyter_output>env: MASTER_PORT=8891<jupyter_text>Set up the RPC if it is not already running (more details at https://pytorch.org/docs/stable/rpc.html):<jupyter_code>try:
    torch.distributed.rpc.init_rpc(name='node1', rank=0, world_size=1)
except RuntimeError as e:
    if str(e) == 'Address already in use':
        pass
    else:
        raise RuntimeError(e)<jupyter_output><empty_output><jupyter_text>This is the main part for running the model on multiple GPUs.1. We wrap the individual blocks into a `Sequential` model2. The chunks refer to the `microbatches`, for more details, see https://pytorch.org/docs/1.8.0/pipeline.html?highlight=pipeline<jupyter_code>from torch.distributed.pipeline.sync import Pipe


block1 = block_1.cuda(0)
block2 = block_2.cuda(0)
block3 = block_3.cuda(2)
block4 = block_4.cuda(2)
block4 = block_5.cuda(3)
block4 = classifier.cuda(0)

model_parallel = torch.nn.Sequential(
    block_1, block_2, block_3, block_4, block_5, classifier)
model_parallel = Pipe(model_parallel, chunks=8)
optimizer = torch.optim.Adam(model_parallel.parameters(), lr=learning_rate)
_ = train_classifier_simple_v1(num_epochs=num_epochs, model=model_parallel, 
                               optimizer=optimizer, device=torch.device('cuda:0'), 
                               train_loader=train_loader, valid_loader=valid_loader, 
                               logging_interval=200)<jupyter_output>Epoch: 001/050 | Batch 0000/0352 | Loss: 2.3043
Epoch: 001/050 | Batch 0200/0352 | Loss: 2.0182
***Epoch: 001/050 | Train. Acc.: 21.711% | Loss: 1.914
***Epoch: 001/050 | Valid. Acc.: 21.420% | Loss: 1.906
Time elapsed: 1.35 min
Epoch: 002/050 | Batch 0000/0352 | Loss: 1.9691
Epoch: 002/050 | Batch 0200/0352 | Loss: 1.7432
***Epoch: 002/050 | Train. Acc.: 28.180% | Loss: 1.881
***Epoch: 002/050 | Valid. Acc.: 28.580% | Loss: 1.895
Time elapsed: 2.68 min
Epoch: 003/050 | Batch 0000/0352 | Loss: 1.8837
Epoch: 003/050 | Batch 0200/0352 | Loss: 1.5357
***Epoch: 003/050 | Train. Acc.: 38.922% | Loss: 1.584
***Epoch: 003/050 | Valid. Acc.: 38.960% | Loss: 1.587
Time elapsed: 4.02 min
Epoch: 004/050 | Batch 0000/0352 | Loss: 1.6131
Epoch: 004/050 | Batch 0200/0352 | Loss: 1.4481
***Epoch: 004/050 | Train. Acc.: 48.900% | Loss: 1.360
***Epoch: 004/050 | Valid. Acc.: 49.240% | Loss: 1.367
Time elapsed: 5.36 min
Epoch: 005/050 | Batch 0000/0352 | Loss: 1.3710
Epoch: 005/050 | Batch 0200/0352 | L[...]
{'fraction_non_alphanumeric': 0.12942265723166457, 'fraction_numerical': 0.07790490046954567, 'mean_word_length': 1.6072413793103448, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '223559', 'license': 'MIT', 'n_tokens_mistral': 5575, 'n_tokens_neox': 4822, 'n_words': 935, 'path': 'pytorch_ipynb/mechanics/model-pipeline-vgg16.ipynb', 'repo_name': 'NeoGitCrt1/deeplearning-models', 'size': '15154'}