<jupyter_start><jupyter_text>IntroductionIn this tutorial, we will go through:- Basics of efficient blocks in PytorchVideo/Accelerator;- Design, train and deploy a model composed of efficient blocks for mobile CPU.Before we start, let's install PytorchVideo.<jupyter_code>!pip install pytorchvideo<jupyter_output><empty_output><jupyter_text>Basics of efficient blocks in PytorchVideo/AcceleratorEfficient blocks are blocks with high efficiency. For a target device, we benchmark efficiency of basic network components and provide a collection of efficient blocks under `pytorchvideo/layers/accelerator/` (for simple layers) and `pytorchvideo/models/accelerator/` (for complex modules such as residual block). Inferencing of a model built up with corresponding efficient blocks on target device is guranteed to be efficient.Each efficient block module is an instance of nn.Module, and has two forms: **original form** (for training) and **deploy form** (for inference). When in original form, the efficient block module has exactly the same behavior as a corresponding vanilla nn.Module for both forward and backward operation. User can freely mix and match efficient blocks for the same target device and build up their own model. Once model is built and trained, user can convert each efficient block in model into deploy form. The conversion will do graph and kernel optimization on each efficient block, and efficient block in deploy form is arithmetically equivalent to original form but has much higher efficiency during inference.   Design, train and deploy a model composed of efficient blocks for mobile CPU Build a modelIn this section, let's go through the process of design, train and deploy using a example toy model using efficient blocks under `pytorchvideo/layers/accelerator/mobile_cpu` and `pytorchvideo/models/accelerator/mobile_cpu`, which includes:- One conv3d head layer with 5x1x1 kernel followed by ReLU activation;- One residual block with squeeze-excite;- One average pool and fully connected layer as final output.First, let's import efficient blocks.<jupyter_code># Imports
import torch.nn as nn
from pytorchvideo.layers.accelerator.mobile_cpu.activation_functions import (
    supported_act_functions,
)
from pytorchvideo.layers.accelerator.mobile_cpu.convolutions import (
    Conv3d5x1x1BnAct,
)
from pytorchvideo.models.accelerator.mobile_cpu.residual_blocks import (
    X3dBottleneckBlock,
)
from pytorchvideo.layers.accelerator.mobile_cpu.pool import AdaptiveAvgPool3dOutSize1
from pytorchvideo.layers.accelerator.mobile_cpu.fully_connected import FullyConnected<jupyter_output><empty_output><jupyter_text>Then we can build a model using those efficient blocks.<jupyter_code>class MyNet(nn.Module):
    def __init__(
        self,
        in_channel=3,  # input channel of first 5x1x1 layer
        residual_block_channel=24,  # input channel of residual block
        expansion_ratio=3, # expansion ratio of residual block
        num_classes=4, # final output classes
    ):
        super().__init__()
        # s1 - 5x1x1 conv3d layer
        self.s1 = Conv3d5x1x1BnAct(
            in_channel,
            residual_block_channel,
            bias=False,
            groups=1,
            use_bn=False,
        )
        # s2 - residual block
        mid_channel = int(residual_block_channel * expansion_ratio)
        self.s2 = X3dBottleneckBlock(
                in_channels=residual_block_channel,
                mid_channels=mid_channel,
                out_channels=residual_block_channel,
                use_residual=True,
                spatial_stride=1,
                se_ratio=0.0625,
                act_functions=("relu", "swish", "relu"),
                use_bn=(True, True, True),
            )
        # Average pool and fully connected layer
        self.avg_pool = AdaptiveAvgPool3dOutSize1()
        self.projection = FullyConnected(residual_block_channel, num_classes, bias=True)
        self.act = supported_act_functions['relu']()

    def forward(self, x):
        x = self.s1(x)
        x = self.s2(x)
        x = self.avg_pool(x)
        # (N, C, T, H, W) -> (N, T, H, W, C).
        x = x.permute((0, 2, 3, 4, 1))
        x = self.projection(x)
        # Performs fully convlutional inference.
        if not self.training:
            x = self.act(x)
            x = x.mean([1, 2, 3])
        x = x.view(x.shape[0], -1)

        return x<jupyter_output><empty_output><jupyter_text>We can instantiate MyNet and its efficient blocks will be in original form.<jupyter_code>net_inst = MyNet()
print(net_inst)<jupyter_output><empty_output><jupyter_text>Train modelThen we can train the model with your dataset/optimizer. Here we skip this training step, and just leave the weight as initial value.  Deploy modelNow the model is ready to deploy. First of all, let's convert the model into deploy form. In order to do that, we need to use `convert_to_deployable_form` utility and provide an example input tensor to the model. Note that once the model is converted into deploy form, the input size should be the same as the example input tensor size during conversion.<jupyter_code>import torch
from pytorchvideo.accelerator.deployment.mobile_cpu.utils.model_conversion import (
    convert_to_deployable_form,
)
input_blob_size = (1, 3, 4, 6, 6)
input_tensor = torch.randn(input_blob_size)
net_inst_deploy = convert_to_deployable_form(net_inst, input_tensor)<jupyter_output><empty_output><jupyter_text>We can see that the network graph has been changed after conversion, which did kernel and graph optimization.<jupyter_code>print(net_inst_deploy)<jupyter_output><empty_output><jupyter_text>Let's check whether the network after conversion is arithmetically equivalent. We expect the output to be very close before/after conversion, with some small difference due to numeric noise from floating point operation.<jupyter_code>net_inst.eval()
out_ref = net_inst(input_tensor)
out = net_inst_deploy(input_tensor)

max_err = float(torch.max(torch.abs(out_ref - out)))
print(f"max error is {max_err}")<jupyter_output><empty_output><jupyter_text>Next we have two options: either deploy floating point model, or quantize model into int8 and then deploy.Let's first assume we want to deploy floating point model. In this case, all we need to do is to export jit trace and then apply `optimize_for_mobile` for final optimization.<jupyter_code>from torch.utils.mobile_optimizer import (
    optimize_for_mobile,
)
traced_model = torch.jit.trace(net_inst_deploy, input_tensor, strict=False)
traced_model_opt = optimize_for_mobile(traced_model)
# Here we can save the traced_model_opt to JIT file using traced_model_opt.save(<file_path>)<jupyter_output><empty_output><jupyter_text>Alternatively, we may also want to deploy a quantized model. Efficient blocks are quantization-friendly by design - just wrap the model in deploy form with `QuantStub/DeQuantStub` and it is ready for Pytorch eager mode quantization.<jupyter_code># Wrapper class for adding QuantStub/DeQuantStub.
class quant_stub_wrapper(nn.Module):
    def __init__(self, module_in):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.model = module_in
        self.dequant = torch.quantization.DeQuantStub()
    def forward(self, x):
        x = self.quant(x)
        x = self.model(x)
        x = self.dequant(x)
        return x
net_inst_quant_stub_wrapper = quant_stub_wrapper(net_inst_deploy)<jupyter_output><empty_output><jupyter_text>Preparation step of quantization. Fusion has been done for efficient blocks automatically during `convert_to_deployable_form`, so we can just proceed to `torch.quantization.prepare`<jupyter_code>net_inst_quant_stub_wrapper.qconfig = torch.quantization.default_qconfig
net_inst_quant_stub_wrapper_prepared = torch.quantization.prepare(net_inst_quant_stub_wrapper)<jupyter_output><empty_output><jupyter_text>Calibration and quantization. After preparation we will do calibration of quantization by feeding calibration dataset (skipped here) and then do quantization.<jupyter_code># calibration is skipped here.
net_inst_quant_stub_wrapper_quantized = torch.quantization.convert(net_inst_quant_stub_wrapper_prepared)
print(net_inst_quant_stub_wrapper_quantized)<jupyter_output><empty_output><jupyter_text>Then we can export trace of int8 model and deploy on mobile devices.<jupyter_code>traced_model_int8 = torch.jit.trace(net_inst_quant_stub_wrapper_quantized, input_tensor, strict=False)
traced_model_int8_opt = optimize_for_mobile(traced_model_int8)
# Here we can save the traced_model_opt to JIT file using traced_model_int8_opt.save(<file_path>)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06861953638565332, 'fraction_numerical': 0.007380924922154307, 'mean_word_length': 4.534141671984684, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '313654', 'license': 'Apache-2.0', 'n_tokens_mistral': 2610, 'n_tokens_neox': 2525, 'n_words': 812, 'path': 'tutorials/accelerator/Build_your_model_with_PytorchVideo_Accelerator.ipynb', 'repo_name': 'nateraw/pytorchvideo', 'size': '8671'}