<jupyter_start><jupyter_text>Custom LayersOne factor behind deep learning's successis the availability of a wide range of layersthat can be composed in creative waysto design architectures suitablefor a wide variety of tasks.For instance, researchers have invented layersspecifically for handling images, text,looping over sequential data,andperforming dynamic programming.Sooner or later, you will encounter or inventa layer that does not exist yet in the deep learning framework.In these cases, you must build a custom layer.In this section, we show you how. Layers without ParametersTo start, we construct a custom layerthat does not have any parameters of its own.This should look familiar if you recall ourintroduction to block in :numref:`sec_model_construction`.The following `CenteredLayer` class simplysubtracts the mean from its input.To build it, we simply need to inheritfrom the base layer class and implement the forward propagation function.<jupyter_code>import torch
from torch import nn
from torch.nn import functional as F

class CenteredLayer(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, X):
        return X - X.mean()<jupyter_output><empty_output><jupyter_text>Let us verify that our layer works as intended by feeding some data through it.<jupyter_code>layer = CenteredLayer()
layer(torch.FloatTensor([1, 2, 3, 4, 5]))<jupyter_output><empty_output><jupyter_text>We can now incorporate our layer as a componentin constructing more complex models.<jupyter_code>net = nn.Sequential(nn.Linear(8, 128), CenteredLayer())<jupyter_output><empty_output><jupyter_text>As an extra sanity check, we can send random datathrough the network and check that the mean is in fact 0.Because we are dealing with floating point numbers,we may still see a very small nonzero numberdue to quantization.<jupyter_code>Y = net(torch.rand(4, 8))
Y.mean()<jupyter_output><empty_output><jupyter_text>Layers with ParametersNow that we know how to define simple layers,let us move on to defining layers with parametersthat can be adjusted through training.We can use built-in functions to create parameters, whichprovide some basic housekeeping functionality.In particular, they govern access, initialization,sharing, saving, and loading model parameters.This way, among other benefits, we will not need to writecustom serialization routines for every custom layer.Now let us implement our own version of the  fully-connected layer.Recall that this layer requires two parameters,one to represent the weight and the other for the bias.In this implementation, we bake in the ReLU activation as a default.This layer requires to input arguments: `in_units` and `units`, whichdenote the number of inputs and outputs, respectively.<jupyter_code>class MyLinear(nn.Module):
    def __init__(self, in_units, units):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(in_units, units))
        self.bias = nn.Parameter(torch.randn(units,))
    def forward(self, X):
        linear = torch.matmul(X, self.weight.data) + self.bias.data
        return F.relu(linear)<jupyter_output><empty_output><jupyter_text>Next, we instantiate the `MyDense` classand access its model parameters.<jupyter_code>dense = MyLinear(5, 3)
dense.weight<jupyter_output><empty_output><jupyter_text>We can directly carry out forward propagation calculations using custom layers.<jupyter_code>dense(torch.rand(2, 5))<jupyter_output><empty_output><jupyter_text>We can also construct models using custom layers.Once we have that we can use it just like the built-in fully-connected layer.<jupyter_code>net = nn.Sequential(MyLinear(64, 8), MyLinear(8, 1))
net(torch.rand(2, 64))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07173562600752284, 'fraction_numerical': 0.006448146157979581, 'mean_word_length': 5.756805807622505, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '15299', 'license': 'MIT', 'n_tokens_mistral': 1038, 'n_tokens_neox': 1002, 'n_words': 409, 'path': 'd2l-en/pytorch/chapter_deep-learning-computation/custom-layer.ipynb', 'repo_name': 'gr8khan/d2lai', 'size': '3722'}