<jupyter_start><jupyter_text>InitializationIn the [Neural Networks](nn.html) section we played fast and loose with settingup our networks. In particular we did the following things that *shouldn't*work:* We defined the network architecture with no regard to the input  dimensionality.* We added layers without regard to the output dimension of the previous layer.* We even 'initialized' these parameters without knowing how many parameters  were were to initialize.All of those things sound impossible and indeed, they are. After all, there'sno way MXNet (or any other framework for that matter) could predict what theinput dimensionality of a network would be. Later on, when working withconvolutional networks and images this problem will become even more pertinent,since the input dimensionality (i.e. the resolution of an image) will affectthe dimensionality of subsequent layers. The ability todetermine parameter dimensionality during run-time rather than at coding timegreatly simplifies the process of doing deep learning.  Instantiating a NetworkLet's see what happens when we instantiate a network. We start by defining a multi-layer perceptron.<jupyter_code>from mxnet import init, nd
from mxnet.gluon import nn


def getnet():
    net = nn.Sequential()
    net.add(nn.Dense(256, activation='relu'))
    net.add(nn.Dense(10))
    return net

net = getnet()<jupyter_output><empty_output><jupyter_text>At this point the network doesn't really know yet what the dimensionalities ofthe various parameters should be. All one could tell at this point is that eachlayer needs weights and bias, albeit of unspecified dimensionality. If we tryaccessing the parameters, that's exactly what happens.<jupyter_code>print(net.collect_params())<jupyter_output><empty_output><jupyter_text>You'll notice `None` here in each `Dense` layer. This absense of value is howMXNet keeps track of unspecified dimensionality. In particular, trying to access`net[0].weight.data()` at this point would trigger a runtime error stating thatthe network needs initializing before it can do anything. Note that if we did want to specify dimensionality, we could have done so byusing the kwarg `in_units`, e.g. `Dense(256, activiation='relu', in_units=20)`.Let's see whether anything changes after we initialize the parameters:<jupyter_code>net.initialize()
net.collect_params()<jupyter_output><empty_output><jupyter_text>As we can see, nothing really changed. Only once we provide the network withsome data do we see a difference. Let's try it out.<jupyter_code>x = nd.random.uniform(shape=(2, 20))
net(x)  # Forward computation
print(net.collect_params())<jupyter_output><empty_output><jupyter_text>We see all the dimensions have been determined and the parameters initialized.This is because shape inference and parameter initialization have beenperformed in a lazy manner, so they are performed only when needed. In theabove case, they are performed as a prerequisite to the forward computation.Dimensional inference works like this: as soon as we knew the inputdimensionality, $\mathbf{x} \in \mathbb{R}^{20}$ it was possible to define theweight matrix for the first layer, i.e. $\mathbf{W}_1 \in \mathbb{R}^{256 \times20}$. With that out of the way, we can progress to the second layer, define itsdimensionality to be $10 \times 256$ and so on through the computational graphand resolve all the dimensions as they become available. Once this is known, wecan proceed by initializing parameters. This is the solution to the threeproblems outlined above. Deferred Initialization in PracticeNow that we know how it works in theory, let's see when the initialization isactually triggered. In order to do so, we mock up an initializer which doesnothing but report a debug message stating when it was invoked and with whichparameters.<jupyter_code>class PrintInit(init.Initializer):
    def _init_weight(self, name, data):
        print('Init', name, data.shape)
        # The actual initialization logic is omitted here.

net = getnet()
net.initialize(init=PrintInit())<jupyter_output><empty_output><jupyter_text>Note that, although `MyInit` will print information about the model parameterswhen it is called, the above `initialize` function does not print anyinformation after it has been executed.  Therefore there is no actualinitialization when calling the `initialize` function - this+initialization is deferred until forward is called for the first time. Next,we define the input and perform a forward calculation.<jupyter_code>x = nd.random.uniform(shape=(2, 20))
y = net(x)<jupyter_output><empty_output><jupyter_text>At this time, information on the model parameters is printed. When performing aforward calculation based on the input `x`, the system can automatically inferthe shape of the weight parameters of all layers based on the shape of theinput. Once the system has created these parameters, it calls the `MyInit`instance to initialize them before proceeding to the forward calculation.Of course, this initialization will only be called when completing the initialforward calculation. After that, we will not re-initialize when we run theforward calculation `net(x)`, so the output of the `MyInit` instance will not begenerated again.<jupyter_code>y = net(x)<jupyter_output><empty_output><jupyter_text>As mentioned at the beginning of this section, deferred initialization can alsocause confusion. Before the first forward calculation, we were unable todirectly manipulate the model parameters, for example, we could not use the`data` and `set_data` functions to get and modify the parameters. Therefore, weoften force initialization by sending a sample observation through the network. Forced InitializationDeferred initialization does not occur if the system knows the shape of allparameters when calling the `initialize` function. This can occur in two cases:* We've already seen some data and we just want to reset the parameters.* We specified all input and output dimensions of the network or layer when  defining it.The first case works just fine, as illustrated below.<jupyter_code>net.initialize(init=MyInit(), force_reinit=True)<jupyter_output><empty_output><jupyter_text>The second case requires us to specify the remaining set of parameters whencreating the layer. For instance, for dense layers we also need to specify the`in_units` so that initialization can occur immediately once `initialize` iscalled.<jupyter_code>net = nn.Sequential()
net.add(nn.Dense(256, in_units=20, activation='relu'))
net.add(nn.Dense(10, in_units=256))

net.initialize(init=MyInit())<jupyter_output><empty_output><jupyter_text>Parameter InitializationBy default, MXNet initializes the weight matrices uniformly by drawing randomvalues with uniform-distribution between $-0.07$ and $0.07$ ($U[-0.07, 0.07]$)and updates the bias parameters by setting them all to $0$.  However, we oftenneed to use other methods to initialize the weights.  MXNet's `init` moduleprovides a variety of preset initialization methods, but if we want somethingout of the ordinary, we need a bit of extra work. Built-in InitializationLet's begin with the built-in initializers. The code below initializes allparameters with Gaussian random variables.<jupyter_code># force_reinit ensures that the variables are initialized again, regardless of
# whether they were already initialized previously.
net.initialize(init=init.Normal(sigma=0.01), force_reinit=True)
print(net[0].weight.data()[0])<jupyter_output><empty_output><jupyter_text>If we wanted to initialize all parameters to $1$, we could do this simply bychanging the initializer to `Constant(1)`.<jupyter_code>net.initialize(init=init.Constant(1), force_reinit=True)
net[0].weight.data()[0]<jupyter_output><empty_output><jupyter_text>If we want to initialize only a specific parameter in a different manner, wecan simply set the initializer only for the appropriate subblock (orparameter). For instance, below we initialize the second layer to a constantvalue of $42$ and we use the `Xavier` initializer for the weights of thefirst layer.<jupyter_code>net[0].weight.initialize(init=init.Xavier(), force_reinit=True)
net[1].initialize(init=init.Constant(42), force_reinit=True)

# First layer
print(net[0].weight.data()[0])
print(net[0].bias.data()[0])  # initialized to 0

# Second layer
print(net[1].weight.data()[0,0])
print(net[1].bias.data()[0])  # initialized to 0<jupyter_output><empty_output><jupyter_text>Custom InitializationSometimes, the initialization methods we need are not provided in the `init`module. At this point, we can implement a subclass of the `Initializer` classso that we can use it like any other initialization method. Usually, we onlyneed to implement the `_init_weight` function to suit our needs. In the examplebelow, we pick a decidedly bizarre and nontrivial distribution, just to provethe point. We draw the coefficients from the following distribution:$$\begin{aligned}    w \sim \begin{cases}        U[5, 10] & \text{ with probability } \frac{1}{4} \\            0    & \text{ with probability } \frac{1}{2} \\        U[-10, -5] & \text{ with probability } \frac{1}{4}    \end{cases}\end{aligned}$$<jupyter_code>class MyInit(init.Initializer):
    def _init_weight(self, name, data):
        print('Init', name, data.shape)
        data[:] = nd.random.uniform(low=-10, high=10, shape=data.shape)
        data *= data.abs() >= 5

net.initialize(MyInit(), force_reinit=True)
net[0].weight.data()[0]<jupyter_output><empty_output><jupyter_text>If this functionality is insufficient, we can even set parameters directly.Since `data()` returns an `NDArray` we can access it just like any other matrix.A note for advanced users - if you want to adjust parameters within an`autograd` scope you need to use `set_data` to avoid confusing the automaticdifferentiation mechanics.<jupyter_code>net[0].weight.data()[:] += 1
net[0].weight.data()[0,0] = 42
net[0].weight.data()[0]<jupyter_output><empty_output><jupyter_text>Tied ParametersIn some cases, we want to share model parameters across multiple layers. Forinstance when we want to find good word embeddings we may decide to use thesame parameters both for encoding and decoding of words. Let's see how to dothis a bit more elegantly. In the following we construct a dense layer and thenuse its parameters specifically to set those of another layer.<jupyter_code>net = nn.Sequential()
# We need to give the shared layer a name such that we can reference its
# parameters.
shared = nn.Dense(8, activation='relu')
net.add(nn.Dense(8, activation='relu'),
        shared,
        nn.Dense(8, activation='relu', params=shared.params),
        nn.Dense(10))
net.initialize()

x = nd.random.uniform(shape=(2, 20))
net(x)

# Check whether the parameters are the same.
print(net[1].weight.data()[0] == net[2].weight.data()[0])
net[1].weight.data()[0,0] = 100
# And make sure that they're actually the same object rather than just having
# the same value.
print(net[1].weight.data()[0] == net[2].weight.data()[0])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08244150190458915, 'fraction_numerical': 0.011881008525303827, 'mean_word_length': 5.595095693779904, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '546481', 'license': 'MIT', 'n_tokens_mistral': 3023, 'n_tokens_neox': 2911, 'n_words': 1289, 'path': 'static_websites/python/docs/_sources/tutorials/packages/gluon/init.ipynb', 'repo_name': 'IvyBazan/mxnet.io-v2', 'size': '11026'}