<commit_before>Squeeze-and- Excitation (SE) Block
=============================

We will show you how to...

* Implement a Squeeze-and- Excitation (SE) block

Squeeze-and- Excitation blocks `(Hu at el., 2017) <https://arxiv.org/abs/1709.01507>`_ seek to
improve the quality of representations learned by a network by modelling the dependencies between channels of convolutional features.
The first step in the block is the Squeeze operation: a copy of all channels are aggregated by taking their means, generating 
a tensor with shape (1, 1, C). Then we have the Excitation operation: the mean vector passes through a 2 layer MLP
before being  used in a self-gating mechanism, where it modulates each channel of the original block. 
The output retains the shape (H, W, C) and therefore can just be passed on to the next layer.
.. code-block:: python


  class SEBlock(nn.Module):
    """The Squeeze-and-Excitation block."""
    hidden_size: int
    act: Callable = nn.relu 
    axis: Tuple[int, int] = (1, 2), 

    @nn.compact
    def __call__(self, x):
        y = x.mean(axis=axis, keepdims=True)
        y = nn.Dense(
            y, features=self.hidden_size,
            name='reduce')
        y = act(y)
        y = nn.Dense(
            y, features=x.shape[-1],
            name='expand')<commit_msg>Apply suggestions from code review<commit_after>Squeeze-and-Excitation Block
=============================

We will show you how to...

* Implement a Squeeze-and-Excitation block

Squeeze-and-Excitation blocks `(Hu at el., 2017) <https://arxiv.org/abs/1709.01507>`_ seek to
improve the quality of representations learned by a network by modelling the dependencies between channels of convolutional features.
The first step in the block is the Squeeze operation: a copy of all channels are aggregated by taking their means, generating 
a tensor with shape (1, 1, C). Then we have the Excitation operation: the mean vector passes through a 2 layer MLP
before being  used in a self-gating mechanism, where it modulates each channel of the original block. 
The output retains the shape (H, W, C) and therefore can just be passed on to the next layer.
.. code-block:: python


  class SEBlock(nn.Module):
    """The Squeeze-and-Excitation block."""
    hidden_size: int
    act: Callable = nn.relu 
    axis: Tuple[int, int] = (1, 2), 

    @nn.compact
    def __call__(self, x):
        y = x.mean(axis=axis, keepdims=True)
        y = nn.Dense(
            y, features=self.hidden_size,
            name='reduce')
        y = act(y)
        y = nn.Dense(
            y, features=x.shape[-1],
            name='expand')
{'fraction_non_alphanumeric': 0.11289092295957284, 'fraction_numerical': 0.014492753623188406, 'mean_word_length': 3.0984375, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '227918', 'n_tokens_mistral': 819, 'n_tokens_neox': 752, 'n_words': 316}