title understanding masking padding s scott zhu francois chollet date created 20190716 last modified 20230625 description complete guide to using maskaware sequence layers in keras accelerator none setup introduction masking is a way to tell sequenceprocessing layers that certain timesteps in an input are missing and thus should be skipped when processing the data padding is a special form of masking where the masked steps are at the start or the end of a sequence padding comes from the need to encode sequence data into contiguous batches in order to make all sequences in a batch fit a given standard length it is necessary to pad or truncate some sequences let s take a close look padding sequence data when processing sequence data it is very common for individual samples to have different lengths consider the following example text tokenized as words hello world how are you doing today the weather will be nice tomorrow after vocabulary lookup the data might be vectorized as integers e g 71 1331 4231 73 8 3215 55 927 83 91 1 645 1253 927 the data is a nested list where individual samples have length 3 5 and 6 respectively since the input data for a deep learning model must be a single tensor of shape e g batchsize 6 vocabsize in this case samples that are shorter than the longest item need to be padded with some placeholder value alternatively one might also truncate long samples before padding short samples keras provides a utility function to truncate and pad python lists to a common length keras utils padsequences by default this will pad using 0s it is configurable via the value parameter note that you could use pre padding at the beginning or post padding at the end we recommend using post padding when working with rnn layers in order to be able to use the cudnn implementation of the layers masking now that all samples have a uniform length the model must be informed that some part of the data is actually padding and should be ignored that mechanism is masking there are three ways to introduce input masks in keras models add a keras layers masking layer configure a keras layers embedding layer with maskzerotrue pass a mask argument manually when calling layers that support this argument e g rnn layers maskgenerating layers embedding and masking under the hood these layers will create a mask tensor 2d tensor with shape batch sequencelength and attach it to the tensor output returned by the masking or embedding layer simulate the embedding lookup by expanding the 2d input to 3d with embedding dimension of 10 as you can see from the printed result the mask is a 2d boolean tensor with shape batchsize sequencelength where each individual false entry indicates that the corresponding timestep should be ignored during processing mask propagation in the functional api and sequential api when using the functional api or the sequential api a mask generated by an embedding or masking layer will be propagated through the network for any layer that is capable of using them for example rnn layers keras will automatically fetch the mask corresponding to an input and pass it to any layer that knows how to use it for instance in the following sequential model the lstm layer will automatically receive a mask which means it will ignore padded values this is also the case for the following functional api model passing mask tensors directly to layers layers that can handle masks such as the lstm layer have a mask argument in their call method meanwhile layers that produce a mask e g embedding expose a computemaskinput previousmask method which you can call thus you can pass the output of the computemask method of a maskproducing layer to the call method of a maskconsuming layer like this note that you could also prepare a mask tensor manually it only needs to be a boolean tensor with the right shape i e batchsize timesteps supporting masking in your custom layers sometimes you may need to write layers that generate a mask like embedding or layers that need to modify the current mask for instance any layer that produces a tensor with a different time dimension than its input such as a concatenate layer that concatenates on the time dimension will need to modify the current mask so that downstream layers will be able to properly take masked timesteps into account to do this your layer should implement the layer computemask method which produces a new mask given the input and the current mask here is an example of a temporalsplit layer that needs to modify the current mask split the input tensor into 2 tensors along the time dimension def callself inputs expect the input to be 3d and mask to be 2d split the input tensor into 2 subtensors along the time axis axis 1 return ops splitinputs 2 axis1 def computemaskself inputs masknone also split the mask into 2 if it presents if mask is none return none return ops splitmask 2 axis1 firsthalf secondhalf temporalsplitmaskedembedding printfirsthalf kerasmask printsecondhalf kerasmask class customembeddingkeras layers layer def initself inputdim outputdim maskzerofalse kwargs super initkwargs self inputdim inputdim self outputdim outputdim self maskzero maskzero def buildself inputshape self embeddings self addweight shapeself inputdim self outputdim initializerrandomnormal dtypefloat32 def callself inputs inputs ops castinputs int32 return ops takeself embeddings inputs def computemaskself inputs masknone if not self maskzero return none return ops notequalinputs 0 layer customembedding10 32 maskzerotrue x np random random3 10 9 x x astypeint32 y layerx mask layer computemaskx printmask optingin to mask propagation on compatible layers class myactivationkeras layers layer def initself kwargs super initkwargs signal that the layer is safe for mask propagation self supportsmasking true def callself inputs return ops reluinputs inputs keras inputshapenone dtypeint32 x layers embeddinginputdim5000 outputdim16 maskzerotrueinputs x myactivationx will pass the mask along printmask found x kerasmask outputs layers lstm32x will receive the mask model keras modelinputs outputs y modelnp random randint0 5000 size32 100 writing layers that need mask information class temporalsoftmaxkeras layers layer def initself kwargs super initkwargs self supportsmasking true def callself inputs masknone assert mask is not none broadcastfloatmask ops expanddimsops castmask float32 1 inputsexp ops expinputs broadcastfloatmask inputssum ops sum inputsexp broadcastfloatmask axis1 keepdimstrue return inputsexp inputssum inputs keras inputshapenone dtypeint32 x layers embeddinginputdim10 outputdim32 maskzerotrueinputs x layers dense1x outputs temporalsoftmaxx model keras modelinputs outputs y modelnp random randint0 10 size32 100 summary setup introduction masking is a way to tell sequence processing layers that certain timesteps in an input are missing and thus should be skipped when processing the data padding is a special form of masking where the masked steps are at the start or the end of a sequence padding comes from the need to encode sequence data into contiguous batches in order to make all sequences in a batch fit a given standard length it is necessary to pad or truncate some sequences let s take a close look padding sequence data when processing sequence data it is very common for individual samples to have different lengths consider the following example text tokenized as words hello world how are you doing today the weather will be nice tomorrow after vocabulary lookup the data might be vectorized as integers e g 71 1331 4231 73 8 3215 55 927 83 91 1 645 1253 927 the data is a nested list where individual samples have length 3 5 and 6 respectively since the input data for a deep learning model must be a single tensor of shape e g batch_size 6 vocab_size in this case samples that are shorter than the longest item need to be padded with some placeholder value alternatively one might also truncate long samples before padding short samples keras provides a utility function to truncate and pad python lists to a common length keras utils pad_sequences by default this will pad using 0s it is configurable via the value parameter note that you could use pre padding at the beginning or post padding at the end we recommend using post padding when working with rnn layers in order to be able to use the cudnn implementation of the layers masking now that all samples have a uniform length the model must be informed that some part of the data is actually padding and should be ignored that mechanism is masking there are three ways to introduce input masks in keras models add a keras layers masking layer configure a keras layers embedding layer with mask_zero true pass a mask argument manually when calling layers that support this argument e g rnn layers mask generating layers embedding and masking under the hood these layers will create a mask tensor 2d tensor with shape batch sequence_length and attach it to the tensor output returned by the masking or embedding layer simulate the embedding lookup by expanding the 2d input to 3d with embedding dimension of 10 as you can see from the printed result the mask is a 2d boolean tensor with shape batch_size sequence_length where each individual false entry indicates that the corresponding timestep should be ignored during processing mask propagation in the functional api and sequential api when using the functional api or the sequential api a mask generated by an embedding or masking layer will be propagated through the network for any layer that is capable of using them for example rnn layers keras will automatically fetch the mask corresponding to an input and pass it to any layer that knows how to use it for instance in the following sequential model the lstm layer will automatically receive a mask which means it will ignore padded values this is also the case for the following functional api model passing mask tensors directly to layers layers that can handle masks such as the lstm layer have a mask argument in their __call__ method meanwhile layers that produce a mask e g embedding expose a compute_mask input previous_mask method which you can call thus you can pass the output of the compute_mask method of a mask producing layer to the __call__ method of a mask consuming layer like this note that you could also prepare a mask tensor manually it only needs to be a boolean tensor with the right shape i e batch_size timesteps the layer will ignore the masked values supporting masking in your custom layers sometimes you may need to write layers that generate a mask like embedding or layers that need to modify the current mask for instance any layer that produces a tensor with a different time dimension than its input such as a concatenate layer that concatenates on the time dimension will need to modify the current mask so that downstream layers will be able to properly take masked timesteps into account to do this your layer should implement the layer compute_mask method which produces a new mask given the input and the current mask here is an example of a temporalsplit layer that needs to modify the current mask split the input tensor into 2 tensors along the time dimension expect the input to be 3d and mask to be 2d split the input tensor into 2 subtensors along the time axis axis 1 also split the mask into 2 if it presents here is another example of a customembedding layer that is capable of generating a mask from input values note for more details about format limitations related to masking see the serialization guide guides serialization_and_saving opting in to mask propagation on compatible layers most layers don t modify the time dimension so don t need to modify the current mask however they may still want to be able to propagate the current mask unchanged to the next layer this is an opt in behavior by default a custom layer will destroy the current mask since the framework has no way to tell whether propagating the mask is safe to do if you have a custom layer that does not modify the time dimension and if you want it to be able to propagate the current input mask you should set self supports_masking true in the layer constructor in this case the default behavior of compute_mask is to just pass the current mask through here s an example of a layer that is whitelisted for mask propagation signal that the layer is safe for mask propagation you can now use this custom layer in between a mask generating layer like embedding and a mask consuming layer like lstm and it will pass the mask along so that it reaches the mask consuming layer will pass the mask along will receive the mask writing layers that need mask information some layers are mask consumers they accept a mask argument in call and use it to determine whether to skip certain time steps to write such a layer you can simply add a mask none argument in your call signature the mask associated with the inputs will be passed to your layer whenever it is available here s a simple example below a layer that computes a softmax over the time dimension axis 1 of an input sequence while discarding masked timesteps summary that is all you need to know about padding masking in keras to recap masking is how layers are able to know when to skip ignore certain timesteps in sequence inputs some layers are mask generators embedding can generate a mask from input values if mask_zero true and so can the masking layer some layers are mask consumers they expose a mask argument in their __call__ method this is the case for rnn layers in the functional api and sequential api mask information is propagated automatically when using layers in a standalone way you can pass the mask arguments to layers manually you can easily write layers that modify the current mask that generate a new mask or that consume the mask associated with the inputs
import numpy as np
import keras
from keras import ops
from keras import layers





raw_inputs = [
    [711, 632, 71],
    [73, 8, 3215, 55, 927],
    [83, 91, 1, 645, 1253, 927],
]








padded_inputs = keras.utils.pad_sequences(raw_inputs, padding="post")
print(padded_inputs)






embedding = layers.Embedding(input_dim=5000, output_dim=16, mask_zero=True)
masked_output = embedding(padded_inputs)

print(masked_output._keras_mask)

masking_layer = layers.Masking()


unmasked_embedding = ops.cast(
    ops.tile(ops.expand_dims(padded_inputs, axis=-1), [1, 1, 10]),
    dtype="float32",
)

masked_embedding = masking_layer(unmasked_embedding)
print(masked_embedding._keras_mask)





model = keras.Sequential(
    [
        layers.Embedding(input_dim=5000, output_dim=16, mask_zero=True),
        layers.LSTM(32),
    ]
)



inputs = keras.Input(shape=(None,), dtype="int32")
x = layers.Embedding(input_dim=5000, output_dim=16, mask_zero=True)(inputs)
outputs = layers.LSTM(32)(x)

model = keras.Model(inputs, outputs)






class MyLayer(layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.embedding = layers.Embedding(
            input_dim=5000, output_dim=16, mask_zero=True
        )
        self.lstm = layers.LSTM(32)

    def call(self, inputs):
        x = self.embedding(inputs)
        
        
        
        mask = self.embedding.compute_mask(inputs)
        output = self.lstm(
            x, mask=mask
        )  
        return output


layer = MyLayer()
x = np.random.random((32, 10)) * 100
x = x.astype("int32")
layer(x)






class TemporalSplit(keras.layers.Layer):
    

    def call(self, inputs):
        
        
        return ops.split(inputs, 2, axis=1)

    def compute_mask(self, inputs, mask=None):
        
        if mask is None:
            return None
        return ops.split(mask, 2, axis=1)


first_half, second_half = TemporalSplit()(masked_embedding)
print(first_half._keras_mask)
print(second_half._keras_mask)




class CustomEmbedding(keras.layers.Layer):
    def __init__(self, input_dim, output_dim, mask_zero=False, **kwargs):
        super().__init__(**kwargs)
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.mask_zero = mask_zero

    def build(self, input_shape):
        self.embeddings = self.add_weight(
            shape=(self.input_dim, self.output_dim),
            initializer="random_normal",
            dtype="float32",
        )

    def call(self, inputs):
        inputs = ops.cast(inputs, "int32")
        return ops.take(self.embeddings, inputs)

    def compute_mask(self, inputs, mask=None):
        if not self.mask_zero:
            return None
        return ops.not_equal(inputs, 0)


layer = CustomEmbedding(10, 32, mask_zero=True)
x = np.random.random((3, 10)) * 9
x = x.astype("int32")

y = layer(x)
mask = layer.compute_mask(x)

print(mask)






class MyActivation(keras.layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        
        self.supports_masking = True

    def call(self, inputs):
        return ops.relu(inputs)




inputs = keras.Input(shape=(None,), dtype="int32")
x = layers.Embedding(input_dim=5000, output_dim=16, mask_zero=True)(inputs)
x = MyActivation()(x)  
print("Mask found:", x._keras_mask)
outputs = layers.LSTM(32)(x)  

model = keras.Model(inputs, outputs)
y = model(np.random.randint(0, 5000, size=(32, 100)))




class TemporalSoftmax(keras.layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.supports_masking = True

    def call(self, inputs, mask=None):
        assert mask is not None
        broadcast_float_mask = ops.expand_dims(ops.cast(mask, "float32"), -1)
        inputs_exp = ops.exp(inputs) * broadcast_float_mask
        inputs_sum = ops.sum(
            inputs_exp * broadcast_float_mask, axis=-1, keepdims=True
        )
        return inputs_exp / inputs_sum


inputs = keras.Input(shape=(None,), dtype="int32")
x = layers.Embedding(input_dim=10, output_dim=32, mask_zero=True)(inputs)
x = layers.Dense(1)(x)
outputs = TemporalSoftmax()(x)

model = keras.Model(inputs, outputs)
y = model(np.random.randint(0, 10, size=(32, 100)))