regularizer base class regularizers allow you to apply penalties on layer parameters or layer activity during optimization these penalties are summed into the loss function that the network optimizes regularization penalties are applied on a perlayer basis the exact api will depend on the layer but many layers e g dense conv1d conv2d and conv3d have a unified api these layers expose 3 keyword arguments kernelregularizer regularizer to apply a penalty on the layer s kernel biasregularizer regularizer to apply a penalty on the layer s bias activityregularizer regularizer to apply a penalty on the layer s output all layers including custom layers expose activityregularizer as a settable property whether or not it is in the constructor arguments the value returned by the activityregularizer is divided by the input batch size so that the relative weighting between the weight regularizers and the activity regularizers does not change with the batch size you can access a layer s regularization penalties by calling layer losses after calling the layer on inputs example layer dense 5 inputdim5 kernelinitializer ones kernelregularizerl10 01 activityregularizerl20 01 tensor ops onesshape5 5 2 0 out layertensor the kernel regularization term is 0 25 the activity regularization term after dividing by the batch size is 5 ops sumlayer losses 5 25 available penalties python l10 3 l1 regularization penalty l20 1 l2 regularization penalty l1l2l10 01 l20 01 l1 l2 penalties directly calling a regularizer compute a regularization loss on a tensor by directly calling a regularizer as if it is a oneargument function e g regularizer l22 tensor ops onesshape5 5 regularizertensor 50 0 developing new regularizers any function that takes in a weight matrix and returns a scalar tensor can be used as a regularizer e g def l1regweightmatrix return 0 01 ops sumops absoluteweightmatrix layer dense5 inputdim5 kernelinitializer ones kernelregularizerl1reg tensor ops onesshape5 5 out layertensor layer losses 0 25 alternatively you can write your custom regularizers in an objectoriented way by extending this regularizer base class e g class l2regularizerregularizer def initself l20 self l2 l2 def callself x return self l2 ops sumops squarex def getconfigself return l2 floatself l2 layer dense 5 inputdim5 kernelinitializer ones kernelregularizerl2regularizerl20 5 tensor ops onesshape5 5 out layertensor layer losses 12 5 a note on serialization and deserialization registering the regularizers as serializable is optional if you are just training and executing models exporting to and from savedmodels or saving and loading weight checkpoints registration is required for saving and loading models to hdf5 format keras model cloning some visualization utilities and exporting models to and from json if using this functionality you must make sure any python process running your model has also defined and registered your custom regularizer compute a regularization penalty from an input tensor return 0 0 classmethod def fromconfigcls config return clsconfig def getconfigself raise notimplementederrorfself does not implement getconfig kerasexportkeras regularizers l1l2 keras regularizers l1l2 class l1l2regularizer def initself l10 0 l20 0 the default value for l1 and l2 are different from the value in l1l2 for backward compatibility reason eg l1l2l20 1 will only have l2 and no l1 penalty l1 0 0 if l1 is none else l1 l2 0 0 if l2 is none else l2 validatefloatargl1 namel1 validatefloatargl2 namel2 self l1 l1 self l2 l2 def callself x regularization ops converttotensor0 0 dtypex dtype if self l1 regularization self l1 ops sumops absolutex if self l2 regularization self l2 ops sumops squarex return regularization def getconfigself return l1 floatself l1 l2 floatself l2 kerasexportkeras regularizers l1 keras regularizers l1 class l1regularizer def initself l10 01 l1 0 01 if l1 is none else l1 validatefloatargl1 namel1 self l1 ops converttotensorl1 def callself x return self l1 ops sumops absolutex def getconfigself return l1 floatself l1 kerasexportkeras regularizers l2 keras regularizers l2 class l2regularizer def initself l20 01 l2 0 01 if l2 is none else l2 validatefloatargl2 namel2 self l2 l2 def callself x return self l2 ops sumops squarex def getconfigself return l2 floatself l2 kerasexport keras regularizers orthogonalregularizer keras regularizers orthogonalregularizer class orthogonalregularizerregularizer def initself factor0 01 moderows validatefloatargfactor namefactor self factor ops converttotensorfactor if mode not in rows columns raise valueerror invalid value for argument mode expected one of f rows columns received modemode self mode mode def callself inputs if leninputs shape 2 raise valueerror inputs to orthogonalregularizer must have rank 2 received finputs shapeinputs shape if self mode rows inputs normalizeinputs axis1 product ops matmulinputs ops transposeinputs size inputs shape0 else inputs normalizeinputs axis0 product ops matmulops transposeinputs inputs size inputs shape1 productnodiagonal product 1 0 ops eyesize dtypeinputs dtype numpairs size size 1 0 2 0 return self factor 0 5 ops sumops absoluteproductnodiagonal numpairs def getconfigself return factor floatself factor mode self mode def validatefloatargvalue name regularizer base class regularizers allow you to apply penalties on layer parameters or layer activity during optimization these penalties are summed into the loss function that the network optimizes regularization penalties are applied on a per layer basis the exact api will depend on the layer but many layers e g dense conv1d conv2d and conv3d have a unified api these layers expose 3 keyword arguments kernel_regularizer regularizer to apply a penalty on the layer s kernel bias_regularizer regularizer to apply a penalty on the layer s bias activity_regularizer regularizer to apply a penalty on the layer s output all layers including custom layers expose activity_regularizer as a settable property whether or not it is in the constructor arguments the value returned by the activity_regularizer is divided by the input batch size so that the relative weighting between the weight regularizers and the activity regularizers does not change with the batch size you can access a layer s regularization penalties by calling layer losses after calling the layer on inputs example layer dense 5 input_dim 5 kernel_initializer ones kernel_regularizer l1 0 01 activity_regularizer l2 0 01 tensor ops ones shape 5 5 2 0 out layer tensor the kernel regularization term is 0 25 the activity regularization term after dividing by the batch size is 5 ops sum layer losses 5 25 available penalties python l1 0 3 l1 regularization penalty l2 0 1 l2 regularization penalty l1l2 l1 0 01 l2 0 01 l1 l2 penalties directly calling a regularizer compute a regularization loss on a tensor by directly calling a regularizer as if it is a one argument function e g regularizer l2 2 tensor ops ones shape 5 5 regularizer tensor 50 0 developing new regularizers any function that takes in a weight matrix and returns a scalar tensor can be used as a regularizer e g def l1_reg weight_matrix return 0 01 ops sum ops absolute weight_matrix layer dense 5 input_dim 5 kernel_initializer ones kernel_regularizer l1_reg tensor ops ones shape 5 5 out layer tensor layer losses 0 25 alternatively you can write your custom regularizers in an object oriented way by extending this regularizer base class e g class l2regularizer regularizer def __init__ self l2 0 self l2 l2 def __call__ self x return self l2 ops sum ops square x def get_config self return l2 float self l2 layer dense 5 input_dim 5 kernel_initializer ones kernel_regularizer l2regularizer l2 0 5 tensor ops ones shape 5 5 out layer tensor layer losses 12 5 a note on serialization and deserialization registering the regularizers as serializable is optional if you are just training and executing models exporting to and from savedmodels or saving and loading weight checkpoints registration is required for saving and loading models to hdf5 format keras model cloning some visualization utilities and exporting models to and from json if using this functionality you must make sure any python process running your model has also defined and registered your custom regularizer compute a regularization penalty from an input tensor creates a regularizer from its config this method is the reverse of get_config capable of instantiating the same regularizer from the config dictionary this method is used by keras model_to_estimator saving and loading models to hdf5 formats keras model cloning some visualization utilities and exporting models to and from json args config a python dictionary typically the output of get_config returns a regularizer instance returns the config of the regularizer an regularizer config is a python dictionary serializable containing all configuration parameters of the regularizer the same regularizer can be reinstantiated later without any saved state from this configuration this method is optional if you are just training and executing models exporting to and from savedmodels or using weight checkpoints this method is required for keras model_to_estimator saving and loading models to hdf5 formats keras model cloning some visualization utilities and exporting models to and from json returns python dictionary a regularizer that applies both l1 and l2 regularization penalties the l1 regularization penalty is computed as loss l1 reduce_sum abs x the l2 regularization penalty is computed as loss l2 reduce_sum square x l1l2 may be passed to a layer as a string identifier dense dense 3 kernel_regularizer l1_l2 in this case the default values used are l1 0 01 and l2 0 01 arguments l1 float l1 regularization factor l2 float l2 regularization factor the default value for l1 and l2 are different from the value in l1_l2 for backward compatibility reason eg l1l2 l2 0 1 will only have l2 and no l1 penalty a regularizer that applies a l1 regularization penalty the l1 regularization penalty is computed as loss l1 reduce_sum abs x l1 may be passed to a layer as a string identifier dense dense 3 kernel_regularizer l1 in this case the default value used is l1 0 01 arguments l1 float l1 regularization factor a regularizer that applies a l2 regularization penalty the l2 regularization penalty is computed as loss l2 reduce_sum square x l2 may be passed to a layer as a string identifier dense dense 3 kernel_regularizer l2 in this case the default value used is l2 0 01 arguments l2 float l2 regularization factor regularizer that encourages input vectors to be orthogonal to each other it can be applied to either the rows of a matrix mode rows or its columns mode columns when applied to a dense kernel of shape input_dim units rows mode will seek to make the feature vectors i e the basis of the output space orthogonal to each other arguments factor float the regularization factor the regularization penalty will be proportional to factor times the mean of the dot products between the l2 normalized rows if mode rows or columns if mode columns of the inputs excluding the product of each row column with itself defaults to 0 01 mode string one of rows columns defaults to rows in rows mode the regularization effect seeks to make the rows of the input orthogonal to each other in columns mode it seeks to make the columns of the input orthogonal to each other example regularizer orthogonalregularizer factor 0 01 layer dense units 4 kernel_regularizer regularizer check penalty number availability raise valueerror if failed
import math

from keras import ops
from keras.api_export import keras_export
from keras.utils.numerical_utils import normalize


@keras_export(["keras.Regularizer", "keras.regularizers.Regularizer"])
class Regularizer:
    

    def __call__(self, x):
        
        return 0.0

    @classmethod
    def from_config(cls, config):
        
        return cls(**config)

    def get_config(self):
        
        raise NotImplementedError(f"{self} does not implement get_config()")


@keras_export(["keras.regularizers.L1L2", "keras.regularizers.l1_l2"])
class L1L2(Regularizer):
    

    def __init__(self, l1=0.0, l2=0.0):
        
        
        
        l1 = 0.0 if l1 is None else l1
        l2 = 0.0 if l2 is None else l2
        validate_float_arg(l1, name="l1")
        validate_float_arg(l2, name="l2")

        self.l1 = l1
        self.l2 = l2

    def __call__(self, x):
        regularization = ops.convert_to_tensor(0.0, dtype=x.dtype)
        if self.l1:
            regularization += self.l1 * ops.sum(ops.absolute(x))
        if self.l2:
            regularization += self.l2 * ops.sum(ops.square(x))
        return regularization

    def get_config(self):
        return {"l1": float(self.l1), "l2": float(self.l2)}


@keras_export(["keras.regularizers.L1", "keras.regularizers.l1"])
class L1(Regularizer):
    

    def __init__(self, l1=0.01):
        l1 = 0.01 if l1 is None else l1
        validate_float_arg(l1, name="l1")
        self.l1 = ops.convert_to_tensor(l1)

    def __call__(self, x):
        return self.l1 * ops.sum(ops.absolute(x))

    def get_config(self):
        return {"l1": float(self.l1)}


@keras_export(["keras.regularizers.L2", "keras.regularizers.l2"])
class L2(Regularizer):
    

    def __init__(self, l2=0.01):
        l2 = 0.01 if l2 is None else l2
        validate_float_arg(l2, name="l2")
        self.l2 = l2

    def __call__(self, x):
        return self.l2 * ops.sum(ops.square(x))

    def get_config(self):
        return {"l2": float(self.l2)}


@keras_export(
    [
        "keras.regularizers.OrthogonalRegularizer",
        "keras.regularizers.orthogonal_regularizer",
    ]
)
class OrthogonalRegularizer(Regularizer):
    

    def __init__(self, factor=0.01, mode="rows"):
        validate_float_arg(factor, name="factor")
        self.factor = ops.convert_to_tensor(factor)
        if mode not in {"rows", "columns"}:
            raise ValueError(
                "Invalid value for argument `mode`. Expected one of "
                f'{{"rows", "columns"}}. Received: mode={mode}'
            )
        self.mode = mode

    def __call__(self, inputs):
        if len(inputs.shape) != 2:
            raise ValueError(
                "Inputs to OrthogonalRegularizer must have rank 2. Received: "
                f"inputs.shape={inputs.shape}"
            )
        if self.mode == "rows":
            inputs = normalize(inputs, axis=1)
            product = ops.matmul(inputs, ops.transpose(inputs))
            size = inputs.shape[0]
        else:
            inputs = normalize(inputs, axis=0)
            product = ops.matmul(ops.transpose(inputs), inputs)
            size = inputs.shape[1]
        product_no_diagonal = product * (
            1.0 - ops.eye(size, dtype=inputs.dtype)
        )
        num_pairs = size * (size - 1.0) / 2.0
        return (
            self.factor
            * 0.5
            * ops.sum(ops.absolute(product_no_diagonal))
            / num_pairs
        )

    def get_config(self):
        return {"factor": float(self.factor), "mode": self.mode}


def validate_float_arg(value, name):
    
    if (
        not isinstance(value, (float, int))
        or (math.isinf(value) or math.isnan(value))
        or value < 0
    ):
        raise ValueError(
            f"Invalid value for argument {name}: expected a non-negative float."
            f"Received: {name}={value}"
        )
    return float(value)