a preprocessing layer which hashes and bins categorical features this layer transforms categorical inputs to hashed output it elementwise converts a ints or strings to ints in a fixed range the stable hash function uses tensorflow ops fingerprint to produce the same output consistently across all platforms this layer uses farmhash64https github comgooglefarmhash by default which provides a consistent hashed output across different platforms and is stable across invocations regardless of device and context by mixing the input bits thoroughly if you want to obfuscate the hashed output you can also pass a random salt argument in the constructor in that case the layer will use the siphash64https github comgooglehighwayhash hash function with the salt value serving as additional input to the hash function note this layer internally uses tensorflow it cannot be used as part of the compiled computation graph of a model with any backend other than tensorflow it can however be used with any backend when running eagerly it can also always be used as part of an input preprocessing pipeline with any backend outside the model itself which is how we recommend to use this layer note this layer is safe to use inside a tf data pipeline independently of which backend you re using example farmhash64 layer keras layers hashingnumbins3 inp a b c d e layerinp array1 0 1 1 2 example farmhash64 with a mask value layer keras layers hashingnumbins3 maskvalue inp a b c d layerinp array1 1 0 2 2 example siphash64 layer keras layers hashingnumbins3 salt133 137 inp a b c d e layerinp array1 2 1 0 2 example siphash64 with a single integer same as salt133 133 layer keras layers hashingnumbins3 salt133 inp a b c d e layerinp array0 0 2 1 0 args numbins number of hash bins note that this includes the maskvalue bin so the effective number of bins is numbins 1 if maskvalue is set maskvalue a value that represents masked inputs which are mapped to index 0 none means no mask term will be added and the hashing will start at index 0 defaults to none salt a single unsigned integer or none if passed the hash function used will be siphash64 with these values used as an additional input known as a salt in cryptography these should be nonzero if none uses the farmhash64 hash function it also supports tuplelist of 2 unsigned integer numbers see reference paper for details defaults to none outputmode specification for the output of the layer values can be int onehot multihot or count configuring the layer as follows int return the integer bin indices directly onehot encodes each individual element in the input into an array the same size as numbins containing a 1 at the input s bin index if the last dimension is size 1 will encode on that dimension if the last dimension is not size 1 will append a new dimension for the encoded output multihot encodes each sample in the input into a single array the same size as numbins containing a 1 for each bin index index present in the sample treats the last dimension as the sample dimension if input shape is samplelength output shape will be numtokens count as multihot but the int array contains a count of the number of times the bin index appeared in the sample defaults to int sparse boolean only applicable to onehot multihot and count output modes only supported with tensorflow backend if true returns a sparsetensor instead of a dense tensor defaults to false kwargs keyword arguments to construct a layer input shape a single string a list of strings or an int32 or int64 tensor of shape batchsize output shape an int32 tensor of shape batchsize reference siphash with salthttps www 131002 netsiphashsiphash pdf by default output int32 when outputmode int and floats otherwise outputmode must be one of int onehot multihot count converts a nonsparse tensor of values to bin indices hashbins self numbins mask none if maskvalue is set the zeroth bin is reserved for it if self maskvalue is not none and hashbins 1 hashbins 1 mask tf equalvalues self maskvalue convert all values to strings before hashing floats are first normalized to int64 if values dtype isfloating values tf castvalues dtypeint64 if values dtype tf string values tf asstringvalues hash the strings if self stronghash values tf strings tohashbucketstrong values hashbins namehash keyself salt else values tf strings tohashbucketfast values hashbins namehash if mask is not none values tf addvalues tf oneslikevalues values tf wheremask tf zeroslikevalues values return values def computeoutputspecself inputs if self outputmode int return backend kerastensorshapeinputs shape dtypeself dtype if leninputs shape 1 baseshape tupleinputs shape 1 else baseshape return backend kerastensor shapebaseshape self numbins dtypeself dtype def getconfigself config super getconfig config update numbins self numbins salt self salt maskvalue self maskvalue outputmode self outputmode sparse self sparse return config a preprocessing layer which hashes and bins categorical features this layer transforms categorical inputs to hashed output it element wise converts a ints or strings to ints in a fixed range the stable hash function uses tensorflow ops fingerprint to produce the same output consistently across all platforms this layer uses farmhash64 https github com google farmhash by default which provides a consistent hashed output across different platforms and is stable across invocations regardless of device and context by mixing the input bits thoroughly if you want to obfuscate the hashed output you can also pass a random salt argument in the constructor in that case the layer will use the siphash64 https github com google highwayhash hash function with the salt value serving as additional input to the hash function note this layer internally uses tensorflow it cannot be used as part of the compiled computation graph of a model with any backend other than tensorflow it can however be used with any backend when running eagerly it can also always be used as part of an input preprocessing pipeline with any backend outside the model itself which is how we recommend to use this layer note this layer is safe to use inside a tf data pipeline independently of which backend you re using example farmhash64 layer keras layers hashing num_bins 3 inp a b c d e layer inp array 1 0 1 1 2 example farmhash64 with a mask value layer keras layers hashing num_bins 3 mask_value inp a b c d layer inp array 1 1 0 2 2 example siphash64 layer keras layers hashing num_bins 3 salt 133 137 inp a b c d e layer inp array 1 2 1 0 2 example siphash64 with a single integer same as salt 133 133 layer keras layers hashing num_bins 3 salt 133 inp a b c d e layer inp array 0 0 2 1 0 args num_bins number of hash bins note that this includes the mask_value bin so the effective number of bins is num_bins 1 if mask_value is set mask_value a value that represents masked inputs which are mapped to index 0 none means no mask term will be added and the hashing will start at index 0 defaults to none salt a single unsigned integer or none if passed the hash function used will be siphash64 with these values used as an additional input known as a salt in cryptography these should be non zero if none uses the farmhash64 hash function it also supports tuple list of 2 unsigned integer numbers see reference paper for details defaults to none output_mode specification for the output of the layer values can be int one_hot multi_hot or count configuring the layer as follows int return the integer bin indices directly one_hot encodes each individual element in the input into an array the same size as num_bins containing a 1 at the input s bin index if the last dimension is size 1 will encode on that dimension if the last dimension is not size 1 will append a new dimension for the encoded output multi_hot encodes each sample in the input into a single array the same size as num_bins containing a 1 for each bin index index present in the sample treats the last dimension as the sample dimension if input shape is sample_length output shape will be num_tokens count as multi_hot but the int array contains a count of the number of times the bin index appeared in the sample defaults to int sparse boolean only applicable to one_hot multi_hot and count output modes only supported with tensorflow backend if true returns a sparsetensor instead of a dense tensor defaults to false kwargs keyword arguments to construct a layer input shape a single string a list of strings or an int32 or int64 tensor of shape batch_size output shape an int32 tensor of shape batch_size reference siphash with salt https www 131002 net siphash siphash pdf by default output int32 when output_mode int and floats otherwise output_mode must be one of int one_hot multi_hot count converts a non sparse tensor of values to bin indices if mask_value is set the zeroth bin is reserved for it convert all values to strings before hashing floats are first normalized to int64 hash the strings
from keras import backend
from keras.api_export import keras_export
from keras.layers.layer import Layer
from keras.utils import backend_utils
from keras.utils import tf_utils
from keras.utils.module_utils import tensorflow as tf


@keras_export("keras.layers.Hashing")
class Hashing(Layer):
    

    def __init__(
        self,
        num_bins,
        mask_value=None,
        salt=None,
        output_mode="int",
        sparse=False,
        **kwargs,
    ):
        if not tf.available:
            raise ImportError(
                "Layer Hashing requires TensorFlow. "
                "Install it via `pip install tensorflow`."
            )

        
        if "dtype" not in kwargs or kwargs["dtype"] is None:
            kwargs["dtype"] = (
                "int64" if output_mode == "int" else backend.floatx()
            )

        super().__init__(**kwargs)

        if num_bins is None or num_bins <= 0:
            raise ValueError(
                "The `num_bins` for `Hashing` cannot be `None` or "
                f"non-positive values. Received: num_bins={num_bins}."
            )

        if output_mode == "int" and not kwargs["dtype"] in ("int32", "int64"):
            raise ValueError(
                'When `output_mode="int"`, `dtype` should be an integer '
                f"type, 'int32' or 'in64'. Received: dtype={kwargs['dtype']}"
            )

        
        accepted_output_modes = ("int", "one_hot", "multi_hot", "count")
        if output_mode not in accepted_output_modes:
            raise ValueError(
                "Invalid value for argument `output_mode`. "
                f"Expected one of {accepted_output_modes}. "
                f"Received: output_mode={output_mode}"
            )

        if sparse and output_mode == "int":
            raise ValueError(
                "`sparse` may only be true if `output_mode` is "
                '`"one_hot"`, `"multi_hot"`, or `"count"`. '
                f"Received: sparse={sparse} and "
                f"output_mode={output_mode}"
            )

        self.num_bins = num_bins
        self.mask_value = mask_value
        self.strong_hash = True if salt is not None else False
        self.output_mode = output_mode
        self.sparse = sparse
        self.salt = None
        if salt is not None:
            if isinstance(salt, (tuple, list)) and len(salt) == 2:
                self.salt = list(salt)
            elif isinstance(salt, int):
                self.salt = [salt, salt]
            else:
                raise ValueError(
                    "The `salt` argument for `Hashing` can only be a tuple of "
                    "size 2 integers, or a single integer. "
                    f"Received: salt={salt}."
                )
        self._convert_input_args = False
        self._allow_non_tensor_positional_args = True
        self.supports_jit = False

    def call(self, inputs):
        if not isinstance(
            inputs, (tf.Tensor, tf.SparseTensor, tf.RaggedTensor)
        ):
            inputs = tf.convert_to_tensor(backend.convert_to_numpy(inputs))
        if isinstance(inputs, tf.SparseTensor):
            indices = tf.SparseTensor(
                indices=inputs.indices,
                values=self._hash_values_to_bins(inputs.values),
                dense_shape=inputs.dense_shape,
            )
        else:
            indices = self._hash_values_to_bins(inputs)
        outputs = tf_utils.encode_categorical_inputs(
            indices,
            output_mode=self.output_mode,
            depth=self.num_bins,
            sparse=self.sparse,
            dtype=self.dtype,
        )
        if (
            backend.backend() != "tensorflow"
            and not backend_utils.in_tf_graph()
        ):
            outputs = backend.convert_to_tensor(outputs)
        return outputs

    def _hash_values_to_bins(self, values):
        
        hash_bins = self.num_bins
        mask = None
        
        if self.mask_value is not None and hash_bins > 1:
            hash_bins -= 1
            mask = tf.equal(values, self.mask_value)
        
        
        if values.dtype.is_floating:
            values = tf.cast(values, dtype="int64")
        if values.dtype != tf.string:
            values = tf.as_string(values)
        
        if self.strong_hash:
            values = tf.strings.to_hash_bucket_strong(
                values, hash_bins, name="hash", key=self.salt
            )
        else:
            values = tf.strings.to_hash_bucket_fast(
                values, hash_bins, name="hash"
            )
        if mask is not None:
            values = tf.add(values, tf.ones_like(values))
            values = tf.where(mask, tf.zeros_like(values), values)
        return values

    def compute_output_spec(self, inputs):
        if self.output_mode == "int":
            return backend.KerasTensor(shape=inputs.shape, dtype=self.dtype)
        if len(inputs.shape) >= 1:
            base_shape = tuple(inputs.shape)[:-1]
        else:
            base_shape = ()
        return backend.KerasTensor(
            shape=base_shape + (self.num_bins,), dtype=self.dtype
        )

    def get_config(self):
        config = super().get_config()
        config.update(
            {
                "num_bins": self.num_bins,
                "salt": self.salt,
                "mask_value": self.mask_value,
                "output_mode": self.output_mode,
                "sparse": self.sparse,
            }
        )
        return config