a preprocessing layer that maps strings to possibly encoded indices this layer translates a set of arbitrary strings into integer output via a tablebased vocabulary lookup this layer will perform no splitting or transformation of input strings for a layer than can split and tokenize natural language see the keras layers textvectorization layer the vocabulary for the layer must be either supplied on construction or learned via adapt during adapt the layer will analyze a data set determine the frequency of individual strings tokens and create a vocabulary from them if the vocabulary is capped in size the most frequent tokens will be used to create the vocabulary and all others will be treated as outofvocabulary oov there are two possible output modes for the layer when outputmode is int input strings are converted to their index in the vocabulary an integer when outputmode is multihot count or tfidf input strings are encoded into an array where each dimension corresponds to an element in the vocabulary the vocabulary can optionally contain a mask token as well as an oov token which can optionally occupy multiple indices in the vocabulary as set by numoovindices the position of these tokens in the vocabulary is fixed when outputmode is int the vocabulary will begin with the mask token if set followed by oov indices followed by the rest of the vocabulary when outputmode is multihot count or tfidf the vocabulary will begin with oov indices and instances of the mask token will be dropped note this layer uses tensorflow internally it cannot be used as part of the compiled computation graph of a model with any backend other than tensorflow it can however be used with any backend when running eagerly it can also always be used as part of an input preprocessing pipeline with any backend outside the model itself which is how we recommend to use this layer note this layer is safe to use inside a tf data pipeline independently of which backend you re using args maxtokens maximum size of the vocabulary for this layer this should only be specified when adapting the vocabulary or when setting padtomaxtokenstrue if none there is no cap on the size of the vocabulary note that this size includes the oov and mask tokens defaults to none numoovindices the number of outofvocabulary tokens to use if this value is more than 1 oov inputs are modulated to determine their oov value if this value is 0 oov inputs will cause an error when calling the layer defaults to 1 masktoken a token that represents masked inputs when outputmode is int the token is included in vocabulary and mapped to index 0 in other output modes the token will not appear in the vocabulary and instances of the mask token in the input will be dropped if set to none no mask term will be added defaults to none oovtoken only used when invert is true the token to return for oov indices defaults to unk vocabulary optional either an array of integers or a string path to a text file if passing an array can pass a tuple list 1d numpy array or 1d tensor containing the integer vocbulary terms if passing a file path the file should contain one line per term in the vocabulary if this argument is set there is no need to adapt the layer vocabularydtype the dtype of the vocabulary terms for example int64 or int32 defaults to int64 idfweights only valid when outputmode is tfidf a tuple list 1d numpy array or 1d tensor or the same length as the vocabulary containing the floating point inverse document frequency weights which will be multiplied by per sample term counts for the final tfidf weight if the vocabulary argument is set and outputmode is tfidf this argument must be supplied invert only valid when outputmode is int if true this layer will map indices to vocabulary items instead of mapping vocabulary items to indices defaults to false outputmode specification for the output of the layer values can be int onehot multihot count or tfidf configuring the layer as follows int return the vocabulary indices of the input tokens onehot encodes each individual element in the input into an array the same size as the vocabulary containing a 1 at the element index if the last dimension is size 1 will encode on that dimension if the last dimension is not size 1 will append a new dimension for the encoded output multihot encodes each sample in the input into a single array the same size as the vocabulary containing a 1 for each vocabulary term present in the sample treats the last dimension as the sample dimension if input shape is samplelength output shape will be numtokens count as multihot but the int array contains a count of the number of times the token at that index appeared in the sample tfidf as multihot but the tfidf algorithm is applied to find the value in each token slot for int output any shape of input and output is supported for all other output modes currently only output up to rank 2 is supported defaults to int padtomaxtokens only applicable when outputmode is multihot count or tfidf if true the output will have its feature axis padded to maxtokens even if the number of unique tokens in the vocabulary is less than maxtokens resulting in a tensor of shape batchsize maxtokens regardless of vocabulary size defaults to false sparse boolean only applicable to multihot count and tfidf output modes only supported with tensorflow backend if true returns a sparsetensor instead of a dense tensor defaults to false encoding optional the text encoding to use to interpret the input strings defaults to utf8 examples creating a lookup layer with a known vocabulary this example creates a lookup layer with a preexisting vocabulary vocab a b c d data a c d d z b layer stringlookupvocabularyvocab layerdata array1 3 4 4 0 2 creating a lookup layer with an adapted vocabulary this example creates a lookup layer and generates the vocabulary by analyzing the dataset data a c d d z b layer stringlookup layer adaptdata layer getvocabulary unk d z c b a note that the oov token unk has been added to the vocabulary the remaining tokens are sorted by frequency d which has 2 occurrences is first then by inverse sort order data a c d d z b layer stringlookup layer adaptdata layerdata array5 3 1 1 2 4 lookups with multiple oov indices this example demonstrates how to use a lookup layer with multiple oov indices when a layer is created with more than one oov index any oov values are hashed into the number of oov buckets distributing oov values in a deterministic fashion across the set vocab a b c d data a c d m z b layer stringlookupvocabularyvocab numoovindices2 layerdata array2 4 5 0 1 3 note that the output for oov value m is 0 while the output for oov value z is 1 the invocab terms have their output index increased by 1 from earlier examples a maps to 2 etc in order to make space for the extra oov value onehot output configure the layer with outputmode onehot note that the first numoovindices dimensions in the onthot encoding represent oov values vocab a b c d data a b c d z layer stringlookupvocabularyvocab outputmode onehot layerdata array0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 dtypefloat32 multihot output configure the layer with outputmode multihot note that the first numoovindices dimensions in the multihot encoding represent oov values vocab a b c d data a c d d d z b z layer stringlookupvocabularyvocab outputmode multihot layerdata array0 1 0 1 1 1 0 1 0 1 dtypefloat32 token count output configure the layer with outputmode count as with multihot output the first numoovindices dimensions in the output represent oov values vocab a b c d data a c d d d z b z layer stringlookupvocabularyvocab outputmode count layerdata array0 1 0 1 2 2 0 1 0 1 dtypefloat32 tfidf output configure the layer with outputmodetfidf as with multihot output the first numoovindices dimensions in the output represent oov values each token bin will output tokencount idfweight where the idf weights are the inverse document frequency weights per token these should be provided along with the vocabulary note that the idfweight for oov values will default to the average of all idf weights passed in vocab a b c d idfweights 0 25 0 75 0 6 0 4 data a c d d d z b z layer stringlookupoutputmodetfidf layer setvocabularyvocab idfweightsidfweights layerdata array0 0 25 0 0 6 0 8 1 0 0 0 75 0 0 4 dtypefloat32 to specify the idf weights for oov values you will need to pass the entire vocabularly including the leading oov token vocab unk a b c d idfweights 0 9 0 25 0 75 0 6 0 4 data a c d d d z b z layer stringlookupoutputmodetfidf layer setvocabularyvocab idfweightsidfweights layerdata array0 0 25 0 0 6 0 8 1 8 0 0 75 0 0 4 dtypefloat32 when adapting the layer in tfidf mode each input sample will be considered a document and idf weight per token will be calculated as log1 numdocuments 1 tokendocumentcount inverse lookup this example demonstrates how to map indices to strings using this layer you can also use adapt with inversetrue but for simplicity we ll pass the vocab in this example vocab a b c d data 1 3 4 4 0 2 layer stringlookupvocabularyvocab inverttrue layerdata arrayb a b c b d b d b unk b b dtypeobject note that the first index correspond to the oov token by default forward and inverse lookup pairs this example demonstrates how to use the vocabulary of a standard lookup layer to create an inverse lookup layer vocab a b c d data a c d d z b layer stringlookupvocabularyvocab ilayer stringlookupvocabularyvocab inverttrue intdata layerdata ilayerintdata arrayb a b c b d b d b unk b b dtypeobject in this example the input value z resulted in an output of unk since 1000 was not in the vocabulary it got represented as an oov and all oov values are returned as unk in the inverse layer also note that for the inverse to work you must have already set the forward layer vocabulary either directly or via adapt before calling getvocabulary computes a vocabulary of interger terms from tokens in a dataset calling adapt on a stringlookup layer is an alternative to passing in a precomputed vocabulary on construction via the vocabulary argument a stringlookup layer should always be either adapted over a dataset or supplied with a vocabulary during adapt the layer will build a vocabulary of all string tokens seen in the dataset sorted by occurrence count with ties broken by sort order of the tokens high to low at the end of adapt if maxtokens is set the vocabulary wil be truncated to maxtokens size for example adapting a layer with maxtokens1000 will compute the 1000 most frequent tokens occurring in the input dataset if outputmode tfidf adapt will also learn the document frequencies of each token in the input dataset arguments data the data to train on it can be passed either as a batched tf data dataset as a list of strings or as a numpy array steps integer or none total number of steps batches of samples to process if data is a tf data dataset and steps is none adapt will run until the input dataset is exhausted when passing an infinitely repeating dataset you must specify the steps argument this argument is not supported with array inputs or list inputs overridden methods from indexlookup there is only one valid dtype for strings so we don t expose this a preprocessing layer that maps strings to possibly encoded indices this layer translates a set of arbitrary strings into integer output via a table based vocabulary lookup this layer will perform no splitting or transformation of input strings for a layer than can split and tokenize natural language see the keras layers textvectorization layer the vocabulary for the layer must be either supplied on construction or learned via adapt during adapt the layer will analyze a data set determine the frequency of individual strings tokens and create a vocabulary from them if the vocabulary is capped in size the most frequent tokens will be used to create the vocabulary and all others will be treated as out of vocabulary oov there are two possible output modes for the layer when output_mode is int input strings are converted to their index in the vocabulary an integer when output_mode is multi_hot count or tf_idf input strings are encoded into an array where each dimension corresponds to an element in the vocabulary the vocabulary can optionally contain a mask token as well as an oov token which can optionally occupy multiple indices in the vocabulary as set by num_oov_indices the position of these tokens in the vocabulary is fixed when output_mode is int the vocabulary will begin with the mask token if set followed by oov indices followed by the rest of the vocabulary when output_mode is multi_hot count or tf_idf the vocabulary will begin with oov indices and instances of the mask token will be dropped note this layer uses tensorflow internally it cannot be used as part of the compiled computation graph of a model with any backend other than tensorflow it can however be used with any backend when running eagerly it can also always be used as part of an input preprocessing pipeline with any backend outside the model itself which is how we recommend to use this layer note this layer is safe to use inside a tf data pipeline independently of which backend you re using args max_tokens maximum size of the vocabulary for this layer this should only be specified when adapting the vocabulary or when setting pad_to_max_tokens true if none there is no cap on the size of the vocabulary note that this size includes the oov and mask tokens defaults to none num_oov_indices the number of out of vocabulary tokens to use if this value is more than 1 oov inputs are modulated to determine their oov value if this value is 0 oov inputs will cause an error when calling the layer defaults to 1 mask_token a token that represents masked inputs when output_mode is int the token is included in vocabulary and mapped to index 0 in other output modes the token will not appear in the vocabulary and instances of the mask token in the input will be dropped if set to none no mask term will be added defaults to none oov_token only used when invert is true the token to return for oov indices defaults to unk vocabulary optional either an array of integers or a string path to a text file if passing an array can pass a tuple list 1d numpy array or 1d tensor containing the integer vocbulary terms if passing a file path the file should contain one line per term in the vocabulary if this argument is set there is no need to adapt the layer vocabulary_dtype the dtype of the vocabulary terms for example int64 or int32 defaults to int64 idf_weights only valid when output_mode is tf_idf a tuple list 1d numpy array or 1d tensor or the same length as the vocabulary containing the floating point inverse document frequency weights which will be multiplied by per sample term counts for the final tf idf weight if the vocabulary argument is set and output_mode is tf_idf this argument must be supplied invert only valid when output_mode is int if true this layer will map indices to vocabulary items instead of mapping vocabulary items to indices defaults to false output_mode specification for the output of the layer values can be int one_hot multi_hot count or tf_idf configuring the layer as follows int return the vocabulary indices of the input tokens one_hot encodes each individual element in the input into an array the same size as the vocabulary containing a 1 at the element index if the last dimension is size 1 will encode on that dimension if the last dimension is not size 1 will append a new dimension for the encoded output multi_hot encodes each sample in the input into a single array the same size as the vocabulary containing a 1 for each vocabulary term present in the sample treats the last dimension as the sample dimension if input shape is sample_length output shape will be num_tokens count as multi_hot but the int array contains a count of the number of times the token at that index appeared in the sample tf_idf as multi_hot but the tf idf algorithm is applied to find the value in each token slot for int output any shape of input and output is supported for all other output modes currently only output up to rank 2 is supported defaults to int pad_to_max_tokens only applicable when output_mode is multi_hot count or tf_idf if true the output will have its feature axis padded to max_tokens even if the number of unique tokens in the vocabulary is less than max_tokens resulting in a tensor of shape batch_size max_tokens regardless of vocabulary size defaults to false sparse boolean only applicable to multi_hot count and tf_idf output modes only supported with tensorflow backend if true returns a sparsetensor instead of a dense tensor defaults to false encoding optional the text encoding to use to interpret the input strings defaults to utf 8 examples creating a lookup layer with a known vocabulary this example creates a lookup layer with a pre existing vocabulary vocab a b c d data a c d d z b layer stringlookup vocabulary vocab layer data array 1 3 4 4 0 2 creating a lookup layer with an adapted vocabulary this example creates a lookup layer and generates the vocabulary by analyzing the dataset data a c d d z b layer stringlookup layer adapt data layer get_vocabulary unk d z c b a note that the oov token unk has been added to the vocabulary the remaining tokens are sorted by frequency d which has 2 occurrences is first then by inverse sort order data a c d d z b layer stringlookup layer adapt data layer data array 5 3 1 1 2 4 lookups with multiple oov indices this example demonstrates how to use a lookup layer with multiple oov indices when a layer is created with more than one oov index any oov values are hashed into the number of oov buckets distributing oov values in a deterministic fashion across the set vocab a b c d data a c d m z b layer stringlookup vocabulary vocab num_oov_indices 2 layer data array 2 4 5 0 1 3 note that the output for oov value m is 0 while the output for oov value z is 1 the in vocab terms have their output index increased by 1 from earlier examples a maps to 2 etc in order to make space for the extra oov value one hot output configure the layer with output_mode one_hot note that the first num_oov_indices dimensions in the ont_hot encoding represent oov values vocab a b c d data a b c d z layer stringlookup vocabulary vocab output_mode one_hot layer data array 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 dtype float32 multi hot output configure the layer with output_mode multi_hot note that the first num_oov_indices dimensions in the multi_hot encoding represent oov values vocab a b c d data a c d d d z b z layer stringlookup vocabulary vocab output_mode multi_hot layer data array 0 1 0 1 1 1 0 1 0 1 dtype float32 token count output configure the layer with output_mode count as with multi_hot output the first num_oov_indices dimensions in the output represent oov values vocab a b c d data a c d d d z b z layer stringlookup vocabulary vocab output_mode count layer data array 0 1 0 1 2 2 0 1 0 1 dtype float32 tf idf output configure the layer with output_mode tf_idf as with multi_hot output the first num_oov_indices dimensions in the output represent oov values each token bin will output token_count idf_weight where the idf weights are the inverse document frequency weights per token these should be provided along with the vocabulary note that the idf_weight for oov values will default to the average of all idf weights passed in vocab a b c d idf_weights 0 25 0 75 0 6 0 4 data a c d d d z b z layer stringlookup output_mode tf_idf layer set_vocabulary vocab idf_weights idf_weights layer data array 0 0 25 0 0 6 0 8 1 0 0 0 75 0 0 4 dtype float32 to specify the idf weights for oov values you will need to pass the entire vocabularly including the leading oov token vocab unk a b c d idf_weights 0 9 0 25 0 75 0 6 0 4 data a c d d d z b z layer stringlookup output_mode tf_idf layer set_vocabulary vocab idf_weights idf_weights layer data array 0 0 25 0 0 6 0 8 1 8 0 0 75 0 0 4 dtype float32 when adapting the layer in tf_idf mode each input sample will be considered a document and idf weight per token will be calculated as log 1 num_documents 1 token_document_count inverse lookup this example demonstrates how to map indices to strings using this layer you can also use adapt with inverse true but for simplicity we ll pass the vocab in this example vocab a b c d data 1 3 4 4 0 2 layer stringlookup vocabulary vocab invert true layer data array b a b c b d b d b unk b b dtype object note that the first index correspond to the oov token by default forward and inverse lookup pairs this example demonstrates how to use the vocabulary of a standard lookup layer to create an inverse lookup layer vocab a b c d data a c d d z b layer stringlookup vocabulary vocab i_layer stringlookup vocabulary vocab invert true int_data layer data i_layer int_data array b a b c b d b d b unk b b dtype object in this example the input value z resulted in an output of unk since 1000 was not in the vocabulary it got represented as an oov and all oov values are returned as unk in the inverse layer also note that for the inverse to work you must have already set the forward layer vocabulary either directly or via adapt before calling get_vocabulary computes a vocabulary of interger terms from tokens in a dataset calling adapt on a stringlookup layer is an alternative to passing in a precomputed vocabulary on construction via the vocabulary argument a stringlookup layer should always be either adapted over a dataset or supplied with a vocabulary during adapt the layer will build a vocabulary of all string tokens seen in the dataset sorted by occurrence count with ties broken by sort order of the tokens high to low at the end of adapt if max_tokens is set the vocabulary wil be truncated to max_tokens size for example adapting a layer with max_tokens 1000 will compute the 1000 most frequent tokens occurring in the input dataset if output_mode tf idf adapt will also learn the document frequencies of each token in the input dataset arguments data the data to train on it can be passed either as a batched tf data dataset as a list of strings or as a numpy array steps integer or none total number of steps batches of samples to process if data is a tf data dataset and steps is none adapt will run until the input dataset is exhausted when passing an infinitely repeating dataset you must specify the steps argument this argument is not supported with array inputs or list inputs overridden methods from indexlookup there is only one valid dtype for strings so we don t expose this
import numpy as np

from keras import backend
from keras.api_export import keras_export
from keras.layers.preprocessing.index_lookup import IndexLookup
from keras.utils import backend_utils
from keras.utils.module_utils import tensorflow as tf


@keras_export("keras.layers.StringLookup")
class StringLookup(IndexLookup):
    

    def __init__(
        self,
        max_tokens=None,
        num_oov_indices=1,
        mask_token=None,
        oov_token="[UNK]",
        vocabulary=None,
        idf_weights=None,
        invert=False,
        output_mode="int",
        pad_to_max_tokens=False,
        sparse=False,
        encoding="utf-8",
        name=None,
        **kwargs,
    ):
        if not tf.available:
            raise ImportError(
                "Layer StringLookup requires TensorFlow. "
                "Install it via `pip install tensorflow`."
            )
        if sparse and backend.backend() != "tensorflow":
            raise ValueError(
                "`sparse` can only be set to True with the "
                "TensorFlow backend."
            )
        super().__init__(
            max_tokens=max_tokens,
            num_oov_indices=num_oov_indices,
            mask_token=mask_token,
            oov_token=oov_token,
            vocabulary=vocabulary,
            idf_weights=idf_weights,
            invert=invert,
            output_mode=output_mode,
            pad_to_max_tokens=pad_to_max_tokens,
            sparse=sparse,
            name=name,
            vocabulary_dtype="string",
            **kwargs,
        )
        self.encoding = encoding
        self._convert_input_args = False
        self._allow_non_tensor_positional_args = True
        self.supports_jit = False

    def adapt(self, data, steps=None):
        
        super().adapt(data, steps=steps)

    
    def _tensor_vocab_to_numpy(self, vocabulary):
        vocabulary = vocabulary.numpy()
        return np.array(
            [tf.compat.as_text(x, self.encoding) for x in vocabulary]
        )

    def get_config(self):
        config = {"encoding": self.encoding}
        base_config = super().get_config()
        
        del base_config["vocabulary_dtype"]
        return {**base_config, **config}

    def call(self, inputs):
        if isinstance(inputs, (tf.Tensor, tf.RaggedTensor)):
            tf_inputs = True
        else:
            tf_inputs = False
            if not isinstance(inputs, (np.ndarray, list, tuple)):
                inputs = tf.convert_to_tensor(backend.convert_to_numpy(inputs))
        outputs = super().call(inputs)
        if (
            not tf_inputs
            and backend.backend() != "tensorflow"
            and not backend_utils.in_tf_graph()
        ):
            outputs = backend.convert_to_tensor(outputs)
        return outputs