codingutf8 2021 iz beltagy matthew e peters arman cohan and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tokenization classes for led import json from typing import dict list optional tuple union from tokenizers import pretokenizers processors from tokenizationutilsbase import addedtoken batchencoding encodedinput from tokenizationutilsfast import pretrainedtokenizerfast from utils import paddingstrategy logging from tokenizationled import ledtokenizer logger logging getloggername vocabfilesnames vocabfile vocab json mergesfile merges txt tokenizerfile tokenizer json pretrainedvocabfilesmap vocabfile allenailedbase16384 https huggingface coallenailedbase16384resolvemainvocab json mergesfile allenailedbase16384 https huggingface coallenailedbase16384resolvemainmerges txt tokenizerfile allenailedbase16384 https huggingface coallenailedbase16384resolvemaintokenizer json pretrainedpositionalembeddingssizes allenailedbase16384 16384 class ledtokenizerfastpretrainedtokenizerfast r construct a fast led tokenizer backed by huggingface s tokenizers library derived from the gpt2 tokenizer using bytelevel bytepairencoding this tokenizer has been trained to treat spaces like parts of the tokens a bit like sentencepiece so a word will be encoded differently whether it is at the beginning of the sentence without space or not python from transformers import ledtokenizerfast tokenizer ledtokenizerfast frompretrainedallenailedbase16384 tokenizerhello worldinputids 0 31414 232 2 tokenizer hello worldinputids 0 20920 232 2 you can get around that behavior by passing addprefixspacetrue when instantiating this tokenizer or when you call it on some text but since the model was not pretrained this way it might yield a decrease in performance tip when used with issplitintowordstrue this tokenizer needs to be instantiated with addprefixspacetrue tip this tokenizer inherits from pretrainedtokenizerfast which contains most of the main methods users should refer to this superclass for more information regarding those methods args vocabfile str path to the vocabulary file mergesfile str path to the merges file errors str optional defaults to replace paradigm to follow when decoding bytes to utf8 see bytes decodehttps docs python org3librarystdtypes htmlbytes decode for more information bostoken str optional defaults to s the beginning of sequence token that was used during pretraining can be used a sequence classifier token tip when building a sequence using special tokens this is not the token that is used for the beginning of sequence the token used is the clstoken tip eostoken str optional defaults to s the end of sequence token tip when building a sequence using special tokens this is not the token that is used for the end of sequence the token used is the septoken tip septoken str optional defaults to s the separator token which is used when building a sequence from multiple sequences e g two sequences for sequence classification or for a text and a question for question answering it is also used as the last token of a sequence built with special tokens clstoken str optional defaults to s the classifier token which is used when doing sequence classification classification of the whole sequence instead of pertoken classification it is the first token of the sequence when built with special tokens unktoken str optional defaults to unk the unknown token a token that is not in the vocabulary cannot be converted to an id and is set to be this token instead padtoken str optional defaults to pad the token used for padding for example when batching sequences of different lengths masktoken str optional defaults to mask the token used for masking values this is the token used when training this model with masked language modeling this is the token which the model will try to predict addprefixspace bool optional defaults to false whether or not to add an initial space to the input this allows to treat the leading word just as any other word led tokenizer detect beginning of words by the preceding space trimoffsets bool optional defaults to true whether the post processing step should trim offsets to avoid including whitespaces copied from transformers models bart tokenizationbartfast barttokenizerfast init we have to specify that this tokens is special otherwise adding it will reset the normalized flag to false in addspecialtokens the pretokenizer is already updated in the gpt2tokenizerfast init the lists sep and cls must be cased in tuples for the object postprocessorclass copied from transformers models bart tokenizationbartfast barttokenizerfast masktoken with bartled str mask token to use when training a model with maskedlanguage modeling log an error if used while not having been set led tokenizer has a special mask token to be usable in the fillmask pipeline the mask token will greedily comprise the space before the mask overriding the default behavior of the mask token to have it eat the space before it this is needed to preserve backward compatibility with all the previously used models based on led mask token behave like a normal word i e include the space before it so we set lstrip to true copied from transformers models bart tokenizationbartfast barttokenizerfast batchencodeplus copied from transformers models bart tokenizationbartfast barttokenizerfast encodeplus copied from transformers models bart tokenizationbartfast barttokenizerfast savevocabulary copied from transformers models bart tokenizationbartfast barttokenizerfast buildinputswithspecialtokens copied from transformers models bart tokenizationbartfast barttokenizerfast createtokentypeidsfromsequences with bartled create a mask from the two sequences passed to be used in a sequencepair classification task led does not make use of token type ids therefore a list of zeros is returned args tokenids0 listint list of ids tokenids1 listint optional optional second list of ids for sequence pairs returns listint list of zeros copied from transformers models led tokenizationled ledtokenizer pad load from model defaults globalattentionmask need to have the same length as other sequential inputs use 1 since 0 in globalattentionmask means local attention instead of not to attend coding utf 8 2021 iz beltagy matthew e peters arman cohan and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tokenization classes for led construct a fast led tokenizer backed by huggingface s tokenizers library derived from the gpt 2 tokenizer using byte level byte pair encoding this tokenizer has been trained to treat spaces like parts of the tokens a bit like sentencepiece so a word will be encoded differently whether it is at the beginning of the sentence without space or not python from transformers import ledtokenizerfast tokenizer ledtokenizerfast from_pretrained allenai led base 16384 tokenizer hello world input_ids 0 31414 232 2 tokenizer hello world input_ids 0 20920 232 2 you can get around that behavior by passing add_prefix_space true when instantiating this tokenizer or when you call it on some text but since the model was not pretrained this way it might yield a decrease in performance tip when used with is_split_into_words true this tokenizer needs to be instantiated with add_prefix_space true tip this tokenizer inherits from pretrainedtokenizerfast which contains most of the main methods users should refer to this superclass for more information regarding those methods args vocab_file str path to the vocabulary file merges_file str path to the merges file errors str optional defaults to replace paradigm to follow when decoding bytes to utf 8 see bytes decode https docs python org 3 library stdtypes html bytes decode for more information bos_token str optional defaults to s the beginning of sequence token that was used during pretraining can be used a sequence classifier token tip when building a sequence using special tokens this is not the token that is used for the beginning of sequence the token used is the cls_token tip eos_token str optional defaults to s the end of sequence token tip when building a sequence using special tokens this is not the token that is used for the end of sequence the token used is the sep_token tip sep_token str optional defaults to s the separator token which is used when building a sequence from multiple sequences e g two sequences for sequence classification or for a text and a question for question answering it is also used as the last token of a sequence built with special tokens cls_token str optional defaults to s the classifier token which is used when doing sequence classification classification of the whole sequence instead of per token classification it is the first token of the sequence when built with special tokens unk_token str optional defaults to unk the unknown token a token that is not in the vocabulary cannot be converted to an id and is set to be this token instead pad_token str optional defaults to pad the token used for padding for example when batching sequences of different lengths mask_token str optional defaults to mask the token used for masking values this is the token used when training this model with masked language modeling this is the token which the model will try to predict add_prefix_space bool optional defaults to false whether or not to add an initial space to the input this allows to treat the leading word just as any other word led tokenizer detect beginning of words by the preceding space trim_offsets bool optional defaults to true whether the post processing step should trim offsets to avoid including whitespaces copied from transformers models bart tokenization_bart_fast barttokenizerfast __init__ we have to specify that this tokens is special otherwise adding it will reset the normalized flag to false in add_special_tokens the pre_tokenizer is already updated in the gpt2tokenizerfast __init__ the lists sep and cls must be cased in tuples for the object post_processor_class copied from transformers models bart tokenization_bart_fast barttokenizerfast mask_token with bart led str mask token to use when training a model with masked language modeling log an error if used while not having been set led tokenizer has a special mask token to be usable in the fill mask pipeline the mask token will greedily comprise the space before the mask overriding the default behavior of the mask token to have it eat the space before it this is needed to preserve backward compatibility with all the previously used models based on led mask token behave like a normal word i e include the space before it so we set lstrip to true copied from transformers models bart tokenization_bart_fast barttokenizerfast _batch_encode_plus copied from transformers models bart tokenization_bart_fast barttokenizerfast _encode_plus copied from transformers models bart tokenization_bart_fast barttokenizerfast save_vocabulary copied from transformers models bart tokenization_bart_fast barttokenizerfast build_inputs_with_special_tokens copied from transformers models bart tokenization_bart_fast barttokenizerfast create_token_type_ids_from_sequences with bart led create a mask from the two sequences passed to be used in a sequence pair classification task led does not make use of token type ids therefore a list of zeros is returned args token_ids_0 list int list of ids token_ids_1 list int optional optional second list of ids for sequence pairs returns list int list of zeros copied from transformers models led tokenization_led ledtokenizer _pad load from model defaults global_attention_mask need to have the same length as other sequential inputs use 1 since 0 in global_attention_mask means local attention instead of not to attend
import json
from typing import Dict, List, Optional, Tuple, Union

from tokenizers import pre_tokenizers, processors

from ...tokenization_utils_base import AddedToken, BatchEncoding, EncodedInput
from ...tokenization_utils_fast import PreTrainedTokenizerFast
from ...utils import PaddingStrategy, logging
from .tokenization_led import LEDTokenizer


logger = logging.get_logger(__name__)


VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt", "tokenizer_file": "tokenizer.json"}

PRETRAINED_VOCAB_FILES_MAP = {
    "vocab_file": {
        "allenai/led-base-16384": "https://huggingface.co/allenai/led-base-16384/resolve/main/vocab.json",
    },
    "merges_file": {
        "allenai/led-base-16384": "https://huggingface.co/allenai/led-base-16384/resolve/main/merges.txt",
    },
    "tokenizer_file": {
        "allenai/led-base-16384": "https://huggingface.co/allenai/led-base-16384/resolve/main/tokenizer.json",
    },
}

PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {
    "allenai/led-base-16384": 16384,
}


class LEDTokenizerFast(PreTrainedTokenizerFast):
    r

    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    slow_tokenizer_class = LEDTokenizer
    model_input_names = ["input_ids", "attention_mask"]

    
    def __init__(
        self,
        vocab_file=None,
        merges_file=None,
        tokenizer_file=None,
        errors="replace",
        bos_token="<s>",
        eos_token="</s>",
        sep_token="</s>",
        cls_token="<s>",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        add_prefix_space=False,
        trim_offsets=True,
        **kwargs,
    ):
        
        mask_token = (
            AddedToken(mask_token, lstrip=True, normalized=True, special=True)
            if isinstance(mask_token, str)
            else mask_token
        )
        super().__init__(
            vocab_file,
            merges_file,
            tokenizer_file=tokenizer_file,
            errors=errors,
            bos_token=bos_token,
            eos_token=eos_token,
            sep_token=sep_token,
            cls_token=cls_token,
            unk_token=unk_token,
            pad_token=pad_token,
            mask_token=mask_token,
            add_prefix_space=add_prefix_space,
            trim_offsets=trim_offsets,
            **kwargs,
        )

        pre_tok_state = json.loads(self.backend_tokenizer.pre_tokenizer.__getstate__())
        if pre_tok_state.get("add_prefix_space", add_prefix_space) != add_prefix_space:
            pre_tok_class = getattr(pre_tokenizers, pre_tok_state.pop("type"))
            pre_tok_state["add_prefix_space"] = add_prefix_space
            self.backend_tokenizer.pre_tokenizer = pre_tok_class(**pre_tok_state)

        self.add_prefix_space = add_prefix_space

        
        tokenizer_component = "post_processor"
        tokenizer_component_instance = getattr(self.backend_tokenizer, tokenizer_component, None)
        if tokenizer_component_instance:
            state = json.loads(tokenizer_component_instance.__getstate__())

            
            if "sep" in state:
                state["sep"] = tuple(state["sep"])
            if "cls" in state:
                state["cls"] = tuple(state["cls"])

            changes_to_apply = False

            if state.get("add_prefix_space", add_prefix_space) != add_prefix_space:
                state["add_prefix_space"] = add_prefix_space
                changes_to_apply = True

            if state.get("trim_offsets", trim_offsets) != trim_offsets:
                state["trim_offsets"] = trim_offsets
                changes_to_apply = True

            if changes_to_apply:
                component_class = getattr(processors, state.pop("type"))
                new_value = component_class(**state)
                setattr(self.backend_tokenizer, tokenizer_component, new_value)

    @property
    
    def mask_token(self) -> str:
        
        if self._mask_token is None:
            if self.verbose:
                logger.error("Using mask_token, but it is not set yet.")
            return None
        return str(self._mask_token)

    @mask_token.setter
    def mask_token(self, value):
        
        
        
        value = AddedToken(value, lstrip=True, rstrip=False) if isinstance(value, str) else value
        self._mask_token = value

    
    def _batch_encode_plus(self, *args, **kwargs) -> BatchEncoding:
        is_split_into_words = kwargs.get("is_split_into_words", False)

        if is_split_into_words and not self.add_prefix_space:
            raise ValueError(
                f"You need to instantiate {self.__class__.__name__} with add_prefix_space=True "
                "to use it with pretokenized inputs."
            )

        return super()._batch_encode_plus(*args, **kwargs)

    
    def _encode_plus(self, *args, **kwargs) -> BatchEncoding:
        is_split_into_words = kwargs.get("is_split_into_words", False)

        if is_split_into_words and not self.add_prefix_space:
            raise ValueError(
                f"You need to instantiate {self.__class__.__name__} with add_prefix_space=True "
                "to use it with pretokenized inputs."
            )

        return super()._encode_plus(*args, **kwargs)

    
    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        files = self._tokenizer.model.save(save_directory, name=filename_prefix)
        return tuple(files)

    
    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
        output = [self.bos_token_id] + token_ids_0 + [self.eos_token_id]
        if token_ids_1 is None:
            return output

        return output + [self.eos_token_id] + token_ids_1 + [self.eos_token_id]

    
    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

    
    def _pad(
        self,
        encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
        max_length: Optional[int] = None,
        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
        pad_to_multiple_of: Optional[int] = None,
        return_attention_mask: Optional[bool] = None,
    ) -> dict:
        encoded_inputs = super()._pad(
            encoded_inputs=encoded_inputs,
            max_length=max_length,
            padding_strategy=padding_strategy,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
        )

        
        if return_attention_mask is None:
            return_attention_mask = "attention_mask" in self.model_input_names

        if return_attention_mask and "global_attention_mask" in encoded_inputs:
            required_input = encoded_inputs[self.model_input_names[0]]
            
            needs_to_be_padded = len(encoded_inputs["global_attention_mask"]) != len(required_input)

            if needs_to_be_padded:
                difference = len(required_input) - len(encoded_inputs["global_attention_mask"])

                if self.padding_side == "right":
                    
                    encoded_inputs["global_attention_mask"] = (
                        encoded_inputs["global_attention_mask"] + [-1] * difference
                    )
                elif self.padding_side == "left":
                    encoded_inputs["global_attention_mask"] = [-1] * difference + encoded_inputs[
                        "global_attention_mask"
                    ]
                else:
                    raise ValueError("Invalid padding strategy:" + str(self.padding_side))

        return encoded_inputs