codingutf8 2020 the google ai language team s allegro pl facebook inc and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license copied from transformers models xlm tokenizationxlm getpairs return set of symbol pairs in a word word is represented as tuple of symbols symbols being variablelength strings copied from transformers models xlm tokenizationxlm replaceunicodepunct port of https github commosessmtmosesdecoderblobmasterscriptstokenizerreplaceunicodepunctuation perl copied from transformers models xlm tokenizationxlm removenonprintingchar port of https github commosessmtmosesdecoderblobmasterscriptstokenizerremovenonprintingchar perl copied from transformers models bert tokenizationbert whitespacetokenize runs basic whitespace cleaning and splitting on a piece of text text text strip if not text return tokens text split return tokens copied from transformers models bert tokenizationbert basictokenizer class basictokenizerobject def init self dolowercasetrue neversplitnone tokenizechinesecharstrue stripaccentsnone dosplitonpunctrue if neversplit is none neversplit self dolowercase dolowercase self neversplit setneversplit self tokenizechinesechars tokenizechinesechars self stripaccents stripaccents self dosplitonpunc dosplitonpunc def tokenizeself text neversplitnone union returns a new set by concatenating the two sets neversplit self neversplit unionsetneversplit if neversplit else self neversplit text self cleantexttext this was added on november 1st 2018 for the multilingual and chinese models this is also applied to the english models now but it doesn t matter since the english models were not trained on any chinese data and generally don t have any chinese data in them there are chinese characters in the vocabulary because wikipedia does have some chinese words in the english wikipedia if self tokenizechinesechars text self tokenizechinesecharstext prevents treating the same character with different unicode codepoints as different characters unicodenormalizedtext unicodedata normalizenfc text origtokens whitespacetokenizeunicodenormalizedtext splittokens for token in origtokens if token not in neversplit if self dolowercase token token lower if self stripaccents is not false token self runstripaccentstoken elif self stripaccents token self runstripaccentstoken splittokens extendself runsplitonpunctoken neversplit outputtokens whitespacetokenize joinsplittokens return outputtokens def runstripaccentsself text splits punctuation on a piece of text if not self dosplitonpunc or neversplit is not none and text in neversplit return text chars listtext i 0 startnewword true output while i lenchars char charsi if ispunctuationchar output appendchar startnewword true else if startnewword output append startnewword false output1 appendchar i 1 return joinx for x in output def tokenizechinesecharsself text checks whether cp is the codepoint of a cjk character this defines a chinese character as anything in the cjk unicode block https en wikipedia orgwikicjkunifiedideographsunicodeblock note that the cjk unicode block is not all japanese and korean characters despite its name the modern korean hangul alphabet is a different block as is japanese hiragana and katakana those alphabets are used to write spaceseparated words so they are not treated specially and handled like the all of the other languages if cp 0x4e00 and cp 0x9fff or cp 0x3400 and cp 0x4dbf or cp 0x20000 and cp 0x2a6df or cp 0x2a700 and cp 0x2b73f or cp 0x2b740 and cp 0x2b81f or cp 0x2b820 and cp 0x2ceaf or cp 0xf900 and cp 0xfaff or cp 0x2f800 and cp 0x2fa1f return true return false def cleantextself text construct a bpe tokenizer for herbert peculiarities uses bert s pretokenizer basetokenizer splits tokens on spaces and also on punctuation each occurrence of a punctuation character will be treated separately such pretokenized input is bpe subtokenized this tokenizer inherits from xlmtokenizer which contains most of the methods users should refer to the superclass for more information regarding methods cache of sm mosespunctnormalizer instance cache of sm mosestokenizer instance true for current supported model v1 2 0 false for xlm17 100 copied from transformers models xlm tokenizationxlm xlmtokenizer dolowercase copied from transformers models xlm tokenizationxlm xlmtokenizer mosespunctnorm copied from transformers models xlm tokenizationxlm xlmtokenizer mosestokenize copied from transformers models xlm tokenizationxlm xlmtokenizer mosespipeline copied from transformers models xlm tokenizationxlm xlmtokenizer jatokenize copied from transformers models xlm tokenizationxlm xlmtokenizer vocabsize copied from transformers models xlm tokenizationxlm xlmtokenizer getvocab copied from transformers models xlm tokenizationxlm xlmtokenizer bpe copied from transformers models xlm tokenizationxlm xlmtokenizer converttokentoid converts a token str in an id using the vocab return self encoder gettoken self encoder getself unktoken copied from transformers models xlm tokenizationxlm xlmtokenizer convertidtotoken def convertidtotokenself index copied from transformers models xlm tokenizationxlm xlmtokenizer converttokenstostring converts a sequence of tokens string in a single string outstring jointokens replacew strip return outstring copied from transformers models xlm tokenizationxlm xlmtokenizer buildinputswithspecialtokens def buildinputswithspecialtokens self tokenids0 listint tokenids1 optionallistint none listint bos self bostokenid sep self septokenid if tokenids1 is none return bos tokenids0 sep return bos tokenids0 sep tokenids1 sep copied from transformers models xlm tokenizationxlm xlmtokenizer getspecialtokensmask def getspecialtokensmask self tokenids0 listint tokenids1 optionallistint none alreadyhasspecialtokens bool false listint if alreadyhasspecialtokens return super getspecialtokensmask tokenids0tokenids0 tokenids1tokenids1 alreadyhasspecialtokenstrue if tokenids1 is not none return 1 0 lentokenids0 1 0 lentokenids1 1 return 1 0 lentokenids0 1 copied from transformers models xlm tokenizationxlm xlmtokenizer createtokentypeidsfromsequences def createtokentypeidsfromsequences self tokenids0 listint tokenids1 optionallistint none listint sep self septokenid cls self clstokenid if tokenids1 is none return lencls tokenids0 sep 0 return lencls tokenids0 sep 0 lentokenids1 sep 1 copied from transformers models xlm tokenizationxlm xlmtokenizer savevocabulary def savevocabularyself savedirectory str filenameprefix optionalstr none tuplestr if not os path isdirsavedirectory logger errorfvocabulary path savedirectory should be a directory return vocabfile os path join savedirectory filenameprefix if filenameprefix else vocabfilesnamesvocabfile mergefile os path join savedirectory filenameprefix if filenameprefix else vocabfilesnamesmergesfile with openvocabfile w encodingutf8 as f f writejson dumpsself encoder indent2 sortkeystrue ensureasciifalse n index 0 with openmergefile w encodingutf8 as writer for bpetokens tokenindex in sortedself bperanks items keylambda kv kv1 if index tokenindex logger warning fsaving vocabulary to mergefile bpe merge indices are not consecutive please check that the tokenizer is not corrupted index tokenindex writer write joinbpetokens n index 1 return vocabfile mergefile copied from transformers models xlm tokenizationxlm xlmtokenizer getstate def getstateself state self dict copy statesm none return state copied from transformers models xlm tokenizationxlm xlmtokenizer setstate def setstateself d self dict d try import sacremoses except importerror raise importerror you need to install sacremoses to use xlmtokenizer see https pypi orgprojectsacremoses for installation self sm sacremoses coding utf 8 2020 the google ai language team s allegro pl facebook inc and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license copied from transformers models xlm tokenization_xlm get_pairs return set of symbol pairs in a word word is represented as tuple of symbols symbols being variable length strings copied from transformers models xlm tokenization_xlm replace_unicode_punct port of https github com moses smt mosesdecoder blob master scripts tokenizer replace unicode punctuation perl copied from transformers models xlm tokenization_xlm remove_non_printing_char port of https github com moses smt mosesdecoder blob master scripts tokenizer remove non printing char perl copied from transformers models bert tokenization_bert whitespace_tokenize runs basic whitespace cleaning and splitting on a piece of text copied from transformers models bert tokenization_bert basictokenizer constructs a basictokenizer that will run basic tokenization punctuation splitting lower casing etc args do_lower_case bool optional defaults to true whether or not to lowercase the input when tokenizing never_split iterable optional collection of tokens which will never be split during tokenization only has an effect when do_basic_tokenize true tokenize_chinese_chars bool optional defaults to true whether or not to tokenize chinese characters this should likely be deactivated for japanese see this issue https github com huggingface transformers issues 328 strip_accents bool optional whether or not to strip all accents if this option is not specified then it will be determined by the value for lowercase as in the original bert do_split_on_punc bool optional defaults to true in some instances we want to skip the basic punctuation splitting so that later tokenization can capture the full context of the words such as contractions basic tokenization of a piece of text for sub word tokenization see wordpiecetokenizer args never_split list str optional kept for backward compatibility purposes now implemented directly at the base class level see pretrainedtokenizer tokenize list of token not to split union returns a new set by concatenating the two sets this was added on november 1st 2018 for the multilingual and chinese models this is also applied to the english models now but it doesn t matter since the english models were not trained on any chinese data and generally don t have any chinese data in them there are chinese characters in the vocabulary because wikipedia does have some chinese words in the english wikipedia prevents treating the same character with different unicode codepoints as different characters strips accents from a piece of text splits punctuation on a piece of text adds whitespace around any cjk character checks whether cp is the codepoint of a cjk character this defines a chinese character as anything in the cjk unicode block https en wikipedia org wiki cjk_unified_ideographs_ unicode_block note that the cjk unicode block is not all japanese and korean characters despite its name the modern korean hangul alphabet is a different block as is japanese hiragana and katakana those alphabets are used to write space separated words so they are not treated specially and handled like the all of the other languages performs invalid character removal and whitespace cleanup on text construct a bpe tokenizer for herbert peculiarities uses bert s pre tokenizer basetokenizer splits tokens on spaces and also on punctuation each occurrence of a punctuation character will be treated separately such pretokenized input is bpe subtokenized this tokenizer inherits from xlmtokenizer which contains most of the methods users should refer to the superclass for more information regarding methods cache of sm mosespunctnormalizer instance cache of sm mosestokenizer instance true for current supported model v1 2 0 false for xlm 17 100 copied from transformers models xlm tokenization_xlm xlmtokenizer do_lower_case copied from transformers models xlm tokenization_xlm xlmtokenizer moses_punct_norm copied from transformers models xlm tokenization_xlm xlmtokenizer moses_tokenize copied from transformers models xlm tokenization_xlm xlmtokenizer moses_pipeline copied from transformers models xlm tokenization_xlm xlmtokenizer ja_tokenize copied from transformers models xlm tokenization_xlm xlmtokenizer vocab_size copied from transformers models xlm tokenization_xlm xlmtokenizer get_vocab copied from transformers models xlm tokenization_xlm xlmtokenizer bpe copied from transformers models xlm tokenization_xlm xlmtokenizer _convert_token_to_id converts a token str in an id using the vocab copied from transformers models xlm tokenization_xlm xlmtokenizer _convert_id_to_token converts an index integer in a token str using the vocab copied from transformers models xlm tokenization_xlm xlmtokenizer convert_tokens_to_string converts a sequence of tokens string in a single string copied from transformers models xlm tokenization_xlm xlmtokenizer build_inputs_with_special_tokens build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens an xlm sequence has the following format single sequence s x s pair of sequences s a s b s args token_ids_0 list int list of ids to which the special tokens will be added token_ids_1 list int optional optional second list of ids for sequence pairs returns list int list of input ids glossary input ids with the appropriate special tokens copied from transformers models xlm tokenization_xlm xlmtokenizer get_special_tokens_mask retrieve sequence ids from a token list that has no special tokens added this method is called when adding special tokens using the tokenizer prepare_for_model method args token_ids_0 list int list of ids token_ids_1 list int optional optional second list of ids for sequence pairs already_has_special_tokens bool optional defaults to false whether or not the token list is already formatted with special tokens for the model returns list int a list of integers in the range 0 1 1 for a special token 0 for a sequence token copied from transformers models xlm tokenization_xlm xlmtokenizer create_token_type_ids_from_sequences create a mask from the two sequences passed to be used in a sequence pair classification task an xlm sequence pair mask has the following format 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 first sequence second sequence if token_ids_1 is none this method only returns the first portion of the mask 0s args token_ids_0 list int list of ids token_ids_1 list int optional optional second list of ids for sequence pairs returns list int list of token type ids glossary token type ids according to the given sequence s copied from transformers models xlm tokenization_xlm xlmtokenizer save_vocabulary copied from transformers models xlm tokenization_xlm xlmtokenizer __getstate__ copied from transformers models xlm tokenization_xlm xlmtokenizer __setstate__
import json
import os
import re
import unicodedata
from typing import List, Optional, Tuple

from ...tokenization_utils import PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
from ...utils import logging


logger = logging.get_logger(__name__)

VOCAB_FILES_NAMES = {
    "vocab_file": "vocab.json",
    "merges_file": "merges.txt",
}

PRETRAINED_VOCAB_FILES_MAP = {
    "vocab_file": {
        "allegro/herbert-base-cased": "https://huggingface.co/allegro/herbert-base-cased/resolve/main/vocab.json"
    },
    "merges_file": {
        "allegro/herbert-base-cased": "https://huggingface.co/allegro/herbert-base-cased/resolve/main/merges.txt"
    },
}

PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {"allegro/herbert-base-cased": 514}
PRETRAINED_INIT_CONFIGURATION = {}



def get_pairs(word):
    
    pairs = set()
    prev_char = word[0]
    for char in word[1:]:
        pairs.add((prev_char, char))
        prev_char = char
    return pairs



def replace_unicode_punct(text):
    
    text = text.replace("，", ",")
    text = re.sub(r"。\s*", ". ", text)
    text = text.replace("、", ",")
    text = text.replace("”", '"')
    text = text.replace("“", '"')
    text = text.replace("∶", ":")
    text = text.replace("：", ":")
    text = text.replace("？", "?")
    text = text.replace("《", '"')
    text = text.replace("》", '"')
    text = text.replace("）", ")")
    text = text.replace("！", "!")
    text = text.replace("（", "(")
    text = text.replace("；", ";")
    text = text.replace("１", "1")
    text = text.replace("」", '"')
    text = text.replace("「", '"')
    text = text.replace("０", "0")
    text = text.replace("３", "3")
    text = text.replace("２", "2")
    text = text.replace("５", "5")
    text = text.replace("６", "6")
    text = text.replace("９", "9")
    text = text.replace("７", "7")
    text = text.replace("８", "8")
    text = text.replace("４", "4")
    text = re.sub(r"．\s*", ". ", text)
    text = text.replace("～", "~")
    text = text.replace("’", "'")
    text = text.replace("…", "...")
    text = text.replace("━", "-")
    text = text.replace("〈", "<")
    text = text.replace("〉", ">")
    text = text.replace("【", "[")
    text = text.replace("】", "]")
    text = text.replace("％", "%")
    return text



def remove_non_printing_char(text):
    
    output = []
    for char in text:
        cat = unicodedata.category(char)
        if cat.startswith("C"):
            continue
        output.append(char)
    return "".join(output)



def whitespace_tokenize(text):
    
    text = text.strip()
    if not text:
        return []
    tokens = text.split()
    return tokens



class BasicTokenizer(object):
    

    def __init__(
        self,
        do_lower_case=True,
        never_split=None,
        tokenize_chinese_chars=True,
        strip_accents=None,
        do_split_on_punc=True,
    ):
        if never_split is None:
            never_split = []
        self.do_lower_case = do_lower_case
        self.never_split = set(never_split)
        self.tokenize_chinese_chars = tokenize_chinese_chars
        self.strip_accents = strip_accents
        self.do_split_on_punc = do_split_on_punc

    def tokenize(self, text, never_split=None):
        
        
        never_split = self.never_split.union(set(never_split)) if never_split else self.never_split
        text = self._clean_text(text)

        
        
        
        
        
        
        if self.tokenize_chinese_chars:
            text = self._tokenize_chinese_chars(text)
        
        unicode_normalized_text = unicodedata.normalize("NFC", text)
        orig_tokens = whitespace_tokenize(unicode_normalized_text)
        split_tokens = []
        for token in orig_tokens:
            if token not in never_split:
                if self.do_lower_case:
                    token = token.lower()
                    if self.strip_accents is not False:
                        token = self._run_strip_accents(token)
                elif self.strip_accents:
                    token = self._run_strip_accents(token)
            split_tokens.extend(self._run_split_on_punc(token, never_split))

        output_tokens = whitespace_tokenize(" ".join(split_tokens))
        return output_tokens

    def _run_strip_accents(self, text):
        
        text = unicodedata.normalize("NFD", text)
        output = []
        for char in text:
            cat = unicodedata.category(char)
            if cat == "Mn":
                continue
            output.append(char)
        return "".join(output)

    def _run_split_on_punc(self, text, never_split=None):
        
        if not self.do_split_on_punc or (never_split is not None and text in never_split):
            return [text]
        chars = list(text)
        i = 0
        start_new_word = True
        output = []
        while i < len(chars):
            char = chars[i]
            if _is_punctuation(char):
                output.append([char])
                start_new_word = True
            else:
                if start_new_word:
                    output.append([])
                start_new_word = False
                output[-1].append(char)
            i += 1

        return ["".join(x) for x in output]

    def _tokenize_chinese_chars(self, text):
        
        output = []
        for char in text:
            cp = ord(char)
            if self._is_chinese_char(cp):
                output.append(" ")
                output.append(char)
                output.append(" ")
            else:
                output.append(char)
        return "".join(output)

    def _is_chinese_char(self, cp):
        
        
        
        
        
        
        
        
        
        if (
            (cp >= 0x4E00 and cp <= 0x9FFF)
            or (cp >= 0x3400 and cp <= 0x4DBF)  
            or (cp >= 0x20000 and cp <= 0x2A6DF)  
            or (cp >= 0x2A700 and cp <= 0x2B73F)  
            or (cp >= 0x2B740 and cp <= 0x2B81F)  
            or (cp >= 0x2B820 and cp <= 0x2CEAF)  
            or (cp >= 0xF900 and cp <= 0xFAFF)
            or (cp >= 0x2F800 and cp <= 0x2FA1F)  
        ):  
            return True

        return False

    def _clean_text(self, text):
        
        output = []
        for char in text:
            cp = ord(char)
            if cp == 0 or cp == 0xFFFD or _is_control(char):
                continue
            if _is_whitespace(char):
                output.append(" ")
            else:
                output.append(char)
        return "".join(output)


class HerbertTokenizer(PreTrainedTokenizer):
    

    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    pretrained_init_configuration = PRETRAINED_INIT_CONFIGURATION
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES

    def __init__(
        self,
        vocab_file,
        merges_file,
        tokenizer_file=None,
        cls_token="<s>",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        sep_token="</s>",
        bos_token="<s>",
        do_lowercase_and_remove_accent=False,
        additional_special_tokens=[
            "<special0>",
            "<special1>",
            "<special2>",
            "<special3>",
            "<special4>",
            "<special5>",
            "<special6>",
            "<special7>",
            "<special8>",
            "<special9>",
        ],
        lang2id=None,
        id2lang=None,
        **kwargs,
    ):
        try:
            import sacremoses
        except ImportError:
            raise ImportError(
                "You need to install sacremoses to use HerbertTokenizer. "
                "See https://pypi.org/project/sacremoses/ for installation."
            )

        self.sm = sacremoses

        
        self.cache_moses_punct_normalizer = {}
        
        self.cache_moses_tokenizer = {}
        self.lang_with_custom_tokenizer = {"zh", "th", "ja"}
        
        self.do_lowercase_and_remove_accent = do_lowercase_and_remove_accent
        self.lang2id = lang2id
        self.id2lang = id2lang
        if lang2id is not None and id2lang is not None:
            assert len(lang2id) == len(id2lang)

        self.ja_word_tokenizer = None
        self.zh_word_tokenizer = None

        with open(vocab_file, encoding="utf-8") as vocab_handle:
            self.encoder = json.load(vocab_handle)
        self.decoder = {v: k for k, v in self.encoder.items()}
        with open(merges_file, encoding="utf-8") as merges_handle:
            merges = merges_handle.read().split("\n")[:-1]
        merges = [tuple(merge.split()[:2]) for merge in merges]
        self.bpe_ranks = dict(zip(merges, range(len(merges))))
        self.cache = {}

        super().__init__(
            unk_token=unk_token,
            bos_token=bos_token,
            sep_token=sep_token,
            pad_token=pad_token,
            cls_token=cls_token,
            mask_token=mask_token,
            additional_special_tokens=additional_special_tokens,
            lang2id=lang2id,
            id2lang=id2lang,
            do_lowercase_and_remove_accent=do_lowercase_and_remove_accent,
            tokenizer_file=None,
            **kwargs,
        )

        self.bert_pre_tokenizer = BasicTokenizer(
            do_lower_case=False,
            never_split=self.all_special_tokens,
            tokenize_chinese_chars=False,
            strip_accents=False,
        )

    @property
    
    def do_lower_case(self):
        return self.do_lowercase_and_remove_accent

    
    def moses_punct_norm(self, text, lang):
        if lang not in self.cache_moses_punct_normalizer:
            punct_normalizer = self.sm.MosesPunctNormalizer(lang=lang)
            self.cache_moses_punct_normalizer[lang] = punct_normalizer
        else:
            punct_normalizer = self.cache_moses_punct_normalizer[lang]
        return punct_normalizer.normalize(text)

    
    def moses_tokenize(self, text, lang):
        if lang not in self.cache_moses_tokenizer:
            moses_tokenizer = self.sm.MosesTokenizer(lang=lang)
            self.cache_moses_tokenizer[lang] = moses_tokenizer
        else:
            moses_tokenizer = self.cache_moses_tokenizer[lang]
        return moses_tokenizer.tokenize(text, return_str=False, escape=False)

    
    def moses_pipeline(self, text, lang):
        text = replace_unicode_punct(text)
        text = self.moses_punct_norm(text, lang)
        text = remove_non_printing_char(text)
        return text

    
    def ja_tokenize(self, text):
        if self.ja_word_tokenizer is None:
            try:
                import Mykytea

                self.ja_word_tokenizer = Mykytea.Mykytea(
                    f"-model {os.path.expanduser('~')}/local/share/kytea/model.bin"
                )
            except (AttributeError, ImportError):
                logger.error(
                    "Make sure you install KyTea (https://github.com/neubig/kytea) and it's python wrapper"
                    " (https://github.com/chezou/Mykytea-python) with the following steps"
                )
                logger.error("1. git clone git@github.com:neubig/kytea.git && cd kytea")
                logger.error("2. autoreconf -i")
                logger.error("3. ./configure --prefix=$HOME/local")
                logger.error("4. make && make install")
                logger.error("5. pip install kytea")
                raise
        return list(self.ja_word_tokenizer.getWS(text))

    @property
    
    def vocab_size(self):
        return len(self.encoder)

    
    def get_vocab(self):
        return dict(self.encoder, **self.added_tokens_encoder)

    
    def bpe(self, token):
        word = tuple(token[:-1]) + (token[-1] + "</w>",)
        if token in self.cache:
            return self.cache[token]
        pairs = get_pairs(word)

        if not pairs:
            return token + "</w>"

        while True:
            bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf")))
            if bigram not in self.bpe_ranks:
                break
            first, second = bigram
            new_word = []
            i = 0
            while i < len(word):
                try:
                    j = word.index(first, i)
                except ValueError:
                    new_word.extend(word[i:])
                    break
                else:
                    new_word.extend(word[i:j])
                    i = j

                if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
                    new_word.append(first + second)
                    i += 2
                else:
                    new_word.append(word[i])
                    i += 1
            new_word = tuple(new_word)
            word = new_word
            if len(word) == 1:
                break
            else:
                pairs = get_pairs(word)
        word = " ".join(word)
        if word == "\n  </w>":
            word = "\n</w>"
        self.cache[token] = word
        return word

    def _tokenize(self, text):
        pre_tokens = self.bert_pre_tokenizer.tokenize(text)

        split_tokens = []
        for token in pre_tokens:
            if token:
                split_tokens.extend(list(self.bpe(token).split(" ")))

        return split_tokens

    
    def _convert_token_to_id(self, token):
        
        return self.encoder.get(token, self.encoder.get(self.unk_token))

    
    def _convert_id_to_token(self, index):
        
        return self.decoder.get(index, self.unk_token)

    
    def convert_tokens_to_string(self, tokens):
        
        out_string = "".join(tokens).replace("</w>", " ").strip()
        return out_string

    
    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        
        bos = [self.bos_token_id]
        sep = [self.sep_token_id]

        if token_ids_1 is None:
            return bos + token_ids_0 + sep
        return bos + token_ids_0 + sep + token_ids_1 + sep

    
    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        

        if already_has_special_tokens:
            return super().get_special_tokens_mask(
                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
            )

        if token_ids_1 is not None:
            return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1]

    
    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        
        sep = [self.sep_token_id]
        cls = [self.cls_token_id]
        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1]

    
    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )
        merge_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["merges_file"]
        )

        with open(vocab_file, "w", encoding="utf-8") as f:
            f.write(json.dumps(self.encoder, indent=2, sort_keys=True, ensure_ascii=False) + "\n")

        index = 0
        with open(merge_file, "w", encoding="utf-8") as writer:
            for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
                if index != token_index:
                    logger.warning(
                        f"Saving vocabulary to {merge_file}: BPE merge indices are not consecutive."
                        " Please check that the tokenizer is not corrupted!"
                    )
                    index = token_index
                writer.write(" ".join(bpe_tokens) + "\n")
                index += 1

        return vocab_file, merge_file

    
    def __getstate__(self):
        state = self.__dict__.copy()
        state["sm"] = None
        return state

    
    def __setstate__(self, d):
        self.__dict__ = d

        try:
            import sacremoses
        except ImportError:
            raise ImportError(
                "You need to install sacremoses to use XLMTokenizer. "
                "See https://pypi.org/project/sacremoses/ for installation."
            )

        self.sm = sacremoses