codingutf8 2021 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tokenization classes for layoutxlm model import os from shutil import copyfile from typing import any dict list optional tuple union import sentencepiece as spm from tokenizationutils import addedtoken pretrainedtokenizer from tokenizationutilsbase import batchencoding encodedinput pretokenizedinput textinput textinputpair truncationstrategy from utils import paddingstrategy tensortype addenddocstrings logging from xlmroberta tokenizationxlmroberta import pretrainedpositionalembeddingssizes pretrainedvocabfilesmap spieceunderline vocabfilesnames logger logging getloggername layoutxlmencodekwargsdocstring r addspecialtokens bool optional defaults to true whether or not to encode the sequences with the special tokens relative to their model padding bool str or fileutils paddingstrategy optional defaults to false activates and controls padding accepts the following values true or longest pad to the longest sequence in the batch or no padding if only a single sequence if provided maxlength pad to a maximum length specified with the argument maxlength or to the maximum acceptable input length for the model if that argument is not provided false or donotpad default no padding i e can output a batch with sequences of different lengths truncation bool str or tokenizationutilsbase truncationstrategy optional defaults to false activates and controls truncation accepts the following values true or longestfirst truncate to a maximum length specified with the argument maxlength or to the maximum acceptable input length for the model if that argument is not provided this will truncate token by token removing a token from the longest sequence in the pair if a pair of sequences or a batch of pairs is provided onlyfirst truncate to a maximum length specified with the argument maxlength or to the maximum acceptable input length for the model if that argument is not provided this will only truncate the first sequence of a pair if a pair of sequences or a batch of pairs is provided onlysecond truncate to a maximum length specified with the argument maxlength or to the maximum acceptable input length for the model if that argument is not provided this will only truncate the second sequence of a pair if a pair of sequences or a batch of pairs is provided false or donottruncate default no truncation i e can output batch with sequence lengths greater than the model maximum admissible input size maxlength int optional controls the maximum length to use by one of the truncationpadding parameters if left unset or set to none this will use the predefined model maximum length if a maximum length is required by one of the truncationpadding parameters if the model has no specific maximum input length like xlnet truncationpadding to a maximum length will be deactivated stride int optional defaults to 0 if set to a number along with maxlength the overflowing tokens returned when returnoverflowingtokenstrue will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences the value of this argument defines the number of overlapping tokens padtomultipleof int optional if set will pad the sequence to a multiple of the provided value this is especially useful to enable the use of tensor cores on nvidia hardware with compute capability 7 5 volta returntensors str or fileutils tensortype optional if set will return tensors instead of list of python integers acceptable values are tf return tensorflow tf constant objects pt return pytorch torch tensor objects np return numpy np ndarray objects returntokentypeids bool optional whether to return token type ids if left to the default will return the token type ids according to the specific tokenizer s default defined by the returnoutputs attribute what are token type ids glossarytokentypeids returnattentionmask bool optional whether to return the attention mask if left to the default will return the attention mask according to the specific tokenizer s default defined by the returnoutputs attribute what are attention masks glossaryattentionmask returnoverflowingtokens bool optional defaults to false whether or not to return overflowing token sequences if a pair of sequences of input ids or a batch of pairs is provided with truncationstrategy longestfirst or true an error is raised instead of returning overflowing tokens returnspecialtokensmask bool optional defaults to false whether or not to return special tokens mask information returnoffsetsmapping bool optional defaults to false whether or not to return charstart charend for each token this is only available on fast tokenizers inheriting from pretrainedtokenizerfast if using python s tokenizer this method will raise notimplementederror returnlength bool optional defaults to false whether or not to return the lengths of the encoded inputs verbose bool optional defaults to true whether or not to print more information and warnings kwargs passed to the self tokenize method return batchencoding a batchencoding with the following fields inputids list of token ids to be fed to a model what are input ids glossaryinputids bbox list of bounding boxes to be fed to a model tokentypeids list of token type ids to be fed to a model when returntokentypeidstrue or if tokentypeids is in self modelinputnames what are token type ids glossarytokentypeids attentionmask list of indices specifying which tokens should be attended to by the model when returnattentionmasktrue or if attentionmask is in self modelinputnames what are attention masks glossaryattentionmask labels list of labels to be fed to a model when wordlabels is specified overflowingtokens list of overflowing tokens sequences when a maxlength is specified and returnoverflowingtokenstrue numtruncatedtokens number of tokens truncated when a maxlength is specified and returnoverflowingtokenstrue specialtokensmask list of 0s and 1s with 1 specifying added special tokens and 0 specifying regular sequence tokens when addspecialtokenstrue and returnspecialtokensmasktrue length the length of the inputs when returnlengthtrue adapted from robertatokenizer and xlnettokenizer based on sentencepiecehttps github comgooglesentencepiece this tokenizer inherits from pretrainedtokenizer which contains most of the main methods users should refer to this superclass for more information regarding those methods args vocabfile str path to the vocabulary file bostoken str optional defaults to s the beginning of sequence token that was used during pretraining can be used a sequence classifier token tip when building a sequence using special tokens this is not the token that is used for the beginning of sequence the token used is the clstoken tip eostoken str optional defaults to s the end of sequence token tip when building a sequence using special tokens this is not the token that is used for the end of sequence the token used is the septoken tip septoken str optional defaults to s the separator token which is used when building a sequence from multiple sequences e g two sequences for sequence classification or for a text and a question for question answering it is also used as the last token of a sequence built with special tokens clstoken str optional defaults to s the classifier token which is used when doing sequence classification classification of the whole sequence instead of pertoken classification it is the first token of the sequence when built with special tokens unktoken str optional defaults to unk the unknown token a token that is not in the vocabulary cannot be converted to an id and is set to be this token instead padtoken str optional defaults to pad the token used for padding for example when batching sequences of different lengths masktoken str optional defaults to mask the token used for masking values this is the token used when training this model with masked language modeling this is the token which the model will try to predict clstokenbox listint optional defaults to 0 0 0 0 the bounding box to use for the special cls token septokenbox listint optional defaults to 1000 1000 1000 1000 the bounding box to use for the special sep token padtokenbox listint optional defaults to 0 0 0 0 the bounding box to use for the special pad token padtokenlabel int optional defaults to 100 the label to use for padding tokens defaults to 100 which is the ignoreindex of pytorch s crossentropyloss onlylabelfirstsubword bool optional defaults to true whether or not to only label the first subword in case word labels are provided spmodelkwargs dict optional will be passed to the sentencepieceprocessor init method the python wrapper for sentencepiecehttps github comgooglesentencepiecetreemasterpython can be used among other things to set enablesampling enable subword regularization nbestsize sampling parameters for unigram invalid for bpedropout nbestsize 0 1 no sampling is performed nbestsize 1 samples from the nbestsize results nbestsize 0 assuming that nbestsize is infinite and samples from the all hypothesis lattice using forwardfilteringandbackwardsampling algorithm alpha smoothing parameter for unigram sampling and dropout probability of merge operations for bpedropout attributes spmodel sentencepieceprocessor the sentencepiece processor that is used for every conversion string tokens and ids mask token behave like a normal word i e include the space before it original fairseq vocab and spm vocab must be aligned vocab 0 1 2 3 4 5 6 7 8 9 fairseq s pad s unk s de spm unk s s s de a mimic fairseq tokentoid alignment for the first 4 token the first real token has position 4 in the original fairseq vocab and position 3 in the spm vocab additional properties for backward compatibility build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens an xlmroberta sequence has the following format single sequence s x s pair of sequences s a ss b s args tokenids0 listint list of ids to which the special tokens will be added tokenids1 listint optional optional second list of ids for sequence pairs returns listint list of input ids glossaryinputids with the appropriate special tokens retrieve sequence ids from a token list that has no special tokens added this method is called when adding special tokens using the tokenizer prepareformodel method args tokenids0 listint list of ids tokenids1 listint optional optional second list of ids for sequence pairs alreadyhasspecialtokens bool optional defaults to false whether or not the token list is already formatted with special tokens for the model returns listint a list of integers in the range 0 1 1 for a special token 0 for a sequence token create a mask from the two sequences passed to be used in a sequencepair classification task xlmroberta does not make use of token type ids therefore a list of zeros is returned args tokenids0 listint list of ids tokenids1 listint optional optional second list of ids for sequence pairs returns listint list of zeros converts a token str in an id using the vocab if token in self fairseqtokenstoids return self fairseqtokenstoidstoken spmid self spmodel piecetoidtoken need to return unknown token if the sp model returned 0 return spmid self fairseqoffset if spmid else self unktokenid def convertidtotokenself index converts a sequence of tokens strings for subwords in a single string outstring jointokens replacespieceunderline strip return outstring def savevocabularyself savedirectory str filenameprefix optionalstr none tuplestr if not os path isdirsavedirectory logger errorfvocabulary path savedirectory should be a directory return outvocabfile os path join savedirectory filenameprefix if filenameprefix else vocabfilesnamesvocabfile if os path abspathself vocabfile os path abspathoutvocabfile and os path isfileself vocabfile copyfileself vocabfile outvocabfile elif not os path isfileself vocabfile with openoutvocabfile wb as fi contentspiecemodel self spmodel serializedmodelproto fi writecontentspiecemodel return outvocabfile addenddocstringslayoutxlmencodekwargsdocstring def call self text uniontextinput pretokenizedinput listtextinput listpretokenizedinput textpair optionalunionpretokenizedinput listpretokenizedinput none boxes unionlistlistint listlistlistint none wordlabels optionalunionlistint listlistint none addspecialtokens bool true padding unionbool str paddingstrategy false truncation unionbool str truncationstrategy none maxlength optionalint none stride int 0 padtomultipleof optionalint none returntensors optionalunionstr tensortype none returntokentypeids optionalbool none returnattentionmask optionalbool none returnoverflowingtokens bool false returnspecialtokensmask bool false returnoffsetsmapping bool false returnlength bool false verbose bool true kwargs batchencoding input type checking for clearer error def isvalidtextinputt if isinstancet str strings are fine return true elif isinstancet list tuple list are fine as long as they are if lent 0 empty return true elif isinstancet0 str list of strings return true elif isinstancet0 list tuple list with an empty list or with a list of strings return lent0 0 or isinstancet00 str else return false else return false if textpair is not none in case text textpair are provided text questions textpair words if not isvalidtextinputtext raise valueerrortext input must of type str single example or liststr batch of examples if not isinstancetextpair list tuple raise valueerror words must of type liststr single pretokenized example or listliststr batch of pretokenized examples else in case only text is provided must be words if not isinstancetext list tuple raise valueerror words must of type liststr single pretokenized example or listliststr batch of pretokenized examples if textpair is not none isbatched isinstancetext list tuple else isbatched isinstancetext list tuple and text and isinstancetext0 list tuple words text if textpair is none else textpair if boxes is none raise valueerroryou must provide corresponding bounding boxes if isbatched if lenwords lenboxes raise valueerroryou must provide words and boxes for an equal amount of examples for wordsexample boxesexample in zipwords boxes if lenwordsexample lenboxesexample raise valueerroryou must provide as many words as there are bounding boxes else if lenwords lenboxes raise valueerroryou must provide as many words as there are bounding boxes if isbatched if textpair is not none and lentext lentextpair raise valueerror fbatch length of text lentext does not match batch length of textpair f lentextpair batchtextortextpairs listziptext textpair if textpair is not none else text ispair booltextpair is not none return self batchencodeplus batchtextortextpairsbatchtextortextpairs ispairispair boxesboxes wordlabelswordlabels addspecialtokensaddspecialtokens paddingpadding truncationtruncation maxlengthmaxlength stridestride padtomultipleofpadtomultipleof returntensorsreturntensors returntokentypeidsreturntokentypeids returnattentionmaskreturnattentionmask returnoverflowingtokensreturnoverflowingtokens returnspecialtokensmaskreturnspecialtokensmask returnoffsetsmappingreturnoffsetsmapping returnlengthreturnlength verboseverbose kwargs else return self encodeplus texttext textpairtextpair boxesboxes wordlabelswordlabels addspecialtokensaddspecialtokens paddingpadding truncationtruncation maxlengthmaxlength stridestride padtomultipleofpadtomultipleof returntensorsreturntensors returntokentypeidsreturntokentypeids returnattentionmaskreturnattentionmask returnoverflowingtokensreturnoverflowingtokens returnspecialtokensmaskreturnspecialtokensmask returnoffsetsmappingreturnoffsetsmapping returnlengthreturnlength verboseverbose kwargs def batchencodeplus self batchtextortextpairs union listtextinput listtextinputpair listpretokenizedinput ispair bool none boxes optionallistlistlistint none wordlabels optionallistlistint none addspecialtokens bool true paddingstrategy paddingstrategy paddingstrategy donotpad truncationstrategy truncationstrategy truncationstrategy donottruncate maxlength optionalint none stride int 0 padtomultipleof optionalint none returntensors optionalunionstr tensortype none returntokentypeids optionalbool none returnattentionmask optionalbool none returnoverflowingtokens bool false returnspecialtokensmask bool false returnoffsetsmapping bool false returnlength bool false verbose bool true kwargs batchencoding if returnoffsetsmapping raise notimplementederror returnoffsetmapping is not available when using python tokenizers to use this feature change your tokenizer to one deriving from transformers pretrainedtokenizerfast batchoutputs self batchprepareformodel batchtextortextpairsbatchtextortextpairs ispairispair boxesboxes wordlabelswordlabels addspecialtokensaddspecialtokens paddingstrategypaddingstrategy truncationstrategytruncationstrategy maxlengthmaxlength stridestride padtomultipleofpadtomultipleof returnattentionmaskreturnattentionmask returntokentypeidsreturntokentypeids returnoverflowingtokensreturnoverflowingtokens returnspecialtokensmaskreturnspecialtokensmask returnlengthreturnlength returntensorsreturntensors verboseverbose return batchencodingbatchoutputs addenddocstringslayoutxlmencodekwargsdocstring def batchprepareformodel self batchtextortextpairs ispair bool none boxes optionallistlistint none wordlabels optionallistlistint none addspecialtokens bool true paddingstrategy paddingstrategy paddingstrategy donotpad truncationstrategy truncationstrategy truncationstrategy donottruncate maxlength optionalint none stride int 0 padtomultipleof optionalint none returntensors optionalstr none returntokentypeids optionalbool none returnattentionmask optionalbool none returnoverflowingtokens bool false returnspecialtokensmask bool false returnlength bool false verbose bool true batchencoding batchoutputs for idx example in enumeratezipbatchtextortextpairs boxes batchtextortextpair boxesexample example outputs self prepareformodel batchtextortextpair0 if ispair else batchtextortextpair batchtextortextpair1 if ispair else none boxesexample wordlabelswordlabelsidx if wordlabels is not none else none addspecialtokensaddspecialtokens paddingpaddingstrategy donotpad value we pad in batch afterward truncationtruncationstrategy value maxlengthmaxlength stridestride padtomultipleofnone we pad in batch afterward returnattentionmaskfalse we pad in batch afterward returntokentypeidsreturntokentypeids returnoverflowingtokensreturnoverflowingtokens returnspecialtokensmaskreturnspecialtokensmask returnlengthreturnlength returntensorsnone we convert the whole batch to tensors at the end prependbatchaxisfalse verboseverbose for key value in outputs items if key not in batchoutputs batchoutputskey batchoutputskey appendvalue batchoutputs self pad batchoutputs paddingpaddingstrategy value maxlengthmaxlength padtomultipleofpadtomultipleof returnattentionmaskreturnattentionmask batchoutputs batchencodingbatchoutputs tensortypereturntensors return batchoutputs def encodeplus self text uniontextinput pretokenizedinput textpair optionalpretokenizedinput none boxes optionallistlistint none wordlabels optionallistint none addspecialtokens bool true paddingstrategy paddingstrategy paddingstrategy donotpad truncationstrategy truncationstrategy truncationstrategy donottruncate maxlength optionalint none stride int 0 padtomultipleof optionalint none returntensors optionalunionstr tensortype none returntokentypeids optionalbool none returnattentionmask optionalbool none returnoverflowingtokens bool false returnspecialtokensmask bool false returnoffsetsmapping bool false returnlength bool false verbose bool true kwargs batchencoding if returnoffsetsmapping raise notimplementederror returnoffsetmapping is not available when using python tokenizers to use this feature change your tokenizer to one deriving from transformers pretrainedtokenizerfast more information on available tokenizers at https github comhuggingfacetransformerspull2674 return self prepareformodel texttext textpairtextpair boxesboxes wordlabelswordlabels addspecialtokensaddspecialtokens paddingpaddingstrategy value truncationtruncationstrategy value maxlengthmaxlength stridestride padtomultipleofpadtomultipleof returntensorsreturntensors prependbatchaxistrue returnattentionmaskreturnattentionmask returntokentypeidsreturntokentypeids returnoverflowingtokensreturnoverflowingtokens returnspecialtokensmaskreturnspecialtokensmask returnlengthreturnlength verboseverbose addenddocstringslayoutxlmencodekwargsdocstring def prepareformodel self text uniontextinput pretokenizedinput textpair optionalpretokenizedinput none boxes optionallistlistint none wordlabels optionallistint none addspecialtokens bool true padding unionbool str paddingstrategy false truncation unionbool str truncationstrategy none maxlength optionalint none stride int 0 padtomultipleof optionalint none returntensors optionalunionstr tensortype none returntokentypeids optionalbool none returnattentionmask optionalbool none returnoverflowingtokens bool false returnspecialtokensmask bool false returnoffsetsmapping bool false returnlength bool false verbose bool true prependbatchaxis bool false kwargs batchencoding backward compatibility for truncationstrategy padtomaxlength paddingstrategy truncationstrategy maxlength kwargs self getpaddingtruncationstrategies paddingpadding truncationtruncation maxlengthmaxlength padtomultipleofpadtomultipleof verboseverbose kwargs tokens pairtokens tokenboxes pairtokenboxes labels if textpair is none if wordlabels is none case 1 document image classification training inference case 2 token classification inference for word box in ziptext boxes if lenword 1 skip empty words continue wordtokens self tokenizeword tokens extendwordtokens tokenboxes extendbox lenwordtokens else case 2 token classification training for word box label in ziptext boxes wordlabels if lenword 1 skip empty words continue wordtokens self tokenizeword tokens extendwordtokens tokenboxes extendbox lenwordtokens if self onlylabelfirstsubword use the real label id for the first token of the word and padding ids for the remaining tokens labels extendlabel self padtokenlabel lenwordtokens 1 else labels extendlabel lenwordtokens else case 3 document visual question answering inference text question textpair words tokens self tokenizetext tokenboxes self padtokenbox for in rangelentokens self septokenbox for word box in ziptextpair boxes if lenword 1 skip empty words continue wordtokens self tokenizeword pairtokens extendwordtokens pairtokenboxes extendbox lenwordtokens create ids pairids ids self converttokenstoidstokens pairids self converttokenstoidspairtokens if pairtokens else none compute the total size of the returned encodings pair boolpairids is not none lenids lenids lenpairids lenpairids if pair else 0 totallen lenids lenpairids self numspecialtokenstoaddpairpair if addspecialtokens else 0 truncation handle max sequence length overflowingtokens overflowingtokenboxes overflowinglabels if truncationstrategy truncationstrategy donottruncate and maxlength and totallen maxlength ids tokenboxes pairids pairtokenboxes labels overflowingtokens overflowingtokenboxes overflowinglabels self truncatesequences ids tokenboxes pairidspairids pairtokenboxespairtokenboxes labelslabels numtokenstoremovetotallen maxlength truncationstrategytruncationstrategy stridestride if returntokentypeids and not addspecialtokens raise valueerror asking to return tokentypeids while setting addspecialtokens to false results in an undefined behavior please set addspecialtokens to true or set returntokentypeids to none load from model defaults if returntokentypeids is none returntokentypeids tokentypeids in self modelinputnames if returnattentionmask is none returnattentionmask attentionmask in self modelinputnames encodedinputs if returnoverflowingtokens encodedinputsoverflowingtokens overflowingtokens encodedinputsoverflowingtokenboxes overflowingtokenboxes encodedinputsoverflowinglabels overflowinglabels encodedinputsnumtruncatedtokens totallen maxlength add special tokens if addspecialtokens sequence self buildinputswithspecialtokensids pairids tokentypeids self createtokentypeidsfromsequencesids pairids tokenboxes self clstokenbox tokenboxes self septokenbox if pairtokenboxes pairtokenboxes pairtokenboxes self septokenbox if labels labels self padtokenlabel labels self padtokenlabel else sequence ids pairids if pair else ids tokentypeids 0 lenids 0 lenpairids if pair else build output dictionary encodedinputsinputids sequence encodedinputsbbox tokenboxes pairtokenboxes if returntokentypeids encodedinputstokentypeids tokentypeids if returnspecialtokensmask if addspecialtokens encodedinputsspecialtokensmask self getspecialtokensmaskids pairids else encodedinputsspecialtokensmask 0 lensequence if labels encodedinputslabels labels check lengths self eventualwarnabouttoolongsequenceencodedinputsinputids maxlength verbose padding if paddingstrategy paddingstrategy donotpad or returnattentionmask encodedinputs self pad encodedinputs maxlengthmaxlength paddingpaddingstrategy value padtomultipleofpadtomultipleof returnattentionmaskreturnattentionmask if returnlength encodedinputslength lenencodedinputsinputids batchoutputs batchencoding encodedinputs tensortypereturntensors prependbatchaxisprependbatchaxis return batchoutputs def truncatesequences self ids listint tokenboxes listlistint pairids optionallistint none pairtokenboxes optionallistlistint none labels optionallistint none numtokenstoremove int 0 truncationstrategy unionstr truncationstrategy longestfirst stride int 0 tuplelistint listint listint if numtokenstoremove 0 return ids tokenboxes pairids pairtokenboxes labels if not isinstancetruncationstrategy truncationstrategy truncationstrategy truncationstrategytruncationstrategy overflowingtokens overflowingtokenboxes overflowinglabels if truncationstrategy truncationstrategy longestfirst for in rangenumtokenstoremove if pairids is none or lenids lenpairids if not overflowingtokens windowlen minlenids stride 1 else windowlen 1 overflowingtokens extendidswindowlen overflowingtokenboxes extendtokenboxeswindowlen overflowinglabels extendlabelswindowlen ids ids 1 tokenboxes tokenboxes 1 labels labels 1 else if not overflowingtokens windowlen minlenpairids stride 1 else windowlen 1 overflowingtokens extendpairidswindowlen overflowingtokenboxes extendpairtokenboxeswindowlen pairids pairids 1 pairtokenboxes pairtokenboxes 1 elif truncationstrategy truncationstrategy onlyfirst if lenids numtokenstoremove windowlen minlenids stride numtokenstoremove overflowingtokens idswindowlen overflowingtokenboxes tokenboxeswindowlen overflowinglabels labelswindowlen ids ids numtokenstoremove tokenboxes tokenboxes numtokenstoremove labels labels numtokenstoremove else logger error fwe need to remove numtokenstoremove to truncate the input fbut the first sequence has a length lenids fplease select another truncation strategy than truncationstrategy for instance longestfirst or onlysecond elif truncationstrategy truncationstrategy onlysecond and pairids is not none if lenpairids numtokenstoremove windowlen minlenpairids stride numtokenstoremove overflowingtokens pairidswindowlen overflowingtokenboxes pairtokenboxeswindowlen pairids pairids numtokenstoremove pairtokenboxes pairtokenboxes numtokenstoremove else logger error fwe need to remove numtokenstoremove to truncate the input fbut the second sequence has a length lenpairids fplease select another truncation strategy than truncationstrategy for instance longestfirst or onlyfirst return ids tokenboxes pairids pairtokenboxes labels overflowingtokens overflowingtokenboxes overflowinglabels def pad self encodedinputs uniondictstr encodedinput batchencoding maxlength optionalint none paddingstrategy paddingstrategy paddingstrategy donotpad padtomultipleof optionalint none returnattentionmask optionalbool none dict load from model defaults if returnattentionmask is none returnattentionmask attentionmask in self modelinputnames requiredinput encodedinputsself modelinputnames0 if paddingstrategy paddingstrategy longest maxlength lenrequiredinput if maxlength is not none and padtomultipleof is not none and maxlength padtomultipleof 0 maxlength maxlength padtomultipleof 1 padtomultipleof needstobepadded paddingstrategy paddingstrategy donotpad and lenrequiredinput maxlength initialize attention mask if not present if returnattentionmask and attentionmask not in encodedinputs encodedinputsattentionmask 1 lenrequiredinput if needstobepadded difference maxlength lenrequiredinput if self paddingside right if returnattentionmask encodedinputsattentionmask encodedinputsattentionmask 0 difference if tokentypeids in encodedinputs encodedinputstokentypeids encodedinputstokentypeids self padtokentypeid difference if bbox in encodedinputs encodedinputsbbox encodedinputsbbox self padtokenbox difference if labels in encodedinputs encodedinputslabels encodedinputslabels self padtokenlabel difference if specialtokensmask in encodedinputs encodedinputsspecialtokensmask encodedinputsspecialtokensmask 1 difference encodedinputsself modelinputnames0 requiredinput self padtokenid difference elif self paddingside left if returnattentionmask encodedinputsattentionmask 0 difference encodedinputsattentionmask if tokentypeids in encodedinputs encodedinputstokentypeids self padtokentypeid difference encodedinputs tokentypeids if bbox in encodedinputs encodedinputsbbox self padtokenbox difference encodedinputsbbox if labels in encodedinputs encodedinputslabels self padtokenlabel difference encodedinputslabels if specialtokensmask in encodedinputs encodedinputsspecialtokensmask 1 difference encodedinputsspecialtokensmask encodedinputsself modelinputnames0 self padtokenid difference requiredinput else raise valueerrorinvalid padding strategy strself paddingside return encodedinputs coding utf 8 2021 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tokenization classes for layoutxlm model add_special_tokens bool optional defaults to true whether or not to encode the sequences with the special tokens relative to their model padding bool str or file_utils paddingstrategy optional defaults to false activates and controls padding accepts the following values true or longest pad to the longest sequence in the batch or no padding if only a single sequence if provided max_length pad to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided false or do_not_pad default no padding i e can output a batch with sequences of different lengths truncation bool str or tokenization_utils_base truncationstrategy optional defaults to false activates and controls truncation accepts the following values true or longest_first truncate to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided this will truncate token by token removing a token from the longest sequence in the pair if a pair of sequences or a batch of pairs is provided only_first truncate to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided this will only truncate the first sequence of a pair if a pair of sequences or a batch of pairs is provided only_second truncate to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided this will only truncate the second sequence of a pair if a pair of sequences or a batch of pairs is provided false or do_not_truncate default no truncation i e can output batch with sequence lengths greater than the model maximum admissible input size max_length int optional controls the maximum length to use by one of the truncation padding parameters if left unset or set to none this will use the predefined model maximum length if a maximum length is required by one of the truncation padding parameters if the model has no specific maximum input length like xlnet truncation padding to a maximum length will be deactivated stride int optional defaults to 0 if set to a number along with max_length the overflowing tokens returned when return_overflowing_tokens true will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences the value of this argument defines the number of overlapping tokens pad_to_multiple_of int optional if set will pad the sequence to a multiple of the provided value this is especially useful to enable the use of tensor cores on nvidia hardware with compute capability 7 5 volta return_tensors str or file_utils tensortype optional if set will return tensors instead of list of python integers acceptable values are tf return tensorflow tf constant objects pt return pytorch torch tensor objects np return numpy np ndarray objects return_token_type_ids bool optional whether to return token type ids if left to the default will return the token type ids according to the specific tokenizer s default defined by the return_outputs attribute what are token type ids glossary token type ids return_attention_mask bool optional whether to return the attention mask if left to the default will return the attention mask according to the specific tokenizer s default defined by the return_outputs attribute what are attention masks glossary attention mask return_overflowing_tokens bool optional defaults to false whether or not to return overflowing token sequences if a pair of sequences of input ids or a batch of pairs is provided with truncation_strategy longest_first or true an error is raised instead of returning overflowing tokens return_special_tokens_mask bool optional defaults to false whether or not to return special tokens mask information return_offsets_mapping bool optional defaults to false whether or not to return char_start char_end for each token this is only available on fast tokenizers inheriting from pretrainedtokenizerfast if using python s tokenizer this method will raise notimplementederror return_length bool optional defaults to false whether or not to return the lengths of the encoded inputs verbose bool optional defaults to true whether or not to print more information and warnings kwargs passed to the self tokenize method return batchencoding a batchencoding with the following fields input_ids list of token ids to be fed to a model what are input ids glossary input ids bbox list of bounding boxes to be fed to a model token_type_ids list of token type ids to be fed to a model when return_token_type_ids true or if token_type_ids is in self model_input_names what are token type ids glossary token type ids attention_mask list of indices specifying which tokens should be attended to by the model when return_attention_mask true or if attention_mask is in self model_input_names what are attention masks glossary attention mask labels list of labels to be fed to a model when word_labels is specified overflowing_tokens list of overflowing tokens sequences when a max_length is specified and return_overflowing_tokens true num_truncated_tokens number of tokens truncated when a max_length is specified and return_overflowing_tokens true special_tokens_mask list of 0s and 1s with 1 specifying added special tokens and 0 specifying regular sequence tokens when add_special_tokens true and return_special_tokens_mask true length the length of the inputs when return_length true adapted from robertatokenizer and xlnettokenizer based on sentencepiece https github com google sentencepiece this tokenizer inherits from pretrainedtokenizer which contains most of the main methods users should refer to this superclass for more information regarding those methods args vocab_file str path to the vocabulary file bos_token str optional defaults to s the beginning of sequence token that was used during pretraining can be used a sequence classifier token tip when building a sequence using special tokens this is not the token that is used for the beginning of sequence the token used is the cls_token tip eos_token str optional defaults to s the end of sequence token tip when building a sequence using special tokens this is not the token that is used for the end of sequence the token used is the sep_token tip sep_token str optional defaults to s the separator token which is used when building a sequence from multiple sequences e g two sequences for sequence classification or for a text and a question for question answering it is also used as the last token of a sequence built with special tokens cls_token str optional defaults to s the classifier token which is used when doing sequence classification classification of the whole sequence instead of per token classification it is the first token of the sequence when built with special tokens unk_token str optional defaults to unk the unknown token a token that is not in the vocabulary cannot be converted to an id and is set to be this token instead pad_token str optional defaults to pad the token used for padding for example when batching sequences of different lengths mask_token str optional defaults to mask the token used for masking values this is the token used when training this model with masked language modeling this is the token which the model will try to predict cls_token_box list int optional defaults to 0 0 0 0 the bounding box to use for the special cls token sep_token_box list int optional defaults to 1000 1000 1000 1000 the bounding box to use for the special sep token pad_token_box list int optional defaults to 0 0 0 0 the bounding box to use for the special pad token pad_token_label int optional defaults to 100 the label to use for padding tokens defaults to 100 which is the ignore_index of pytorch s crossentropyloss only_label_first_subword bool optional defaults to true whether or not to only label the first subword in case word labels are provided sp_model_kwargs dict optional will be passed to the sentencepieceprocessor __init__ method the python wrapper for sentencepiece https github com google sentencepiece tree master python can be used among other things to set enable_sampling enable subword regularization nbest_size sampling parameters for unigram invalid for bpe dropout nbest_size 0 1 no sampling is performed nbest_size 1 samples from the nbest_size results nbest_size 0 assuming that nbest_size is infinite and samples from the all hypothesis lattice using forward filtering and backward sampling algorithm alpha smoothing parameter for unigram sampling and dropout probability of merge operations for bpe dropout attributes sp_model sentencepieceprocessor the sentencepiece processor that is used for every conversion string tokens and ids mask token behave like a normal word i e include the space before it original fairseq vocab and spm vocab must be aligned vocab 0 1 2 3 4 5 6 7 8 9 fairseq s pad s unk s de spm unk s s s de a mimic fairseq token to id alignment for the first 4 token the first real token has position 4 in the original fairseq vocab and position 3 in the spm vocab additional properties for backward compatibility build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens an xlm roberta sequence has the following format single sequence s x s pair of sequences s a s s b s args token_ids_0 list int list of ids to which the special tokens will be added token_ids_1 list int optional optional second list of ids for sequence pairs returns list int list of input ids glossary input ids with the appropriate special tokens retrieve sequence ids from a token list that has no special tokens added this method is called when adding special tokens using the tokenizer prepare_for_model method args token_ids_0 list int list of ids token_ids_1 list int optional optional second list of ids for sequence pairs already_has_special_tokens bool optional defaults to false whether or not the token list is already formatted with special tokens for the model returns list int a list of integers in the range 0 1 1 for a special token 0 for a sequence token create a mask from the two sequences passed to be used in a sequence pair classification task xlm roberta does not make use of token type ids therefore a list of zeros is returned args token_ids_0 list int list of ids token_ids_1 list int optional optional second list of ids for sequence pairs returns list int list of zeros add the mask token converts a token str in an id using the vocab need to return unknown token if the sp model returned 0 converts an index integer in a token str using the vocab converts a sequence of tokens strings for sub words in a single string main method to tokenize and prepare for the model one or several sequence s or one or several pair s of sequences with word level normalized bounding boxes and optional labels args text str list str list list str the sequence or batch of sequences to be encoded each sequence can be a string a list of strings words of a single example or questions of a batch of examples or a list of list of strings batch of words text_pair list str list list str the sequence or batch of sequences to be encoded each sequence should be a list of strings pretokenized string boxes list list int list list list int word level bounding boxes each bounding box should be normalized to be on a 0 1000 scale word_labels list int list list int optional word level integer labels for token classification tasks such as funsd cord input type checking for clearer error strings are fine list are fine as long as they are empty list of strings list with an empty list or with a list of strings in case text text_pair are provided text questions text_pair words in case only text is provided must be words prepares a sequence of input id or a pair of sequences of inputs ids so that it can be used by the model it adds special tokens truncates sequences if overflowing while taking into account the special tokens and manages a moving window with user defined stride for overflowing tokens args batch_ids_pairs list of tokenized input ids or input ids pairs we pad in batch afterward we pad in batch afterward we pad in batch afterward we convert the whole batch to tensors at the end prepares a sequence or a pair of sequences so that it can be used by the model it adds special tokens truncates sequences if overflowing while taking into account the special tokens and manages a moving window with user defined stride for overflowing tokens word level boxes are turned into token level bbox if provided word level word_labels are turned into token level labels the word label is used for the first token of the word while remaining tokens are labeled with 100 such that they will be ignored by the loss function args text str list str list list str the first sequence to be encoded this can be a string a list of strings or a list of list of strings text_pair list str or list int optional optional second sequence to be encoded this can be a list of strings words of a single example or a list of list of strings words of a batch of examples backward compatibility for truncation_strategy pad_to_max_length case 1 document image classification training inference case 2 token classification inference skip empty words case 2 token classification training skip empty words use the real label id for the first token of the word and padding ids for the remaining tokens case 3 document visual question answering inference text question text_pair words skip empty words create ids pair_ids compute the total size of the returned encodings truncation handle max sequence length load from model defaults add special tokens build output dictionary check lengths padding truncates a sequence pair in place following the strategy args ids list int tokenized input ids of the first sequence can be obtained from a string by chaining the tokenize and convert_tokens_to_ids methods token_boxes list list int bounding boxes of the first sequence pair_ids list int optional tokenized input ids of the second sequence can be obtained from a string by chaining the tokenize and convert_tokens_to_ids methods pair_token_boxes list list int optional bounding boxes of the second sequence labels list int optional labels of the first sequence for token classification tasks num_tokens_to_remove int optional defaults to 0 number of tokens to remove using the truncation strategy truncation_strategy str or tokenization_utils_base truncationstrategy optional defaults to false the strategy to follow for truncation can be longest_first truncate to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided this will truncate token by token removing a token from the longest sequence in the pair if a pair of sequences or a batch of pairs is provided only_first truncate to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided this will only truncate the first sequence of a pair if a pair of sequences or a batch of pairs is provided only_second truncate to a maximum length specified with the argument max_length or to the maximum acceptable input length for the model if that argument is not provided this will only truncate the second sequence of a pair if a pair of sequences or a batch of pairs is provided do_not_truncate default no truncation i e can output batch with sequence lengths greater than the model maximum admissible input size stride int optional defaults to 0 if set to a positive number the overflowing tokens returned will contain some tokens from the main sequence returned the value of this argument defines the number of additional tokens returns tuple list int list int list int the truncated ids the truncated pair_ids and the list of overflowing tokens pad encoded inputs on left right and up to predefined length or max length in the batch args encoded_inputs dictionary of tokenized inputs list int or batch of tokenized inputs list list int max_length maximum length of the returned list and optionally padding length see below will truncate by taking into account the special tokens padding_strategy paddingstrategy to use for padding paddingstrategy longest pad to the longest sequence in the batch paddingstrategy max_length pad to the max length default paddingstrategy do_not_pad do not pad the tokenizer padding sides are defined in self padding_side left pads on the left of the sequences right pads on the right of the sequences pad_to_multiple_of optional integer if set will pad the sequence to a multiple of the provided value this is especially useful to enable the use of tensor core on nvidia hardware with compute capability 7 5 volta return_attention_mask optional set to false to avoid returning attention mask default set to model specifics load from model defaults initialize attention mask if not present
import os
from shutil import copyfile
from typing import Any, Dict, List, Optional, Tuple, Union

import sentencepiece as spm

from ...tokenization_utils import AddedToken, PreTrainedTokenizer
from ...tokenization_utils_base import (
    BatchEncoding,
    EncodedInput,
    PreTokenizedInput,
    TextInput,
    TextInputPair,
    TruncationStrategy,
)
from ...utils import PaddingStrategy, TensorType, add_end_docstrings, logging
from ..xlm_roberta.tokenization_xlm_roberta import (
    PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES,
    PRETRAINED_VOCAB_FILES_MAP,
    SPIECE_UNDERLINE,
    VOCAB_FILES_NAMES,
)


logger = logging.get_logger(__name__)


LAYOUTXLM_ENCODE_KWARGS_DOCSTRING = r


class LayoutXLMTokenizer(PreTrainedTokenizer):
    

    vocab_files_names = VOCAB_FILES_NAMES
    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
    max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
    model_input_names = ["input_ids", "attention_mask"]

    def __init__(
        self,
        vocab_file,
        bos_token="<s>",
        eos_token="</s>",
        sep_token="</s>",
        cls_token="<s>",
        unk_token="<unk>",
        pad_token="<pad>",
        mask_token="<mask>",
        cls_token_box=[0, 0, 0, 0],
        sep_token_box=[1000, 1000, 1000, 1000],
        pad_token_box=[0, 0, 0, 0],
        pad_token_label=-100,
        only_label_first_subword=True,
        sp_model_kwargs: Optional[Dict[str, Any]] = None,
        **kwargs,
    ) -> None:
        
        mask_token = AddedToken(mask_token, lstrip=True, special=True) if isinstance(mask_token, str) else mask_token

        self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs

        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.Load(str(vocab_file))
        self.vocab_file = vocab_file

        
        
        
        
        

        
        self.fairseq_tokens_to_ids = {"<s>": 0, "<pad>": 1, "</s>": 2, "<unk>": 3}

        
        self.fairseq_offset = 1

        self.fairseq_tokens_to_ids["<mask>"] = len(self.sp_model) + self.fairseq_offset
        self.fairseq_ids_to_tokens = {v: k for k, v in self.fairseq_tokens_to_ids.items()}

        
        self.cls_token_box = cls_token_box
        self.sep_token_box = sep_token_box
        self.pad_token_box = pad_token_box
        self.pad_token_label = pad_token_label
        self.only_label_first_subword = only_label_first_subword

        super().__init__(
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            sep_token=sep_token,
            cls_token=cls_token,
            pad_token=pad_token,
            mask_token=mask_token,
            cls_token_box=cls_token_box,
            sep_token_box=sep_token_box,
            pad_token_box=pad_token_box,
            pad_token_label=pad_token_label,
            only_label_first_subword=only_label_first_subword,
            sp_model_kwargs=self.sp_model_kwargs,
            **kwargs,
        )

    def __getstate__(self):
        state = self.__dict__.copy()
        state["sp_model"] = None
        state["sp_model_proto"] = self.sp_model.serialized_model_proto()
        return state

    def __setstate__(self, d):
        self.__dict__ = d

        
        if not hasattr(self, "sp_model_kwargs"):
            self.sp_model_kwargs = {}

        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
        self.sp_model.LoadFromSerializedProto(self.sp_model_proto)

    def build_inputs_with_special_tokens(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        

        if token_ids_1 is None:
            return [self.cls_token_id] + token_ids_0 + [self.sep_token_id]
        cls = [self.cls_token_id]
        sep = [self.sep_token_id]
        return cls + token_ids_0 + sep + sep + token_ids_1 + sep

    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        

        if already_has_special_tokens:
            return super().get_special_tokens_mask(
                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
            )

        if token_ids_1 is None:
            return [1] + ([0] * len(token_ids_0)) + [1]
        return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]

    def create_token_type_ids_from_sequences(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
    ) -> List[int]:
        

        sep = [self.sep_token_id]
        cls = [self.cls_token_id]

        if token_ids_1 is None:
            return len(cls + token_ids_0 + sep) * [0]
        return len(cls + token_ids_0 + sep + sep + token_ids_1 + sep) * [0]

    @property
    def vocab_size(self):
        return len(self.sp_model) + self.fairseq_offset + 1  

    def get_vocab(self):
        vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
        vocab.update(self.added_tokens_encoder)
        return vocab

    def _tokenize(self, text: str) -> List[str]:
        return self.sp_model.encode(text, out_type=str)

    def _convert_token_to_id(self, token):
        
        if token in self.fairseq_tokens_to_ids:
            return self.fairseq_tokens_to_ids[token]
        spm_id = self.sp_model.PieceToId(token)

        
        return spm_id + self.fairseq_offset if spm_id else self.unk_token_id

    def _convert_id_to_token(self, index):
        
        if index in self.fairseq_ids_to_tokens:
            return self.fairseq_ids_to_tokens[index]
        return self.sp_model.IdToPiece(index - self.fairseq_offset)

    def convert_tokens_to_string(self, tokens):
        
        out_string = "".join(tokens).replace(SPIECE_UNDERLINE, " ").strip()
        return out_string

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        if not os.path.isdir(save_directory):
            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
            return
        out_vocab_file = os.path.join(
            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
        )

        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
            copyfile(self.vocab_file, out_vocab_file)
        elif not os.path.isfile(self.vocab_file):
            with open(out_vocab_file, "wb") as fi:
                content_spiece_model = self.sp_model.serialized_model_proto()
                fi.write(content_spiece_model)

        return (out_vocab_file,)

    @add_end_docstrings(LAYOUTXLM_ENCODE_KWARGS_DOCSTRING)
    def __call__(
        self,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]],
        text_pair: Optional[Union[PreTokenizedInput, List[PreTokenizedInput]]] = None,
        boxes: Union[List[List[int]], List[List[List[int]]]] = None,
        word_labels: Optional[Union[List[int], List[List[int]]]] = None,
        add_special_tokens: bool = True,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        return_token_type_ids: Optional[bool] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        **kwargs,
    ) -> BatchEncoding:
        

        
        def _is_valid_text_input(t):
            if isinstance(t, str):
                
                return True
            elif isinstance(t, (list, tuple)):
                
                if len(t) == 0:
                    
                    return True
                elif isinstance(t[0], str):
                    
                    return True
                elif isinstance(t[0], (list, tuple)):
                    
                    return len(t[0]) == 0 or isinstance(t[0][0], str)
                else:
                    return False
            else:
                return False

        if text_pair is not None:
            
            if not _is_valid_text_input(text):
                raise ValueError("text input must of type `str` (single example) or `List[str]` (batch of examples). ")
            if not isinstance(text_pair, (list, tuple)):
                raise ValueError(
                    "words must of type `List[str]` (single pretokenized example), "
                    "or `List[List[str]]` (batch of pretokenized examples)."
                )
        else:
            
            if not isinstance(text, (list, tuple)):
                raise ValueError(
                    "Words must of type `List[str]` (single pretokenized example), "
                    "or `List[List[str]]` (batch of pretokenized examples)."
                )

        if text_pair is not None:
            is_batched = isinstance(text, (list, tuple))
        else:
            is_batched = isinstance(text, (list, tuple)) and text and isinstance(text[0], (list, tuple))

        words = text if text_pair is None else text_pair
        if boxes is None:
            raise ValueError("You must provide corresponding bounding boxes")
        if is_batched:
            if len(words) != len(boxes):
                raise ValueError("You must provide words and boxes for an equal amount of examples")
            for words_example, boxes_example in zip(words, boxes):
                if len(words_example) != len(boxes_example):
                    raise ValueError("You must provide as many words as there are bounding boxes")
        else:
            if len(words) != len(boxes):
                raise ValueError("You must provide as many words as there are bounding boxes")

        if is_batched:
            if text_pair is not None and len(text) != len(text_pair):
                raise ValueError(
                    f"batch length of `text`: {len(text)} does not match batch length of `text_pair`:"
                    f" {len(text_pair)}."
                )
            batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text
            is_pair = bool(text_pair is not None)
            return self.batch_encode_plus(
                batch_text_or_text_pairs=batch_text_or_text_pairs,
                is_pair=is_pair,
                boxes=boxes,
                word_labels=word_labels,
                add_special_tokens=add_special_tokens,
                padding=padding,
                truncation=truncation,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=pad_to_multiple_of,
                return_tensors=return_tensors,
                return_token_type_ids=return_token_type_ids,
                return_attention_mask=return_attention_mask,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_offsets_mapping=return_offsets_mapping,
                return_length=return_length,
                verbose=verbose,
                **kwargs,
            )
        else:
            return self.encode_plus(
                text=text,
                text_pair=text_pair,
                boxes=boxes,
                word_labels=word_labels,
                add_special_tokens=add_special_tokens,
                padding=padding,
                truncation=truncation,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=pad_to_multiple_of,
                return_tensors=return_tensors,
                return_token_type_ids=return_token_type_ids,
                return_attention_mask=return_attention_mask,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_offsets_mapping=return_offsets_mapping,
                return_length=return_length,
                verbose=verbose,
                **kwargs,
            )

    def _batch_encode_plus(
        self,
        batch_text_or_text_pairs: Union[
            List[TextInput],
            List[TextInputPair],
            List[PreTokenizedInput],
        ],
        is_pair: bool = None,
        boxes: Optional[List[List[List[int]]]] = None,
        word_labels: Optional[List[List[int]]] = None,
        add_special_tokens: bool = True,
        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
        truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        return_token_type_ids: Optional[bool] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        **kwargs,
    ) -> BatchEncoding:
        if return_offsets_mapping:
            raise NotImplementedError(
                "return_offset_mapping is not available when using Python tokenizers. "
                "To use this feature, change your tokenizer to one deriving from "
                "transformers.PreTrainedTokenizerFast."
            )

        batch_outputs = self._batch_prepare_for_model(
            batch_text_or_text_pairs=batch_text_or_text_pairs,
            is_pair=is_pair,
            boxes=boxes,
            word_labels=word_labels,
            add_special_tokens=add_special_tokens,
            padding_strategy=padding_strategy,
            truncation_strategy=truncation_strategy,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
            return_token_type_ids=return_token_type_ids,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_length=return_length,
            return_tensors=return_tensors,
            verbose=verbose,
        )

        return BatchEncoding(batch_outputs)

    @add_end_docstrings(LAYOUTXLM_ENCODE_KWARGS_DOCSTRING)
    def _batch_prepare_for_model(
        self,
        batch_text_or_text_pairs,
        is_pair: bool = None,
        boxes: Optional[List[List[int]]] = None,
        word_labels: Optional[List[List[int]]] = None,
        add_special_tokens: bool = True,
        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
        truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[str] = None,
        return_token_type_ids: Optional[bool] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_length: bool = False,
        verbose: bool = True,
    ) -> BatchEncoding:
        

        batch_outputs = {}
        for idx, example in enumerate(zip(batch_text_or_text_pairs, boxes)):
            batch_text_or_text_pair, boxes_example = example
            outputs = self.prepare_for_model(
                batch_text_or_text_pair[0] if is_pair else batch_text_or_text_pair,
                batch_text_or_text_pair[1] if is_pair else None,
                boxes_example,
                word_labels=word_labels[idx] if word_labels is not None else None,
                add_special_tokens=add_special_tokens,
                padding=PaddingStrategy.DO_NOT_PAD.value,  
                truncation=truncation_strategy.value,
                max_length=max_length,
                stride=stride,
                pad_to_multiple_of=None,  
                return_attention_mask=False,  
                return_token_type_ids=return_token_type_ids,
                return_overflowing_tokens=return_overflowing_tokens,
                return_special_tokens_mask=return_special_tokens_mask,
                return_length=return_length,
                return_tensors=None,  
                prepend_batch_axis=False,
                verbose=verbose,
            )

            for key, value in outputs.items():
                if key not in batch_outputs:
                    batch_outputs[key] = []
                batch_outputs[key].append(value)

        batch_outputs = self.pad(
            batch_outputs,
            padding=padding_strategy.value,
            max_length=max_length,
            pad_to_multiple_of=pad_to_multiple_of,
            return_attention_mask=return_attention_mask,
        )

        batch_outputs = BatchEncoding(batch_outputs, tensor_type=return_tensors)

        return batch_outputs

    def _encode_plus(
        self,
        text: Union[TextInput, PreTokenizedInput],
        text_pair: Optional[PreTokenizedInput] = None,
        boxes: Optional[List[List[int]]] = None,
        word_labels: Optional[List[int]] = None,
        add_special_tokens: bool = True,
        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
        truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        return_token_type_ids: Optional[bool] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        **kwargs,
    ) -> BatchEncoding:
        if return_offsets_mapping:
            raise NotImplementedError(
                "return_offset_mapping is not available when using Python tokenizers. "
                "To use this feature, change your tokenizer to one deriving from "
                "transformers.PreTrainedTokenizerFast. "
                "More information on available tokenizers at "
                "https://github.com/huggingface/transformers/pull/2674"
            )

        return self.prepare_for_model(
            text=text,
            text_pair=text_pair,
            boxes=boxes,
            word_labels=word_labels,
            add_special_tokens=add_special_tokens,
            padding=padding_strategy.value,
            truncation=truncation_strategy.value,
            max_length=max_length,
            stride=stride,
            pad_to_multiple_of=pad_to_multiple_of,
            return_tensors=return_tensors,
            prepend_batch_axis=True,
            return_attention_mask=return_attention_mask,
            return_token_type_ids=return_token_type_ids,
            return_overflowing_tokens=return_overflowing_tokens,
            return_special_tokens_mask=return_special_tokens_mask,
            return_length=return_length,
            verbose=verbose,
        )

    @add_end_docstrings(LAYOUTXLM_ENCODE_KWARGS_DOCSTRING)
    def prepare_for_model(
        self,
        text: Union[TextInput, PreTokenizedInput],
        text_pair: Optional[PreTokenizedInput] = None,
        boxes: Optional[List[List[int]]] = None,
        word_labels: Optional[List[int]] = None,
        add_special_tokens: bool = True,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: Optional[int] = None,
        stride: int = 0,
        pad_to_multiple_of: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        return_token_type_ids: Optional[bool] = None,
        return_attention_mask: Optional[bool] = None,
        return_overflowing_tokens: bool = False,
        return_special_tokens_mask: bool = False,
        return_offsets_mapping: bool = False,
        return_length: bool = False,
        verbose: bool = True,
        prepend_batch_axis: bool = False,
        **kwargs,
    ) -> BatchEncoding:
        

        
        padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
            padding=padding,
            truncation=truncation,
            max_length=max_length,
            pad_to_multiple_of=pad_to_multiple_of,
            verbose=verbose,
            **kwargs,
        )

        tokens = []
        pair_tokens = []
        token_boxes = []
        pair_token_boxes = []
        labels = []

        if text_pair is None:
            if word_labels is None:
                
                for word, box in zip(text, boxes):
                    if len(word) < 1:  
                        continue
                    word_tokens = self.tokenize(word)
                    tokens.extend(word_tokens)
                    token_boxes.extend([box] * len(word_tokens))
            else:
                
                for word, box, label in zip(text, boxes, word_labels):
                    if len(word) < 1:  
                        continue
                    word_tokens = self.tokenize(word)
                    tokens.extend(word_tokens)
                    token_boxes.extend([box] * len(word_tokens))
                    if self.only_label_first_subword:
                        
                        labels.extend([label] + [self.pad_token_label] * (len(word_tokens) - 1))
                    else:
                        labels.extend([label] * len(word_tokens))
        else:
            
            
            
            tokens = self.tokenize(text)
            token_boxes = [self.pad_token_box for _ in range(len(tokens))] + [self.sep_token_box]

            for word, box in zip(text_pair, boxes):
                if len(word) < 1:  
                    continue
                word_tokens = self.tokenize(word)
                pair_tokens.extend(word_tokens)
                pair_token_boxes.extend([box] * len(word_tokens))

        
        ids = self.convert_tokens_to_ids(tokens)
        pair_ids = self.convert_tokens_to_ids(pair_tokens) if pair_tokens else None

        
        pair = bool(pair_ids is not None)
        len_ids = len(ids)
        len_pair_ids = len(pair_ids) if pair else 0
        total_len = len_ids + len_pair_ids + (self.num_special_tokens_to_add(pair=pair) if add_special_tokens else 0)

        
        overflowing_tokens = []
        overflowing_token_boxes = []
        overflowing_labels = []
        if truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE and max_length and total_len > max_length:
            (
                ids,
                token_boxes,
                pair_ids,
                pair_token_boxes,
                labels,
                overflowing_tokens,
                overflowing_token_boxes,
                overflowing_labels,
            ) = self.truncate_sequences(
                ids,
                token_boxes,
                pair_ids=pair_ids,
                pair_token_boxes=pair_token_boxes,
                labels=labels,
                num_tokens_to_remove=total_len - max_length,
                truncation_strategy=truncation_strategy,
                stride=stride,
            )

        if return_token_type_ids and not add_special_tokens:
            raise ValueError(
                "Asking to return token_type_ids while setting add_special_tokens to False "
                "results in an undefined behavior. Please set add_special_tokens to True or "
                "set return_token_type_ids to None."
            )

        
        if return_token_type_ids is None:
            return_token_type_ids = "token_type_ids" in self.model_input_names
        if return_attention_mask is None:
            return_attention_mask = "attention_mask" in self.model_input_names

        encoded_inputs = {}

        if return_overflowing_tokens:
            encoded_inputs["overflowing_tokens"] = overflowing_tokens
            encoded_inputs["overflowing_token_boxes"] = overflowing_token_boxes
            encoded_inputs["overflowing_labels"] = overflowing_labels
            encoded_inputs["num_truncated_tokens"] = total_len - max_length

        
        if add_special_tokens:
            sequence = self.build_inputs_with_special_tokens(ids, pair_ids)
            token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids)
            token_boxes = [self.cls_token_box] + token_boxes + [self.sep_token_box]
            if pair_token_boxes:
                pair_token_boxes = pair_token_boxes + [self.sep_token_box]
            if labels:
                labels = [self.pad_token_label] + labels + [self.pad_token_label]
        else:
            sequence = ids + pair_ids if pair else ids
            token_type_ids = [0] * len(ids) + ([0] * len(pair_ids) if pair else [])

        
        encoded_inputs["input_ids"] = sequence
        encoded_inputs["bbox"] = token_boxes + pair_token_boxes
        if return_token_type_ids:
            encoded_inputs["token_type_ids"] = token_type_ids
        if return_special_tokens_mask:
            if add_special_tokens:
                encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
            else:
                encoded_inputs["special_tokens_mask"] = [0] * len(sequence)

        if labels:
            encoded_inputs["labels"] = labels

        
        self._eventual_warn_about_too_long_sequence(encoded_inputs["input_ids"], max_length, verbose)

        
        if padding_strategy != PaddingStrategy.DO_NOT_PAD or return_attention_mask:
            encoded_inputs = self.pad(
                encoded_inputs,
                max_length=max_length,
                padding=padding_strategy.value,
                pad_to_multiple_of=pad_to_multiple_of,
                return_attention_mask=return_attention_mask,
            )

        if return_length:
            encoded_inputs["length"] = len(encoded_inputs["input_ids"])

        batch_outputs = BatchEncoding(
            encoded_inputs, tensor_type=return_tensors, prepend_batch_axis=prepend_batch_axis
        )

        return batch_outputs

    def truncate_sequences(
        self,
        ids: List[int],
        token_boxes: List[List[int]],
        pair_ids: Optional[List[int]] = None,
        pair_token_boxes: Optional[List[List[int]]] = None,
        labels: Optional[List[int]] = None,
        num_tokens_to_remove: int = 0,
        truncation_strategy: Union[str, TruncationStrategy] = "longest_first",
        stride: int = 0,
    ) -> Tuple[List[int], List[int], List[int]]:
        
        if num_tokens_to_remove <= 0:
            return ids, token_boxes, pair_ids, pair_token_boxes, labels, [], [], []

        if not isinstance(truncation_strategy, TruncationStrategy):
            truncation_strategy = TruncationStrategy(truncation_strategy)

        overflowing_tokens = []
        overflowing_token_boxes = []
        overflowing_labels = []
        if truncation_strategy == TruncationStrategy.LONGEST_FIRST:
            for _ in range(num_tokens_to_remove):
                if pair_ids is None or len(ids) > len(pair_ids):
                    if not overflowing_tokens:
                        window_len = min(len(ids), stride + 1)
                    else:
                        window_len = 1
                    overflowing_tokens.extend(ids[-window_len:])
                    overflowing_token_boxes.extend(token_boxes[-window_len:])
                    overflowing_labels.extend(labels[-window_len:])
                    ids = ids[:-1]
                    token_boxes = token_boxes[:-1]
                    labels = labels[:-1]
                else:
                    if not overflowing_tokens:
                        window_len = min(len(pair_ids), stride + 1)
                    else:
                        window_len = 1
                    overflowing_tokens.extend(pair_ids[-window_len:])
                    overflowing_token_boxes.extend(pair_token_boxes[-window_len:])
                    pair_ids = pair_ids[:-1]
                    pair_token_boxes = pair_token_boxes[:-1]
        elif truncation_strategy == TruncationStrategy.ONLY_FIRST:
            if len(ids) > num_tokens_to_remove:
                window_len = min(len(ids), stride + num_tokens_to_remove)
                overflowing_tokens = ids[-window_len:]
                overflowing_token_boxes = token_boxes[-window_len:]
                overflowing_labels = labels[-window_len:]
                ids = ids[:-num_tokens_to_remove]
                token_boxes = token_boxes[:-num_tokens_to_remove]
                labels = labels[:-num_tokens_to_remove]
            else:
                logger.error(
                    f"We need to remove {num_tokens_to_remove} to truncate the input "
                    f"but the first sequence has a length {len(ids)}. "
                    f"Please select another truncation strategy than {truncation_strategy}, "
                    "for instance 'longest_first' or 'only_second'."
                )
        elif truncation_strategy == TruncationStrategy.ONLY_SECOND and pair_ids is not None:
            if len(pair_ids) > num_tokens_to_remove:
                window_len = min(len(pair_ids), stride + num_tokens_to_remove)
                overflowing_tokens = pair_ids[-window_len:]
                overflowing_token_boxes = pair_token_boxes[-window_len:]
                pair_ids = pair_ids[:-num_tokens_to_remove]
                pair_token_boxes = pair_token_boxes[:-num_tokens_to_remove]
            else:
                logger.error(
                    f"We need to remove {num_tokens_to_remove} to truncate the input "
                    f"but the second sequence has a length {len(pair_ids)}. "
                    f"Please select another truncation strategy than {truncation_strategy}, "
                    "for instance 'longest_first' or 'only_first'."
                )

        return (
            ids,
            token_boxes,
            pair_ids,
            pair_token_boxes,
            labels,
            overflowing_tokens,
            overflowing_token_boxes,
            overflowing_labels,
        )

    def _pad(
        self,
        encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
        max_length: Optional[int] = None,
        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
        pad_to_multiple_of: Optional[int] = None,
        return_attention_mask: Optional[bool] = None,
    ) -> dict:
        
        
        if return_attention_mask is None:
            return_attention_mask = "attention_mask" in self.model_input_names

        required_input = encoded_inputs[self.model_input_names[0]]

        if padding_strategy == PaddingStrategy.LONGEST:
            max_length = len(required_input)

        if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
            max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of

        needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length

        
        if return_attention_mask and "attention_mask" not in encoded_inputs:
            encoded_inputs["attention_mask"] = [1] * len(required_input)

        if needs_to_be_padded:
            difference = max_length - len(required_input)
            if self.padding_side == "right":
                if return_attention_mask:
                    encoded_inputs["attention_mask"] = encoded_inputs["attention_mask"] + [0] * difference
                if "token_type_ids" in encoded_inputs:
                    encoded_inputs["token_type_ids"] = (
                        encoded_inputs["token_type_ids"] + [self.pad_token_type_id] * difference
                    )
                if "bbox" in encoded_inputs:
                    encoded_inputs["bbox"] = encoded_inputs["bbox"] + [self.pad_token_box] * difference
                if "labels" in encoded_inputs:
                    encoded_inputs["labels"] = encoded_inputs["labels"] + [self.pad_token_label] * difference
                if "special_tokens_mask" in encoded_inputs:
                    encoded_inputs["special_tokens_mask"] = encoded_inputs["special_tokens_mask"] + [1] * difference
                encoded_inputs[self.model_input_names[0]] = required_input + [self.pad_token_id] * difference
            elif self.padding_side == "left":
                if return_attention_mask:
                    encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
                if "token_type_ids" in encoded_inputs:
                    encoded_inputs["token_type_ids"] = [self.pad_token_type_id] * difference + encoded_inputs[
                        "token_type_ids"
                    ]
                if "bbox" in encoded_inputs:
                    encoded_inputs["bbox"] = [self.pad_token_box] * difference + encoded_inputs["bbox"]
                if "labels" in encoded_inputs:
                    encoded_inputs["labels"] = [self.pad_token_label] * difference + encoded_inputs["labels"]
                if "special_tokens_mask" in encoded_inputs:
                    encoded_inputs["special_tokens_mask"] = [1] * difference + encoded_inputs["special_tokens_mask"]
                encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
            else:
                raise ValueError("Invalid padding strategy:" + str(self.padding_side))

        return encoded_inputs