codingutf8 2022 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license make sure tokenizer plays nice with multiprocessing this list contains the model architectures for which a tiny version could not be created avoid to add new architectures here unless we have verified carefully that it s almost impossible to create them one such case is no model tester class is implemented for a model type like mt5 because its architecture is identical to another one mt5 is based on t5 but trained on different datasets or with different techniques return a tuple of processors for configclass we use tuple here to include potentially both slow fast tokenizers to make a uniform return type check first if a model has processormixin otherwise check if it has tokenizers andor an image processor or a feature extractor remark some configurations have no processor at all for example generic composite models like encoderdecodermodel is used for any compatible text models also decisiontransformer doesn t require any processor we might get none for some tokenizers remove them here return a tuple of all possible architectures attributed to a configuration class configclass for example bertconfig bertmodel bertformaskedlm bertforquestionanswering a model architecture could appear in several mappings for example bartforconditionalgeneration is in modelforpretrainingmappingnames modelwithlmheadmappingnames modelformaskedlmmappingnames modelforseqtoseqcausallmmappingnames we avoid the duplication get the config class from a processor class some configmodel classes use tokenizersfeatureextractors from other models for example gptj uses gpt2tokenizer if no checkpoint is found for a config class or a checkpoint is found without necessary files to create the processor for processorclass we get the config class that corresponds to processorclass and use it to find a checkpoint in order to create the processor wav2vec2ctctokenizer wav2vec2config find the new configuration class create a processor for processorclass if a processor is not able to be built with the original arguments this method tries to change the arguments and call itself recursively by inferring a new configclass or a new processorclass from another one in order to find a checkpoint containing the necessary files to build a processor the processor is not saved here instead it will be saved in convertprocessors after further changes in convertprocessors for each model architecture a copy will be created and saved along the built model currently this solely uses the docstring in the source file of configclass to find a checkpoint try to get the checkpoint from the config class for processorclass this helps cases like xclipconfig and videomaefeatureextractor to find a checkpoint from videomaeconfig try to get a new processor class from checkpoint this is helpful for a checkpoint without necessary file to load processor while processorclass is an auto class for example sew has wav2vec2processor in processormappingnames its tokenizerclass is autotokenizer and the checkpoint https huggingface coasappsewtiny100k has no tokenizer file but we can get tokenizerclass wav2vec2ctctokenizer from the config file the new processor class won t be able to load from checkpoint but it helps this recursive method to find a way to build a processor if tokenizerclass is not specified in config let s use config to get the process class via auto mappings but only allow the tokenizer mapping being used this is to make wav2vec2conformer build used to avoid infinite recursion between a pair of fastslow tokenizer types let s use fast tokenizer if there is any try to build each component tokenizer feature extractor of a processormixin this could be a tuple for tokenizers for example clipprocessor has featureextractorclass clipfeatureextractor tokenizerclass cliptokenizer cliptokenizerfast try to build a processormixin so we can return a single value checkpoint might lack some files to load a processor for example facebookhubertbasels960 has no tokenizer file to load wav2vec2ctctokenizer in this case we try to build a processor with the configuration class for example wav2vec2config corresponding to processorclass try to create an image processor or a feature extractor without any checkpoint validation retrieve a tiny configuration from configclass using each model s modeltester args configclass subclass of pretrainedconfig returns an instance of configclass with tiny hyperparameters for model type like data2vecvision and donutswin we can t get the configmodel file name directly via modeltype as it would be sth like configurationdata2vecvision py a simple way is to use inspect getsourcefileconfigclass the modeling file name without prefix modeling and postfix py find the model tester class sort with the length of the class names first then the alphabetical order this is to avoid t5encoderonlymodeltest is used instead of t5modeltest which has isencoderdecoderfalse and causes some pipeline tests failing also failures in optimum ci todo more fine grained control of the desired tester class cliplike models have textmodeltester and visionmodeltester and we need to pass vocabsize to textmodeltester via textkwargs the same trick is also necessary for flava parent is an instance of unittest testcase but we don t need it here poolformer has no getconfig defined furthermore it s better to use prepareconfigandinputs even if getconfig is defined since there might be some extra changes in prepareconfigandinputs make sure this is long enough some model tester has 20 for this attr to pass textgeneration pipeline tests make sure it at least runs speech2textmodel specific change a processor to work with smaller inputs for tokenizers we try to reduce their vocabulary size for feature extractor we use smaller image size or change other attributes using the values from tinyconfig see convertfeatureextractor this method should not fail we catch the errors and put them in resultwarnings with descriptive messages set tokenizers to none if the fastslow tokenizers have different values for vocabsize or length if keepfasttokenizertrue the fast tokenizer will be kept sanity check 1 fast and slow tokenizers should be compatible vocabsize sanity check 2 fast and slow tokenizers should be compatible length currently we only have these 2 possibilities check the built processors have the unique type if the original fastslow tokenizers don t correspond keep only the fast tokenizer this doesn t necessarily imply the fastslow tokenizers in a single hub repo has issues it s more of an issue in buildprocessor which tries to get a checkpoint with as much effort as possible for yosomodel which uses alberttokenizerfast its real hub checkpoint doesn t contain valid files to load the slower tokenizer alberttokenizer and it ends up finding the canonical checkpoint of albertmodel which has different vocabulary todo try to improve buildprocessor s definition andor usage to avoid the above situation in the first place wav2vec2forctc byt5tokenizer etc all are already small enough and have no fast version that can be retrained if fasttokenizer exists slowtokenizer should correspond to it make sure the fast tokenizer can be saved we don t save it to outputfolder at this moment only at the end of this function let s just keep the fast version if the possibly converted fastslow tokenizers don t correspond set them to none and use the original tokenizers if there is any conversion failed we keep the original tokenizers let s use the original version at the end originalfasttokenizer and originalslowtokenizer make sure the fast tokenizer can be saved we don t save it to outputfolder at this moment only at the end of this function make sure the slow tokenizer can be saved we don t save it to outputfolder at this moment only at the end of this function update feature extractors using the tiny config get frameworkagnostic architecture name used to save all pttfflax models into the same directory archname modelarch name if archname startswithtf archname archname2 elif archname startswithflax archname archname4 return os path joinoutputdir archname def buildmodelmodelarch tinyconfig outputdir checkpointdir getcheckpointdiroutputdir modelarch processoroutputdir os path joinoutputdir processors copy the same set of processors for a model type to the model arch specific folder if os path isdirprocessoroutputdir shutil copytreeprocessoroutputdir checkpointdir dirsexistoktrue tinyconfig copy deepcopytinyconfig if anymodelarch name endswithx for x in forcausallm lmheadmodel tinyconfig isencoderdecoder false tinyconfig isdecoder true model modelarchconfigtinyconfig model savepretrainedcheckpointdir model frompretrainedcheckpointdir return model def fillresultwitherrorresult error trace modelstocreate upload the tiny models archname modeldir splitos path sep1 reponame ftinyrandomarchname repoid forganizationreponame repoexist false error none try createreporepoidrepoid existokfalse repotypemodel tokentoken except exception as e error e if you already created in stre error none logger warningremote repository exists and will be cloned repoexist true try createreporepoidrepoid existoktrue repotypemodel tokentoken except exception as e error e if error is not none raise error with tempfile temporarydirectory as tmpdir repo repositorylocaldirtmpdir clonefromrepoid tokentoken repo gitpull shutil copytreemodeldir tmpdir dirsexistoktrue if repoexist open a pr on the existing hub repo hubprurl uploadfolder folderpathmodeldir repoidrepoid repotypemodel commitmessagefupdate tiny models for archname commitdescriptionfupload tiny models for archname createprtrue tokentoken logger warningfpr open in hubprurl todo we need this information else push to hub repo directly repo gitaddautolfstracktrue repo gitcommitfupload tiny models for archname repo gitpushblockingtrue this prints a progress bar with the upload logger warningftiny models archname pushed to repoid def buildcompositemodelsconfigclass outputdir import tempfile from transformers import bertconfig bertlmheadmodel bertmodel berttokenizer berttokenizerfast encoderdecodermodel gpt2config gpt2lmheadmodel gpt2tokenizer gpt2tokenizerfast speechencoderdecodermodel tfencoderdecodermodel tfvisionencoderdecodermodel tfvisiontextdualencodermodel visionencoderdecodermodel visiontextdualencodermodel vitconfig vitfeatureextractor vitmodel wav2vec2config wav2vec2model wav2vec2processor these will be removed at the end if they are empty result error none warnings if configclass modeltype encoderdecoder encoderconfigclass bertconfig decoderconfigclass bertconfig encoderprocessor berttokenizerfast berttokenizer decoderprocessor berttokenizerfast berttokenizer encoderclass bertmodel decoderclass bertlmheadmodel modelclass encoderdecodermodel tfmodelclass tfencoderdecodermodel elif configclass modeltype visionencoderdecoder encoderconfigclass vitconfig decoderconfigclass gpt2config encoderprocessor vitfeatureextractor decoderprocessor gpt2tokenizerfast gpt2tokenizer encoderclass vitmodel decoderclass gpt2lmheadmodel modelclass visionencoderdecodermodel tfmodelclass tfvisionencoderdecodermodel elif configclass modeltype speechencoderdecoder encoderconfigclass wav2vec2config decoderconfigclass bertconfig encoderprocessor wav2vec2processor decoderprocessor berttokenizerfast berttokenizer encoderclass wav2vec2model decoderclass bertlmheadmodel modelclass speechencoderdecodermodel tfmodelclass none elif configclass modeltype visiontextdualencoder not encoderdecoder but encoderencoder we just keep the same name as above to make code easier encoderconfigclass vitconfig decoderconfigclass bertconfig encoderprocessor vitfeatureextractor decoderprocessor berttokenizerfast berttokenizer encoderclass vitmodel decoderclass bertmodel modelclass visiontextdualencodermodel tfmodelclass tfvisiontextdualencodermodel with tempfile temporarydirectory as tmpdir try build encoder modelstocreate processor encoderprocessor pytorch encoderclass tensorflow encoderoutputdir os path jointmpdir encoder buildencoderconfigclass modelstocreate encoderoutputdir build decoder modelstocreate processor decoderprocessor pytorch decoderclass tensorflow decoderoutputdir os path jointmpdir decoder builddecoderconfigclass modelstocreate decoderoutputdir build encoderdecoder encoderpath os path joinencoderoutputdir encoderclass name decoderpath os path joindecoderoutputdir decoderclass name if configclass modeltype visiontextdualencoder specify these explicitly for encoderdecoder like models but not for visiontextdualencoder as it has no decoder decoderconfig decoderconfigclass frompretraineddecoderpath decoderconfig isdecoder true decoderconfig addcrossattention true model modelclass fromencoderdecoderpretrained encoderpath decoderpath decoderconfigdecoderconfig elif configclass modeltype visiontextdualencoder model modelclass fromvisiontextpretrainedencoderpath decoderpath modelpath os path join outputdir fmodelclass nameencoderconfigclass modeltypedecoderconfigclass modeltype model savepretrainedmodelpath if tfmodelclass is not none model tfmodelclass frompretrainedmodelpath model savepretrainedmodelpath copy the processors encoderprocessorpath os path joinencoderoutputdir processors decoderprocessorpath os path joindecoderoutputdir processors if os path isdirencoderprocessorpath shutil copytreeencoderprocessorpath modelpath dirsexistoktrue if os path isdirdecoderprocessorpath shutil copytreedecoderprocessorpath modelpath dirsexistoktrue fill result resultprocessor x name x name for x in encoderprocessor decoderprocessor resultpytorch modelclass name model modelclass name checkpoint modelpath resulttensorflow if tfmodelclass is not none resulttensorflow tfmodelclass name model tfmodelclass name checkpoint modelpath except exception resulterror ffailed to build models for configclass name traceback formatexc if not resulterror del resulterror if not resultwarnings del resultwarnings return result def gettokenidfromtokenizertokenidname tokenizer originaltokenid tokenid originaltokenid if not tokenidname endswithtokenid raise valueerrorftokenidname is tokenidname which doesn t end with tokenid token getattrtokenizer tokenidname replacetokenid token none if token is not none if isinstancetokenizer pretrainedtokenizerfast tokenid tokenizer converttokentoidwithaddedvoctoken else tokenid tokenizer converttokentoidtoken return tokenid def getconfigoverridesconfigclass processors bark configuration is too special let s just not handle this for now if configclass name barkconfig return configoverrides check if there is any tokenizer prefer fast version if any tokenizer none for processor in processors if isinstanceprocessor pretrainedtokenizerfast tokenizer processor break elif isinstanceprocessor pretrainedtokenizer tokenizer processor if tokenizer is none return configoverrides get some properties of the already converted tokenizer smaller vocab size special token ids etc we use lentokenizer instead of tokenizer vocabsize to avoid potential issues for tokenizers with nonempty addedtokensencoder one example is the debertav2tokenizer where the mask token is the extra token vocabsize lentokenizer the original checkpoint has length 35998 but it doesn t have ids 30400 and 30514 but instead 35998 and 35999 if configclass name gptsanjapaneseconfig vocabsize 2 configoverridesvocabsize vocabsize used to create a new model tester with tokenizer vocabsize in order to get the updated special token ids modeltesterkwargs vocabsize vocabsize fsmtmodeltester accepts srcvocabsize and tgtvocabsize but not vocabsize if configclass name fsmtconfig del modeltesterkwargsvocabsize modeltesterkwargssrcvocabsize tokenizer srcvocabsize modeltesterkwargstgtvocabsize tokenizer tgtvocabsize tinyconfig gettinyconfigconfigclass modeltesterkwargs handle the possibility of textconfig inside tinyconfig for cliplike models owlvit groupvit etc if hasattrtinyconfig textconfig tinyconfig tinyconfig textconfig collect values of some special token ids for attr in dirtinyconfig if attr endswithtokenid tokenid getattrtinyconfig attr if tokenid is not none using the token id values from tokenizer instead of from tinyconfig tokenid gettokenidfromtokenizerattr tokenizer originaltokenidtokenid configoverridesattr tokenid if configclass name fsmtconfig configoverridessrcvocabsize tokenizer srcvocabsize configoverridestgtvocabsize tokenizer tgtvocabsize fsmtconfig has decoderconfig as decoder attribute configoverridesdecoder configurationfsmt decoderconfig vocabsizetokenizer tgtvocabsize bostokenidconfigoverrideseostokenid return configoverrides def buildconfigclass modelstocreate outputdir if datatrainingds is none or datatestingds is none ds loaddatasetwikitext wikitext2rawv1 datatrainingds dstrain datatestingds dstest if configclass modeltype in encoderdecoder visionencoderdecoder speechencoderdecoder visiontextdualencoder return buildcompositemodelsconfigclass outputdir result k for k in modelstocreate these will be removed at the end if they are empty resulterror none resultwarnings build processors processorclasses modelstocreateprocessor if lenprocessorclasses 0 error fno processor class could be found in configclass name fillresultwitherrorresult error none modelstocreate logger errorresulterror0 return result for processorclass in processorclasses try processor buildprocessorconfigclass processorclass allownocheckpointtrue if processor is not none resultprocessorprocessorclass processor except exception error ffailed to build processor for processorclass name trace traceback formatexc fillresultwitherrorresult error trace modelstocreate logger errorresulterror0 return result if lenresultprocessor 0 error fno processor could be built for configclass name fillresultwitherrorresult error none modelstocreate logger errorresulterror0 return result try tinyconfig gettinyconfigconfigclass except exception as e error ffailed to get tiny config for configclass name e trace traceback formatexc fillresultwitherrorresult error trace modelstocreate logger errorresulterror0 return result convert the processors reduce vocabulary size smaller image size etc processors listresultprocessor values processoroutputfolder os path joinoutputdir processors try processors convertprocessorsprocessors tinyconfig processoroutputfolder result except exception error failed to convert the processors trace traceback formatexc resultwarnings appenderror trace if lenprocessors 0 error fno processor is returned by convertprocessors for configclass name fillresultwitherrorresult error none modelstocreate logger errorresulterror0 return result try configoverrides getconfigoverridesconfigclass processors except exception as e error ffailure occurs while calling getconfigoverrides e trace traceback formatexc fillresultwitherrorresult error trace modelstocreate logger errorresulterror0 return result just for us to see this easily in the report if vocabsize in configoverrides resultvocabsize configoverridesvocabsize update attributes that vocabsize involves for k v in configoverrides items if hasattrtinyconfig k setattrtinyconfig k v so far we only have to deal with textconfig as configoverrides contains textrelated attributes only fuyuconfig saves data under both fuyuconfig and its textconfig this is not good but let s just update every involved fields to avoid potential failure if hasattrtinyconfig textconfig and tinyconfig textconfig is not none and hasattrtinyconfig textconfig k setattrtinyconfig textconfig k v if textconfigdict exists we need to update its value here too in order to make savepretrained frompretrained work if hasattrtinyconfig textconfigdict tinyconfig textconfigdictk v if resultwarnings logger warningresultwarnings00 update resultprocessor resultprocessor typep name p class name for p in processors for pytorcharch in modelstocreatepytorch resultpytorchpytorcharch name error none try model buildmodelpytorcharch tinyconfig outputdiroutputdir except exception as e model none error ffailed to create the pytorch model for pytorcharch e trace traceback formatexc resultpytorchpytorcharch namemodel model class name if model is not none else none resultpytorchpytorcharch namecheckpoint getcheckpointdiroutputdir pytorcharch if model is not none else none if error is not none resultpytorchpytorcharch nameerror error trace logger errorfpytorcharch name error for tensorflowarch in modelstocreatetensorflow make pttf weights compatible ptarchname tensorflowarch name2 remove tf ptarch getattrtransformersmodule ptarchname resulttensorflowtensorflowarch name error none if ptarch name in resultpytorch and resultpytorchptarch namecheckpoint is not none ckpt getcheckpointdiroutputdir ptarch use the same weights from pytorch try model tensorflowarch frompretrainedckpt model savepretrainedckpt except exception as e conversion may fail let s not create a model with different weights to avoid confusion for now model none error ffailed to convert the pytorch model to the tensorflow model for ptarch e trace traceback formatexc else try model buildmodeltensorflowarch tinyconfig outputdiroutputdir except exception as e model none error ffailed to create the tensorflow model for tensorflowarch e trace traceback formatexc resulttensorflowtensorflowarch namemodel model class name if model is not none else none resulttensorflowtensorflowarch namecheckpoint getcheckpointdiroutputdir tensorflowarch if model is not none else none if error is not none resulttensorflowtensorflowarch nameerror error trace logger errorftensorflowarch name error if not resulterror del resulterror if not resultwarnings del resultwarnings return result def buildtinymodelsummaryresults organizationnone tokennone tinymodelsummary for configname in results processors key for key value in resultsconfignameprocessor items tokenizerclasses sortedx for x in processors if x endswithtokenizerfast or x endswithtokenizer processorclasses sortedx for x in processors if x not in tokenizerclasses for framework in frameworks if framework not in resultsconfigname continue for archname in resultsconfignameframework modelclasses archname basearchname archname2 if archname startswithtf else archname tiny model is not created for archname if resultsconfignameframeworkarchnamemodel is none modelclasses if basearchname not in tinymodelsummary tinymodelsummarybasearchname tinymodelsummarybasearchname update tokenizerclasses tokenizerclasses processorclasses processorclasses tinymodelsummarybasearchnamemodelclasses sorted tinymodelsummarybasearchname getmodelclasses modelclasses if organization is not none reponame ftinyrandombasearchname composite models checkpoints have more precise repo names on the hub if basearchname in compositemodels reponame ftinyrandomcompositemodelsbasearchname repoid forganizationreponame try commithash hfapi repoinforepoid tokentoken sha except exception the directory is not created but processors isare included in results logger warningffailed to get information for repoid ntraceback formatexc del tinymodelsummarybasearchname continue tinymodelsummarybasearchnamesha commithash return tinymodelsummary def buildfailedreportresults includewarningtrue failedresults for configname in results if error in resultsconfigname if configname not in failedresults failedresultsconfigname failedresultsconfigname error resultsconfignameerror if includewarning and warnings in resultsconfigname if configname not in failedresults failedresultsconfigname failedresultsconfignamewarnings resultsconfignamewarnings for framework in frameworks if framework not in resultsconfigname continue for archname in resultsconfignameframework if error in resultsconfignameframeworkarchname if configname not in failedresults failedresultsconfigname if framework not in failedresultsconfigname failedresultsconfignameframework if archname not in failedresultsconfignameframework failedresultsconfignameframeworkarchname error resultsconfignameframeworkarchnameerror failedresultsconfignameframeworkarchnameerror error return failedresults def buildsimplereportresults text failedtext for configname in results for framework in frameworks if framework not in resultsconfigname continue for archname in resultsconfignameframework if error in resultsconfignameframeworkarchname result resultsconfignameframeworkarchnameerror failedtext farchname result0n else result ok text farchname result0n return text failedtext def updatetinymodelsummaryfilereportpath with openos path joinreportpath tinymodelsummary json as fp newdata json loadfp with opentestsutilstinymodelsummary json as fp data json loadfp for key value in newdata items if key not in data datakey value else for attr in tokenizerclasses processorclasses modelclasses we might get duplication here we will remove them below when creating updateddata datakeyattr extendvalueattr newsha value getsha none if newsha is not none datakeysha newsha updateddata for key in sorteddata keys updateddatakey for attr value in datakey items deduplication and sort updateddatakeyattr sortedsetvalue if attr sha else value with openos path joinreportpath updatedtinymodelsummary json w as fp json dumpupdateddata fp indent4 ensureasciifalse def createtinymodels outputpath all modeltypes modelstoskip nocheck upload organization token numworkers1 clonepath os path abspathos path dirnameos path dirnamefile if os getcwd clonepath raise valueerrorfthis script should be run from the root of the clone of transformers clonepath reportpath os path joinoutputpath reports os makedirsreportpath pytorcharchmappings x for x in dirtransformersmodule if x startswithmodel and x endswithmapping and x modelnamesmapping tensorflowarchmappings x for x in dirtransformersmodule if x startswithtfmodel and x endswithmapping pytorcharchmappings getattrtransformersmodule x for x in pytorcharchmappings tensorflowarchmappings getattrtransformersmodule x for x in tensorflowarchmappings configclasses configmapping values if not all configclasses configmappingmodeltype for modeltype in modeltypes a map from config classes to tuples of processors tokenizer feature extractor processor classes processortypemap c getprocessortypesfromconfigclassc for c in configclasses tocreate for c in configclasses processors processortypemapc models getarchitecturesfromconfigclassc pytorcharchmappings modelstoskip tfmodels getarchitecturesfromconfigclassc tensorflowarchmappings modelstoskip if lenmodels lentfmodels 0 tocreatec processor processors pytorch models tensorflow tfmodels results if numworkers 1 for c modelstocreate in listtocreate items printfcreate models for c name result buildc modelstocreate outputdiros path joinoutputpath c modeltype resultsc name result print 40 else allbuildargs for c modelstocreate in listtocreate items allbuildargs appendc modelstocreate os path joinoutputpath c modeltype with multiprocessing pool as pool results pool starmapbuild allbuildargs results buidargs0 name result for buidargs result in zipallbuildargs results if upload if organization is none raise valueerrorthe argument organization could not be none no model is uploaded toupload for modeltype in os listdiroutputpath this is the directory containing the reports if modeltype reports continue for arch in os listdiros path joinoutputpath modeltype if arch processors continue toupload appendos path joinoutputpath modeltype arch toupload sortedtoupload uploadresults if lentoupload 0 for modeldir in toupload try uploadmodelmodeldir organization token except exception as e error ffailed to upload modeldir e class name e logger errorerror uploadresultsmodeldir error with openos path joinreportpath faileduploads json w as fp json dumpuploadresults fp indent4 build the tiny model summary file the tokenizerclasses and processorclasses could be both empty lists when using the items in this file to update the file testsutilstinymodelsummary json the model architectures with tokenizerclasses and processorclasses being both empty should not be added to testsutilstinymodelsummary json tinymodelsummary buildtinymodelsummaryresults organizationorganization tokentoken with openos path joinreportpath tinymodelsummary json w as fp json dumptinymodelsummary fp indent4 with openos path joinreportpath tinymodelcreationreport json w as fp json dumpresults fp indent4 build the warningfailure report json format same format as the complete results except this contains only warnings or errors failedresults buildfailedreportresults with openos path joinreportpath failedreport json w as fp json dumpfailedresults fp indent4 simplereport failedreport buildsimplereportresults the simplified report a txt file with each line of format model architecture name ok or error message with openos path joinreportpath simplereport txt w as fp fp writesimplereport the simplified failure report same above except this only contains line with errors with openos path joinreportpath simplefailedreport txt w as fp fp writefailedreport updatetinymodelsummaryfilereportpathos path joinoutputpath reports if name main this has to be spawn to avoid hanging forever multiprocessing setstartmethodspawn def liststrvalues return values split parser argparse argumentparser parser addargumentall actionstoretrue helpwill create all tiny models parser addargument nocheck actionstoretrue helpif set will not check the validity of architectures use with caution parser addargument m modeltypes typeliststr helpcommaseparated list of model types from which the tiny models will be created parser addargument modelstoskip typeliststr help commaseparated list of model class namess from which the tiny models won t be created nthis is usually the list of model classes that have their tiny versions already uploaded to the hub parser addargumentupload actionstoretrue helpif to upload the created tiny models to the hub parser addargument organization defaultnone typestr helpthe organization on the hub to which the tiny models will be uploaded parser addargument token defaultnone typestr helpa valid authentication token for huggingface hub with write access parser addargumentoutputpath typepath helppath indicating where to store generated model parser addargumentnumworkers default1 typeint helpthe number of workers to run args parser parseargs if not args all and not args modeltypes raise valueerrorplease provide at least one model type or pass all to export all architectures createtinymodels args outputpath args all args modeltypes args modelstoskip args nocheck args upload args organization args token args numworkers coding utf 8 2022 the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license make sure tokenizer plays nice with multiprocessing this list contains the model architectures for which a tiny version could not be created avoid to add new architectures here unless we have verified carefully that it s almost impossible to create them one such case is no model tester class is implemented for a model type like mt5 because its architecture is identical to another one mt5 is based on t5 but trained on different datasets or with different techniques return a tuple of processors for config_class we use tuple here to include potentially both slow fast tokenizers to make a uniform return type check first if a model has processormixin otherwise check if it has tokenizers and or an image processor or a feature extractor remark some configurations have no processor at all for example generic composite models like encoderdecodermodel is used for any compatible text models also decisiontransformer doesn t require any processor we might get none for some tokenizers remove them here return a tuple of all possible architectures attributed to a configuration class config_class for example bertconfig bertmodel bertformaskedlm bertforquestionanswering a model architecture could appear in several mappings for example bartforconditionalgeneration is in model_for_pretraining_mapping_names model_with_lm_head_mapping_names model_for_masked_lm_mapping_names model_for_seq_to_seq_causal_lm_mapping_names we avoid the duplication get the config class from a processor class some config model classes use tokenizers feature_extractors from other models for example gpt j uses gpt2tokenizer if no checkpoint is found for a config class or a checkpoint is found without necessary file s to create the processor for processor_class we get the config class that corresponds to processor_class and use it to find a checkpoint in order to create the processor wav2vec2ctctokenizer wav2vec2config find the new configuration class create a processor for processor_class if a processor is not able to be built with the original arguments this method tries to change the arguments and call itself recursively by inferring a new config_class or a new processor_class from another one in order to find a checkpoint containing the necessary files to build a processor the processor is not saved here instead it will be saved in convert_processors after further changes in convert_processors for each model architecture a copy will be created and saved along the built model currently this solely uses the docstring in the source file of config_class to find a checkpoint try to get the checkpoint from the config class for processor_class this helps cases like xclipconfig and videomaefeatureextractor to find a checkpoint from videomaeconfig try to get a new processor class from checkpoint this is helpful for a checkpoint without necessary file to load processor while processor_class is an auto class for example sew has wav2vec2processor in processor_mapping_names its tokenizer_class is autotokenizer and the checkpoint https huggingface co asapp sew tiny 100k has no tokenizer file but we can get tokenizer_class wav2vec2ctctokenizer from the config file the new processor class won t be able to load from checkpoint but it helps this recursive method to find a way to build a processor if tokenizer_class is not specified in config let s use config to get the process class via auto mappings but only allow the tokenizer mapping being used this is to make wav2vec2conformer build used to avoid infinite recursion between a pair of fast slow tokenizer types let s use fast tokenizer if there is any try to build each component tokenizer feature extractor of a processormixin this could be a tuple for tokenizers for example clipprocessor has feature_extractor_class clipfeatureextractor tokenizer_class cliptokenizer cliptokenizerfast try to build a processormixin so we can return a single value checkpoint might lack some file s to load a processor for example facebook hubert base ls960 has no tokenizer file to load wav2vec2ctctokenizer in this case we try to build a processor with the configuration class for example wav2vec2config corresponding to processor_class try to create an image processor or a feature extractor without any checkpoint validation retrieve a tiny configuration from config_class using each model s modeltester args config_class subclass of pretrainedconfig returns an instance of config_class with tiny hyperparameters for model type like data2vec vision and donut swin we can t get the config model file name directly via model_type as it would be sth like configuration_data2vec_vision py a simple way is to use inspect getsourcefile config_class the modeling file name without prefix modeling_ and postfix py find the model tester class sort with the length of the class names first then the alphabetical order this is to avoid t5encoderonlymodeltest is used instead of t5modeltest which has is_encoder_decoder false and causes some pipeline tests failing also failures in optimum ci todo more fine grained control of the desired tester class clip like models have text_model_tester and vision_model_tester and we need to pass vocab_size to text_model_tester via text_kwargs the same trick is also necessary for flava parent is an instance of unittest testcase but we don t need it here poolformer has no get_config defined furthermore it s better to use prepare_config_and_inputs even if get_config is defined since there might be some extra changes in prepare_config_and_inputs make sure this is long enough some model tester has 20 for this attr to pass text generation pipeline tests make sure it at least runs speech2textmodel specific change a processor to work with smaller inputs for tokenizers we try to reduce their vocabulary size for feature extractor we use smaller image size or change other attributes using the values from tiny_config see convert_feature_extractor this method should not fail we catch the errors and put them in result warnings with descriptive messages set tokenizer s to none if the fast slow tokenizers have different values for vocab_size or length if keep_fast_tokenizer true the fast tokenizer will be kept sanity check 1 fast and slow tokenizers should be compatible vocab_size sanity check 2 fast and slow tokenizers should be compatible length currently we only have these 2 possibilities check the built processors have the unique type if the original fast slow tokenizers don t correspond keep only the fast tokenizer this doesn t necessarily imply the fast slow tokenizers in a single hub repo has issues it s more of an issue in build_processor which tries to get a checkpoint with as much effort as possible for yosomodel which uses alberttokenizer fast its real hub checkpoint doesn t contain valid files to load the slower tokenizer alberttokenizer and it ends up finding the canonical checkpoint of albertmodel which has different vocabulary todo try to improve build_processor s definition and or usage to avoid the above situation in the first place wav2vec2forctc byt5tokenizer etc all are already small enough and have no fast version that can be retrained if fast_tokenizer exists slow_tokenizer should correspond to it make sure the fast tokenizer can be saved we don t save it to output_folder at this moment only at the end of this function let s just keep the fast version if the possibly converted fast slow tokenizers don t correspond set them to none and use the original tokenizers if there is any conversion failed we keep the original tokenizers let s use the original version at the end original_fast_tokenizer and original_slow_tokenizer make sure the fast tokenizer can be saved we don t save it to output_folder at this moment only at the end of this function make sure the slow tokenizer can be saved we don t save it to output_folder at this moment only at the end of this function update feature extractors using the tiny config get framework agnostic architecture name used to save all pt tf flax models into the same directory create and save a model for model_arch also copy the set of processors to each model under the same model type output folder copy the same set of processors for a model type to the model arch specific folder fill result with errors for all target model arch if we can t build processor upload the tiny models open a pr on the existing hub repo todo we need this information push to hub repo directly this prints a progress bar with the upload these will be removed at the end if they are empty not encoder decoder but encoder encoder we just keep the same name as above to make code easier build encoder build decoder build encoder decoder specify these explicitly for encoder decoder like models but not for vision text dual encoder as it has no decoder copy the processors fill result use tokenizer to get the values of bos_token_id eos_token_ids etc the argument token_id_name should be a string ending with _token_id and original_token_id should be an integer that will be return if tokenizer has no token corresponding to token_id_name bark configuration is too special let s just not handle this for now check if there is any tokenizer prefer fast version if any get some properties of the already converted tokenizer smaller vocab size special token ids etc we use len tokenizer instead of tokenizer vocab_size to avoid potential issues for tokenizers with non empty added_tokens_encoder one example is the debertav2tokenizer where the mask token is the extra token the original checkpoint has length 35998 but it doesn t have ids 30400 and 30514 but instead 35998 and 35999 used to create a new model tester with tokenizer vocab_size in order to get the updated special token ids fsmtmodeltester accepts src_vocab_size and tgt_vocab_size but not vocab_size handle the possibility of text_config inside _tiny_config for clip like models owlvit groupvit etc collect values of some special token ids using the token id values from tokenizer instead of from _tiny_config fsmtconfig has decoderconfig as decoder attribute create all models for a certain model type args config_class pretrainedconfig a subclass of pretrainedconfig that is used to determine models_to_create models_to_create dict a dictionary containing the processor model classes that we want to create the instances these models are of the same model type which is associated to config_class output_dir str the directory to save all the checkpoints each model architecture will be saved in a subdirectory under it models in different frameworks with the same architecture will be saved in the same subdirectory these will be removed at the end if they are empty build processors convert the processors reduce vocabulary size smaller image size etc just for us to see this easily in the report update attributes that vocab_size involves so far we only have to deal with text_config as config_overrides contains text related attributes only fuyuconfig saves data under both fuyuconfig and its text_config this is not good but let s just update every involved fields to avoid potential failure if text_config_dict exists we need to update its value here too in order to make save_pretrained from_pretrained work update result processor make pt tf weights compatible remove tf use the same weights from pytorch conversion may fail let s not create a model with different weights to avoid confusion for now build a summary a dictionary of the form model architecture name tokenizer_classes processor_classes model_classes tiny model is not created for arch_name composite models checkpoints have more precise repo names on the hub the directory is not created but processor s is are included in results we might get duplication here we will remove them below when creating updated_data deduplication and sort a map from config classes to tuples of processors tokenizer feature extractor processor classes this is the directory containing the reports build the tiny model summary file the tokenizer_classes and processor_classes could be both empty lists when using the items in this file to update the file tests utils tiny_model_summary json the model architectures with tokenizer_classes and processor_classes being both empty should not be added to tests utils tiny_model_summary json build the warning failure report json format same format as the complete results except this contains only warnings or errors the simplified report a txt file with each line of format model architecture name ok or error message the simplified failure report same above except this only contains line with errors this has to be spawn to avoid hanging forever
import argparse
import collections.abc
import copy
import inspect
import json
import multiprocessing
import os
import shutil
import tempfile
import traceback
from pathlib import Path

from check_config_docstrings import get_checkpoint_from_config_class
from datasets import load_dataset
from get_test_info import get_model_to_tester_mapping, get_tester_classes_for_model
from huggingface_hub import Repository, create_repo, hf_api, upload_folder

from transformers import (
    CONFIG_MAPPING,
    FEATURE_EXTRACTOR_MAPPING,
    IMAGE_PROCESSOR_MAPPING,
    PROCESSOR_MAPPING,
    TOKENIZER_MAPPING,
    AutoTokenizer,
    LayoutLMv3TokenizerFast,
    PreTrainedTokenizer,
    PreTrainedTokenizerFast,
    logging,
)
from transformers.feature_extraction_utils import FeatureExtractionMixin
from transformers.file_utils import is_tf_available, is_torch_available
from transformers.image_processing_utils import BaseImageProcessor
from transformers.models.auto.configuration_auto import AutoConfig, model_type_to_module_name
from transformers.models.fsmt import configuration_fsmt
from transformers.processing_utils import ProcessorMixin, transformers_module
from transformers.tokenization_utils_base import PreTrainedTokenizerBase



os.environ["TOKENIZERS_PARALLELISM"] = "false"

logging.set_verbosity_error()
logging.disable_progress_bar()
logger = logging.get_logger(__name__)

os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"

if not is_torch_available():
    raise ValueError("Please install PyTorch.")

if not is_tf_available():
    raise ValueError("Please install TensorFlow.")


FRAMEWORKS = ["pytorch", "tensorflow"]
INVALID_ARCH = []
TARGET_VOCAB_SIZE = 1024

data = {"training_ds": None, "testing_ds": None}

COMPOSITE_MODELS = {
    "EncoderDecoderModel": "EncoderDecoderModel-bert-bert",
    "SpeechEncoderDecoderModel": "SpeechEncoderDecoderModel-wav2vec2-bert",
    "VisionEncoderDecoderModel": "VisionEncoderDecoderModel-vit-gpt2",
    "VisionTextDualEncoderModel": "VisionTextDualEncoderModel-vit-bert",
}





UNCONVERTIBLE_MODEL_ARCHITECTURES = {
    "BertGenerationEncoder",
    "BertGenerationDecoder",
    "CamembertForSequenceClassification",
    "CamembertForMultipleChoice",
    "CamembertForMaskedLM",
    "CamembertForCausalLM",
    "CamembertForTokenClassification",
    "CamembertForQuestionAnswering",
    "CamembertModel",
    "TFCamembertForMultipleChoice",
    "TFCamembertForTokenClassification",
    "TFCamembertForQuestionAnswering",
    "TFCamembertForSequenceClassification",
    "TFCamembertForMaskedLM",
    "TFCamembertModel",
    "TFCamembertForCausalLM",
    "DecisionTransformerModel",
    "GraphormerModel",
    "InformerModel",
    "JukeboxModel",
    "MarianForCausalLM",
    "MaskFormerSwinModel",
    "MaskFormerSwinBackbone",
    "MT5Model",
    "MT5ForConditionalGeneration",
    "UMT5ForConditionalGeneration",
    "TFMT5ForConditionalGeneration",
    "TFMT5Model",
    "QDQBertForSequenceClassification",
    "QDQBertForMaskedLM",
    "QDQBertModel",
    "QDQBertForTokenClassification",
    "QDQBertLMHeadModel",
    "QDQBertForMultipleChoice",
    "QDQBertForQuestionAnswering",
    "QDQBertForNextSentencePrediction",
    "ReformerModelWithLMHead",
    "RetriBertModel",
    "Speech2Text2ForCausalLM",
    "TimeSeriesTransformerModel",
    "TrajectoryTransformerModel",
    "TrOCRForCausalLM",
    "XLMProphetNetForConditionalGeneration",
    "XLMProphetNetForCausalLM",
    "XLMProphetNetModel",
    "XLMRobertaModel",
    "XLMRobertaForTokenClassification",
    "XLMRobertaForMultipleChoice",
    "XLMRobertaForMaskedLM",
    "XLMRobertaForCausalLM",
    "XLMRobertaForSequenceClassification",
    "XLMRobertaForQuestionAnswering",
    "TFXLMRobertaForSequenceClassification",
    "TFXLMRobertaForMaskedLM",
    "TFXLMRobertaForCausalLM",
    "TFXLMRobertaForQuestionAnswering",
    "TFXLMRobertaModel",
    "TFXLMRobertaForMultipleChoice",
    "TFXLMRobertaForTokenClassification",
}


def get_processor_types_from_config_class(config_class, allowed_mappings=None):
    

    
    def _to_tuple(x):
        if not isinstance(x, collections.abc.Sequence):
            x = (x,)
        else:
            x = tuple(x)
        return x

    if allowed_mappings is None:
        allowed_mappings = ["processor", "tokenizer", "image_processor", "feature_extractor"]

    processor_types = ()

    
    
    if config_class in PROCESSOR_MAPPING and "processor" in allowed_mappings:
        processor_types = _to_tuple(PROCESSOR_MAPPING[config_class])
    else:
        if config_class in TOKENIZER_MAPPING and "tokenizer" in allowed_mappings:
            processor_types = TOKENIZER_MAPPING[config_class]

        if config_class in IMAGE_PROCESSOR_MAPPING and "image_processor" in allowed_mappings:
            processor_types += _to_tuple(IMAGE_PROCESSOR_MAPPING[config_class])
        elif config_class in FEATURE_EXTRACTOR_MAPPING and "feature_extractor" in allowed_mappings:
            processor_types += _to_tuple(FEATURE_EXTRACTOR_MAPPING[config_class])

    
    
    

    
    processor_types = tuple(p for p in processor_types if p is not None)

    return processor_types


def get_architectures_from_config_class(config_class, arch_mappings, models_to_skip=None):
    
    
    
    
    
    
    
    architectures = set()

    if models_to_skip is None:
        models_to_skip = []
    models_to_skip = UNCONVERTIBLE_MODEL_ARCHITECTURES.union(models_to_skip)

    for mapping in arch_mappings:
        if config_class in mapping:
            models = mapping[config_class]
            models = tuple(models) if isinstance(models, collections.abc.Sequence) else (models,)
            for model in models:
                if model.__name__ not in models_to_skip:
                    architectures.add(model)

    architectures = tuple(architectures)

    return architectures


def get_config_class_from_processor_class(processor_class):
    

    processor_prefix = processor_class.__name__
    for postfix in ["TokenizerFast", "Tokenizer", "ImageProcessor", "FeatureExtractor", "Processor"]:
        processor_prefix = processor_prefix.replace(postfix, "")

    
    if processor_prefix == "Wav2Vec2CTC":
        processor_prefix = "Wav2Vec2"

    
    new_config_name = f"{processor_prefix}Config"
    new_config_class = getattr(transformers_module, new_config_name)

    return new_config_class


def build_processor(config_class, processor_class, allow_no_checkpoint=False):
    
    
    checkpoint = get_checkpoint_from_config_class(config_class)

    if checkpoint is None:
        
        
        config_class_from_processor_class = get_config_class_from_processor_class(processor_class)
        checkpoint = get_checkpoint_from_config_class(config_class_from_processor_class)

    processor = None
    try:
        processor = processor_class.from_pretrained(checkpoint)
    except Exception as e:
        logger.error(f"{e.__class__.__name__}: {e}")

    
    
    
    
    
    
    if (
        processor is None
        and checkpoint is not None
        and issubclass(processor_class, (PreTrainedTokenizerBase, AutoTokenizer))
    ):
        try:
            config = AutoConfig.from_pretrained(checkpoint)
        except Exception as e:
            logger.error(f"{e.__class__.__name__}: {e}")
            config = None
        if config is not None:
            if not isinstance(config, config_class):
                raise ValueError(
                    f"`config` (which is of type {config.__class__.__name__}) should be an instance of `config_class`"
                    f" ({config_class.__name__})!"
                )
            tokenizer_class = config.tokenizer_class
            new_processor_class = None
            if tokenizer_class is not None:
                new_processor_class = getattr(transformers_module, tokenizer_class)
                if new_processor_class != processor_class:
                    processor = build_processor(config_class, new_processor_class)
            
            
            if processor is None:
                new_processor_classes = get_processor_types_from_config_class(
                    config.__class__, allowed_mappings=["tokenizer"]
                )
                
                names = [
                    x.__name__.replace("Fast", "") for x in [processor_class, new_processor_class] if x is not None
                ]
                new_processor_classes = [
                    x for x in new_processor_classes if x is not None and x.__name__.replace("Fast", "") not in names
                ]
                if len(new_processor_classes) > 0:
                    new_processor_class = new_processor_classes[0]
                    
                    for x in new_processor_classes:
                        if x.__name__.endswith("Fast"):
                            new_processor_class = x
                            break
                    processor = build_processor(config_class, new_processor_class)

    if processor is None:
        
        if issubclass(processor_class, ProcessorMixin):
            attrs = {}
            for attr_name in processor_class.attributes:
                attrs[attr_name] = []
                
                
                
                attr_class_names = getattr(processor_class, f"{attr_name}_class")
                if not isinstance(attr_class_names, tuple):
                    attr_class_names = (attr_class_names,)

                for name in attr_class_names:
                    attr_class = getattr(transformers_module, name)
                    attr = build_processor(config_class, attr_class)
                    if attr is not None:
                        attrs[attr_name].append(attr)

            
            if all(len(v) > 0 for v in attrs.values()):
                try:
                    processor = processor_class(**{k: v[0] for k, v in attrs.items()})
                except Exception as e:
                    logger.error(f"{e.__class__.__name__}: {e}")
        else:
            
            
            
            config_class_from_processor_class = get_config_class_from_processor_class(processor_class)
            if config_class_from_processor_class != config_class:
                processor = build_processor(config_class_from_processor_class, processor_class)

    
    if (
        processor is None
        and allow_no_checkpoint
        and (issubclass(processor_class, BaseImageProcessor) or issubclass(processor_class, FeatureExtractionMixin))
    ):
        try:
            processor = processor_class()
        except Exception as e:
            logger.error(f"{e.__class__.__name__}: {e}")

    
    if processor is not None:
        if not (isinstance(processor, processor_class) or processor_class.__name__.startswith("Auto")):
            raise ValueError(
                f"`processor` (which is of type {processor.__class__.__name__}) should be an instance of"
                f" {processor_class.__name__} or an Auto class!"
            )

    return processor


def get_tiny_config(config_class, model_class=None, **model_tester_kwargs):
    
    model_type = config_class.model_type

    
    
    
    config_source_file = inspect.getsourcefile(config_class)
    
    modeling_name = config_source_file.split(os.path.sep)[-1].replace("configuration_", "").replace(".py", "")

    try:
        print("Importing", model_type_to_module_name(model_type))
        module_name = model_type_to_module_name(model_type)
        if not modeling_name.startswith(module_name):
            raise ValueError(f"{modeling_name} doesn't start with {module_name}!")
        test_file = os.path.join("tests", "models", module_name, f"test_modeling_{modeling_name}.py")
        models_to_model_testers = get_model_to_tester_mapping(test_file)
        
        model_tester_class = None
        tester_classes = []
        if model_class is not None:
            tester_classes = get_tester_classes_for_model(test_file, model_class)
        else:
            for _tester_classes in models_to_model_testers.values():
                tester_classes.extend(_tester_classes)
        if len(tester_classes) > 0:
            
            
            
            
            model_tester_class = sorted(tester_classes, key=lambda x: (len(x.__name__), x.__name__))[0]
    except ModuleNotFoundError:
        error = f"Tiny config not created for {model_type} - cannot find the testing module from the model name."
        raise ValueError(error)

    if model_tester_class is None:
        error = f"Tiny config not created for {model_type} - no model tester is found in the testing module."
        raise ValueError(error)

    
    

    if "vocab_size" in model_tester_kwargs:
        if "text_kwargs" in inspect.signature(model_tester_class.__init__).parameters.keys():
            vocab_size = model_tester_kwargs.pop("vocab_size")
            model_tester_kwargs["text_kwargs"] = {"vocab_size": vocab_size}

    
    model_tester = model_tester_class(parent=None, **model_tester_kwargs)

    if hasattr(model_tester, "get_pipeline_config"):
        config = model_tester.get_pipeline_config()
    elif hasattr(model_tester, "prepare_config_and_inputs"):
        
        
        config = model_tester.prepare_config_and_inputs()[0]
    elif hasattr(model_tester, "get_config"):
        config = model_tester.get_config()
    else:
        error = (
            f"Tiny config not created for {model_type} - the model tester {model_tester_class.__name__} lacks"
            " necessary method to create config."
        )
        raise ValueError(error)

    
    
    max_positions = []
    for key in ["max_position_embeddings", "max_source_positions", "max_target_positions"]:
        if getattr(config, key, 0) > 0:
            max_positions.append(getattr(config, key))
        if getattr(config, "text_config", None) is not None:
            if getattr(config.text_config, key, None) is not None:
                max_positions.append(getattr(config.text_config, key))
    if len(max_positions) > 0:
        max_position = max(200, min(max_positions))
        for key in ["max_position_embeddings", "max_source_positions", "max_target_positions"]:
            if getattr(config, key, 0) > 0:
                setattr(config, key, max_position)
            if getattr(config, "text_config", None) is not None:
                if getattr(config.text_config, key, None) is not None:
                    setattr(config.text_config, key, max_position)

    return config


def convert_tokenizer(tokenizer_fast: PreTrainedTokenizerFast):
    new_tokenizer = tokenizer_fast.train_new_from_iterator(
        data["training_ds"]["text"], TARGET_VOCAB_SIZE, show_progress=False
    )

    
    if not isinstance(new_tokenizer, LayoutLMv3TokenizerFast):
        new_tokenizer(data["testing_ds"]["text"])

    return new_tokenizer


def convert_feature_extractor(feature_extractor, tiny_config):
    to_convert = False
    kwargs = {}
    if hasattr(tiny_config, "image_size"):
        kwargs["size"] = tiny_config.image_size
        kwargs["crop_size"] = tiny_config.image_size
        to_convert = True
    elif (
        hasattr(tiny_config, "vision_config")
        and tiny_config.vision_config is not None
        and hasattr(tiny_config.vision_config, "image_size")
    ):
        kwargs["size"] = tiny_config.vision_config.image_size
        kwargs["crop_size"] = tiny_config.vision_config.image_size
        to_convert = True

    
    if hasattr(tiny_config, "input_feat_per_channel"):
        kwargs["feature_size"] = tiny_config.input_feat_per_channel
        kwargs["num_mel_bins"] = tiny_config.input_feat_per_channel
        to_convert = True

    if to_convert:
        feature_extractor = feature_extractor.__class__(**kwargs)

    return feature_extractor


def convert_processors(processors, tiny_config, output_folder, result):
    

    def _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=False):
        
        
        if fast_tokenizer is not None and slow_tokenizer is not None:
            if fast_tokenizer.vocab_size != slow_tokenizer.vocab_size:
                warning_messagae = (
                    "The fast/slow tokenizers "
                    f"({fast_tokenizer.__class__.__name__}/{slow_tokenizer.__class__.__name__}) have different "
                    "vocabulary size: "
                    f"fast_tokenizer.vocab_size = {fast_tokenizer.vocab_size} and "
                    f"slow_tokenizer.vocab_size = {slow_tokenizer.vocab_size}."
                )
                result["warnings"].append(warning_messagae)
                if not keep_fast_tokenizer:
                    fast_tokenizer = None
                slow_tokenizer = None

        
        if fast_tokenizer is not None and slow_tokenizer is not None:
            if len(fast_tokenizer) != len(slow_tokenizer):
                warning_messagae = (
                    f"The fast/slow tokenizers () have different length: "
                    f"len(fast_tokenizer) = {len(fast_tokenizer)} and "
                    f"len(slow_tokenizer) = {len(slow_tokenizer)}."
                )
                result["warnings"].append(warning_messagae)
                if not keep_fast_tokenizer:
                    fast_tokenizer = None
                slow_tokenizer = None

        return fast_tokenizer, slow_tokenizer

    tokenizers = []
    feature_extractors = []
    for processor in processors:
        if isinstance(processor, PreTrainedTokenizerBase):
            if processor.__class__.__name__ not in {x.__class__.__name__ for x in tokenizers}:
                tokenizers.append(processor)
        elif isinstance(processor, BaseImageProcessor):
            if processor.__class__.__name__ not in {x.__class__.__name__ for x in feature_extractors}:
                feature_extractors.append(processor)
        elif isinstance(processor, FeatureExtractionMixin):
            if processor.__class__.__name__ not in {x.__class__.__name__ for x in feature_extractors}:
                feature_extractors.append(processor)
        elif isinstance(processor, ProcessorMixin):
            if hasattr(processor, "tokenizer"):
                if processor.tokenizer.__class__.__name__ not in {x.__class__.__name__ for x in tokenizers}:
                    tokenizers.append(processor.tokenizer)
            
            if hasattr(processor, "image_processor"):
                if processor.image_processor.__class__.__name__ not in {
                    x.__class__.__name__ for x in feature_extractors
                }:
                    feature_extractors.append(processor.image_processor)
            elif hasattr(processor, "feature_extractor"):
                if processor.feature_extractor.__class__.__name__ not in {
                    x.__class__.__name__ for x in feature_extractors
                }:
                    feature_extractors.append(processor.feature_extractor)

    
    num_types = len({x.__class__.__name__ for x in feature_extractors})
    if num_types >= 2:
        raise ValueError(f"`feature_extractors` should contain at most 1 type, but it contains {num_types} types!")
    num_types = len({x.__class__.__name__.replace("Fast", "") for x in tokenizers})
    if num_types >= 2:
        raise ValueError(f"`tokenizers` should contain at most 1 tokenizer type, but it contains {num_types} types!")

    fast_tokenizer = None
    slow_tokenizer = None

    for tokenizer in tokenizers:
        if isinstance(tokenizer, PreTrainedTokenizerFast):
            fast_tokenizer = tokenizer
        else:
            slow_tokenizer = tokenizer

    
    
    
    
    
    
    
    fast_tokenizer, slow_tokenizer = _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=True)
    original_fast_tokenizer, original_slow_tokenizer = fast_tokenizer, slow_tokenizer

    if fast_tokenizer:
        try:
            
            
            if fast_tokenizer.vocab_size > TARGET_VOCAB_SIZE:
                fast_tokenizer = convert_tokenizer(fast_tokenizer)
        except Exception:
            result["warnings"].append(
                (
                    f"Failed to convert the fast tokenizer for {fast_tokenizer.__class__.__name__}.",
                    traceback.format_exc(),
                )
            )

    
    if fast_tokenizer:
        
        try:
            
            with tempfile.TemporaryDirectory() as tmpdir:
                fast_tokenizer.save_pretrained(tmpdir)
                try:
                    slow_tokenizer = AutoTokenizer.from_pretrained(tmpdir, use_fast=False)
                except Exception:
                    result["warnings"].append(
                        (
                            f"Failed to load the slow tokenizer saved from {fast_tokenizer.__class__.__name__}.",
                            traceback.format_exc(),
                        )
                    )
                    
                    slow_tokenizer = None
        except Exception:
            result["warnings"].append(
                (
                    f"Failed to save the fast tokenizer for {fast_tokenizer.__class__.__name__}.",
                    traceback.format_exc(),
                )
            )
            fast_tokenizer = None

    
    
    fast_tokenizer, slow_tokenizer = _sanity_check(fast_tokenizer, slow_tokenizer, keep_fast_tokenizer=False)

    
    if (original_fast_tokenizer is not None and fast_tokenizer is None) or (
        original_slow_tokenizer is not None and slow_tokenizer is None
    ):
        warning_messagae = (
            "There are some issues when converting the fast/slow tokenizers. The original tokenizers from the Hub "
            " will be used instead."
        )
        result["warnings"].append(warning_messagae)
        
        fast_tokenizer = original_fast_tokenizer
        slow_tokenizer = original_slow_tokenizer

    
    if fast_tokenizer:
        
        with tempfile.TemporaryDirectory() as tmpdir:
            try:
                fast_tokenizer.save_pretrained(tmpdir)
            except Exception:
                result["warnings"].append(
                    (
                        f"Failed to save the fast tokenizer for {fast_tokenizer.__class__.__name__}.",
                        traceback.format_exc(),
                    )
                )
                fast_tokenizer = None
    
    if slow_tokenizer:
        
        with tempfile.TemporaryDirectory() as tmpdir:
            try:
                slow_tokenizer.save_pretrained(tmpdir)
            except Exception:
                result["warnings"].append(
                    (
                        f"Failed to save the slow tokenizer for {slow_tokenizer.__class__.__name__}.",
                        traceback.format_exc(),
                    )
                )
                slow_tokenizer = None

    
    try:
        feature_extractors = [convert_feature_extractor(p, tiny_config) for p in feature_extractors]
    except Exception:
        result["warnings"].append(
            (
                "Failed to convert feature extractors.",
                traceback.format_exc(),
            )
        )
        feature_extractors = []

    if hasattr(tiny_config, "max_position_embeddings") and tiny_config.max_position_embeddings > 0:
        if fast_tokenizer is not None:
            if fast_tokenizer.__class__.__name__ in [
                "RobertaTokenizerFast",
                "XLMRobertaTokenizerFast",
                "LongformerTokenizerFast",
                "MPNetTokenizerFast",
            ]:
                fast_tokenizer.model_max_length = tiny_config.max_position_embeddings - 2
            else:
                fast_tokenizer.model_max_length = tiny_config.max_position_embeddings
        if slow_tokenizer is not None:
            if slow_tokenizer.__class__.__name__ in [
                "RobertaTokenizer",
                "XLMRobertaTokenizer",
                "LongformerTokenizer",
                "MPNetTokenizer",
            ]:
                slow_tokenizer.model_max_length = tiny_config.max_position_embeddings - 2
            else:
                slow_tokenizer.model_max_length = tiny_config.max_position_embeddings

    processors = [fast_tokenizer, slow_tokenizer] + feature_extractors
    processors = [p for p in processors if p is not None]
    for p in processors:
        p.save_pretrained(output_folder)

    return processors


def get_checkpoint_dir(output_dir, model_arch):
    

    arch_name = model_arch.__name__
    if arch_name.startswith("TF"):
        arch_name = arch_name[2:]
    elif arch_name.startswith("Flax"):
        arch_name = arch_name[4:]

    return os.path.join(output_dir, arch_name)


def build_model(model_arch, tiny_config, output_dir):
    

    checkpoint_dir = get_checkpoint_dir(output_dir, model_arch)

    processor_output_dir = os.path.join(output_dir, "processors")
    
    if os.path.isdir(processor_output_dir):
        shutil.copytree(processor_output_dir, checkpoint_dir, dirs_exist_ok=True)

    tiny_config = copy.deepcopy(tiny_config)

    if any(model_arch.__name__.endswith(x) for x in ["ForCausalLM", "LMHeadModel"]):
        tiny_config.is_encoder_decoder = False
        tiny_config.is_decoder = True

    model = model_arch(config=tiny_config)
    model.save_pretrained(checkpoint_dir)
    model.from_pretrained(checkpoint_dir)

    return model


def fill_result_with_error(result, error, trace, models_to_create):
    
    error = (error, trace)
    result["error"] = error
    for framework in FRAMEWORKS:
        if framework in models_to_create:
            result[framework] = {}
            for model_arch in models_to_create[framework]:
                result[framework][model_arch.__name__] = {"model": None, "checkpoint": None, "error": error}

    result["processor"] = {p.__class__.__name__: p.__class__.__name__ for p in result["processor"].values()}


def upload_model(model_dir, organization, token):
    

    arch_name = model_dir.split(os.path.sep)[-1]
    repo_name = f"tiny-random-{arch_name}"
    repo_id = f"{organization}/{repo_name}"

    repo_exist = False
    error = None
    try:
        create_repo(repo_id=repo_id, exist_ok=False, repo_type="model", token=token)
    except Exception as e:
        error = e
        if "You already created" in str(e):
            error = None
            logger.warning("Remote repository exists and will be cloned.")
            repo_exist = True
            try:
                create_repo(repo_id=repo_id, exist_ok=True, repo_type="model", token=token)
            except Exception as e:
                error = e
    if error is not None:
        raise error

    with tempfile.TemporaryDirectory() as tmpdir:
        repo = Repository(local_dir=tmpdir, clone_from=repo_id, token=token)
        repo.git_pull()
        shutil.copytree(model_dir, tmpdir, dirs_exist_ok=True)

        if repo_exist:
            
            hub_pr_url = upload_folder(
                folder_path=model_dir,
                repo_id=repo_id,
                repo_type="model",
                commit_message=f"Update tiny models for {arch_name}",
                commit_description=f"Upload tiny models for {arch_name}",
                create_pr=True,
                token=token,
            )
            logger.warning(f"PR open in {hub_pr_url}.")
            
        else:
            
            repo.git_add(auto_lfs_track=True)
            repo.git_commit(f"Upload tiny models for {arch_name}")
            repo.git_push(blocking=True)  
            logger.warning(f"Tiny models {arch_name} pushed to {repo_id}.")


def build_composite_models(config_class, output_dir):
    import tempfile

    from transformers import (
        BertConfig,
        BertLMHeadModel,
        BertModel,
        BertTokenizer,
        BertTokenizerFast,
        EncoderDecoderModel,
        GPT2Config,
        GPT2LMHeadModel,
        GPT2Tokenizer,
        GPT2TokenizerFast,
        SpeechEncoderDecoderModel,
        TFEncoderDecoderModel,
        TFVisionEncoderDecoderModel,
        TFVisionTextDualEncoderModel,
        VisionEncoderDecoderModel,
        VisionTextDualEncoderModel,
        ViTConfig,
        ViTFeatureExtractor,
        ViTModel,
        Wav2Vec2Config,
        Wav2Vec2Model,
        Wav2Vec2Processor,
    )

    
    result = {"error": None, "warnings": []}

    if config_class.model_type == "encoder-decoder":
        encoder_config_class = BertConfig
        decoder_config_class = BertConfig
        encoder_processor = (BertTokenizerFast, BertTokenizer)
        decoder_processor = (BertTokenizerFast, BertTokenizer)
        encoder_class = BertModel
        decoder_class = BertLMHeadModel
        model_class = EncoderDecoderModel
        tf_model_class = TFEncoderDecoderModel
    elif config_class.model_type == "vision-encoder-decoder":
        encoder_config_class = ViTConfig
        decoder_config_class = GPT2Config
        encoder_processor = (ViTFeatureExtractor,)
        decoder_processor = (GPT2TokenizerFast, GPT2Tokenizer)
        encoder_class = ViTModel
        decoder_class = GPT2LMHeadModel
        model_class = VisionEncoderDecoderModel
        tf_model_class = TFVisionEncoderDecoderModel
    elif config_class.model_type == "speech-encoder-decoder":
        encoder_config_class = Wav2Vec2Config
        decoder_config_class = BertConfig
        encoder_processor = (Wav2Vec2Processor,)
        decoder_processor = (BertTokenizerFast, BertTokenizer)
        encoder_class = Wav2Vec2Model
        decoder_class = BertLMHeadModel
        model_class = SpeechEncoderDecoderModel
        tf_model_class = None
    elif config_class.model_type == "vision-text-dual-encoder":
        
        encoder_config_class = ViTConfig
        decoder_config_class = BertConfig
        encoder_processor = (ViTFeatureExtractor,)
        decoder_processor = (BertTokenizerFast, BertTokenizer)
        encoder_class = ViTModel
        decoder_class = BertModel
        model_class = VisionTextDualEncoderModel
        tf_model_class = TFVisionTextDualEncoderModel

    with tempfile.TemporaryDirectory() as tmpdir:
        try:
            
            models_to_create = {"processor": encoder_processor, "pytorch": (encoder_class,), "tensorflow": []}
            encoder_output_dir = os.path.join(tmpdir, "encoder")
            build(encoder_config_class, models_to_create, encoder_output_dir)

            
            models_to_create = {"processor": decoder_processor, "pytorch": (decoder_class,), "tensorflow": []}
            decoder_output_dir = os.path.join(tmpdir, "decoder")
            build(decoder_config_class, models_to_create, decoder_output_dir)

            
            encoder_path = os.path.join(encoder_output_dir, encoder_class.__name__)
            decoder_path = os.path.join(decoder_output_dir, decoder_class.__name__)

            if config_class.model_type != "vision-text-dual-encoder":
                
                
                decoder_config = decoder_config_class.from_pretrained(decoder_path)
                decoder_config.is_decoder = True
                decoder_config.add_cross_attention = True
                model = model_class.from_encoder_decoder_pretrained(
                    encoder_path,
                    decoder_path,
                    decoder_config=decoder_config,
                )
            elif config_class.model_type == "vision-text-dual-encoder":
                model = model_class.from_vision_text_pretrained(encoder_path, decoder_path)

            model_path = os.path.join(
                output_dir,
                f"{model_class.__name__}-{encoder_config_class.model_type}-{decoder_config_class.model_type}",
            )
            model.save_pretrained(model_path)

            if tf_model_class is not None:
                model = tf_model_class.from_pretrained(model_path)
                model.save_pretrained(model_path)

            
            encoder_processor_path = os.path.join(encoder_output_dir, "processors")
            decoder_processor_path = os.path.join(decoder_output_dir, "processors")
            if os.path.isdir(encoder_processor_path):
                shutil.copytree(encoder_processor_path, model_path, dirs_exist_ok=True)
            if os.path.isdir(decoder_processor_path):
                shutil.copytree(decoder_processor_path, model_path, dirs_exist_ok=True)

            
            result["processor"] = {x.__name__: x.__name__ for x in encoder_processor + decoder_processor}

            result["pytorch"] = {model_class.__name__: {"model": model_class.__name__, "checkpoint": model_path}}

            result["tensorflow"] = {}
            if tf_model_class is not None:
                result["tensorflow"] = {
                    tf_model_class.__name__: {"model": tf_model_class.__name__, "checkpoint": model_path}
                }
        except Exception:
            result["error"] = (
                f"Failed to build models for {config_class.__name__}.",
                traceback.format_exc(),
            )

    if not result["error"]:
        del result["error"]
    if not result["warnings"]:
        del result["warnings"]

    return result


def get_token_id_from_tokenizer(token_id_name, tokenizer, original_token_id):
    

    token_id = original_token_id

    if not token_id_name.endswith("_token_id"):
        raise ValueError(f"`token_id_name` is {token_id_name}, which doesn't end with `_token_id`!")

    token = getattr(tokenizer, token_id_name.replace("_token_id", "_token"), None)
    if token is not None:
        if isinstance(tokenizer, PreTrainedTokenizerFast):
            token_id = tokenizer._convert_token_to_id_with_added_voc(token)
        else:
            token_id = tokenizer._convert_token_to_id(token)

    return token_id


def get_config_overrides(config_class, processors):
    
    if config_class.__name__ == "BarkConfig":
        return {}

    config_overrides = {}

    
    tokenizer = None
    for processor in processors:
        if isinstance(processor, PreTrainedTokenizerFast):
            tokenizer = processor
            break
        elif isinstance(processor, PreTrainedTokenizer):
            tokenizer = processor

    if tokenizer is None:
        return config_overrides

    
    
    
    vocab_size = len(tokenizer)

    
    
    if config_class.__name__ == "GPTSanJapaneseConfig":
        vocab_size += 2

    config_overrides["vocab_size"] = vocab_size

    
    model_tester_kwargs = {"vocab_size": vocab_size}
    
    if config_class.__name__ == "FSMTConfig":
        del model_tester_kwargs["vocab_size"]
        model_tester_kwargs["src_vocab_size"] = tokenizer.src_vocab_size
        model_tester_kwargs["tgt_vocab_size"] = tokenizer.tgt_vocab_size

    _tiny_config = get_tiny_config(config_class, **model_tester_kwargs)

    
    if hasattr(_tiny_config, "text_config"):
        _tiny_config = _tiny_config.text_config

    
    for attr in dir(_tiny_config):
        if attr.endswith("_token_id"):
            token_id = getattr(_tiny_config, attr)
            if token_id is not None:
                
                token_id = get_token_id_from_tokenizer(attr, tokenizer, original_token_id=token_id)
                config_overrides[attr] = token_id

    if config_class.__name__ == "FSMTConfig":
        config_overrides["src_vocab_size"] = tokenizer.src_vocab_size
        config_overrides["tgt_vocab_size"] = tokenizer.tgt_vocab_size
        
        config_overrides["decoder"] = configuration_fsmt.DecoderConfig(
            vocab_size=tokenizer.tgt_vocab_size, bos_token_id=config_overrides["eos_token_id"]
        )

    return config_overrides


def build(config_class, models_to_create, output_dir):
    
    if data["training_ds"] is None or data["testing_ds"] is None:
        ds = load_dataset("wikitext", "wikitext-2-raw-v1")
        data["training_ds"] = ds["train"]
        data["testing_ds"] = ds["test"]

    if config_class.model_type in [
        "encoder-decoder",
        "vision-encoder-decoder",
        "speech-encoder-decoder",
        "vision-text-dual-encoder",
    ]:
        return build_composite_models(config_class, output_dir)

    result = {k: {} for k in models_to_create}

    
    result["error"] = None
    result["warnings"] = []

    
    processor_classes = models_to_create["processor"]

    if len(processor_classes) == 0:
        error = f"No processor class could be found in {config_class.__name__}."
        fill_result_with_error(result, error, None, models_to_create)
        logger.error(result["error"][0])
        return result

    for processor_class in processor_classes:
        try:
            processor = build_processor(config_class, processor_class, allow_no_checkpoint=True)
            if processor is not None:
                result["processor"][processor_class] = processor
        except Exception:
            error = f"Failed to build processor for {processor_class.__name__}."
            trace = traceback.format_exc()
            fill_result_with_error(result, error, trace, models_to_create)
            logger.error(result["error"][0])
            return result

    if len(result["processor"]) == 0:
        error = f"No processor could be built for {config_class.__name__}."
        fill_result_with_error(result, error, None, models_to_create)
        logger.error(result["error"][0])
        return result

    try:
        tiny_config = get_tiny_config(config_class)
    except Exception as e:
        error = f"Failed to get tiny config for {config_class.__name__}: {e}"
        trace = traceback.format_exc()
        fill_result_with_error(result, error, trace, models_to_create)
        logger.error(result["error"][0])
        return result

    
    processors = list(result["processor"].values())
    processor_output_folder = os.path.join(output_dir, "processors")
    try:
        processors = convert_processors(processors, tiny_config, processor_output_folder, result)
    except Exception:
        error = "Failed to convert the processors."
        trace = traceback.format_exc()
        result["warnings"].append((error, trace))

    if len(processors) == 0:
        error = f"No processor is returned by `convert_processors` for {config_class.__name__}."
        fill_result_with_error(result, error, None, models_to_create)
        logger.error(result["error"][0])
        return result

    try:
        config_overrides = get_config_overrides(config_class, processors)
    except Exception as e:
        error = f"Failure occurs while calling `get_config_overrides`: {e}"
        trace = traceback.format_exc()
        fill_result_with_error(result, error, trace, models_to_create)
        logger.error(result["error"][0])
        return result

    
    if "vocab_size" in config_overrides:
        result["vocab_size"] = config_overrides["vocab_size"]

    
    for k, v in config_overrides.items():
        if hasattr(tiny_config, k):
            setattr(tiny_config, k, v)
        
        
        
        if (
            hasattr(tiny_config, "text_config")
            and tiny_config.text_config is not None
            and hasattr(tiny_config.text_config, k)
        ):
            setattr(tiny_config.text_config, k, v)
            
            
            if hasattr(tiny_config, "text_config_dict"):
                tiny_config.text_config_dict[k] = v

    if result["warnings"]:
        logger.warning(result["warnings"][0][0])

    
    result["processor"] = {type(p).__name__: p.__class__.__name__ for p in processors}

    for pytorch_arch in models_to_create["pytorch"]:
        result["pytorch"][pytorch_arch.__name__] = {}
        error = None
        try:
            model = build_model(pytorch_arch, tiny_config, output_dir=output_dir)
        except Exception as e:
            model = None
            error = f"Failed to create the pytorch model for {pytorch_arch}: {e}"
            trace = traceback.format_exc()

        result["pytorch"][pytorch_arch.__name__]["model"] = model.__class__.__name__ if model is not None else None
        result["pytorch"][pytorch_arch.__name__]["checkpoint"] = (
            get_checkpoint_dir(output_dir, pytorch_arch) if model is not None else None
        )
        if error is not None:
            result["pytorch"][pytorch_arch.__name__]["error"] = (error, trace)
            logger.error(f"{pytorch_arch.__name__}: {error}")

    for tensorflow_arch in models_to_create["tensorflow"]:
        
        pt_arch_name = tensorflow_arch.__name__[2:]  
        pt_arch = getattr(transformers_module, pt_arch_name)

        result["tensorflow"][tensorflow_arch.__name__] = {}
        error = None
        if pt_arch.__name__ in result["pytorch"] and result["pytorch"][pt_arch.__name__]["checkpoint"] is not None:
            ckpt = get_checkpoint_dir(output_dir, pt_arch)
            
            try:
                model = tensorflow_arch.from_pretrained(ckpt)
                model.save_pretrained(ckpt)
            except Exception as e:
                
                model = None
                error = f"Failed to convert the pytorch model to the tensorflow model for {pt_arch}: {e}"
                trace = traceback.format_exc()
        else:
            try:
                model = build_model(tensorflow_arch, tiny_config, output_dir=output_dir)
            except Exception as e:
                model = None
                error = f"Failed to create the tensorflow model for {tensorflow_arch}: {e}"
                trace = traceback.format_exc()

        result["tensorflow"][tensorflow_arch.__name__]["model"] = (
            model.__class__.__name__ if model is not None else None
        )
        result["tensorflow"][tensorflow_arch.__name__]["checkpoint"] = (
            get_checkpoint_dir(output_dir, tensorflow_arch) if model is not None else None
        )
        if error is not None:
            result["tensorflow"][tensorflow_arch.__name__]["error"] = (error, trace)
            logger.error(f"{tensorflow_arch.__name__}: {error}")

    if not result["error"]:
        del result["error"]
    if not result["warnings"]:
        del result["warnings"]

    return result


def build_tiny_model_summary(results, organization=None, token=None):
    
    tiny_model_summary = {}
    for config_name in results:
        processors = [key for key, value in results[config_name]["processor"].items()]
        tokenizer_classes = sorted([x for x in processors if x.endswith("TokenizerFast") or x.endswith("Tokenizer")])
        processor_classes = sorted([x for x in processors if x not in tokenizer_classes])
        for framework in FRAMEWORKS:
            if framework not in results[config_name]:
                continue
            for arch_name in results[config_name][framework]:
                model_classes = [arch_name]
                base_arch_name = arch_name[2:] if arch_name.startswith("TF") else arch_name
                
                if results[config_name][framework][arch_name]["model"] is None:
                    model_classes = []
                if base_arch_name not in tiny_model_summary:
                    tiny_model_summary[base_arch_name] = {}
                tiny_model_summary[base_arch_name].update(
                    {
                        "tokenizer_classes": tokenizer_classes,
                        "processor_classes": processor_classes,
                    }
                )
                tiny_model_summary[base_arch_name]["model_classes"] = sorted(
                    tiny_model_summary[base_arch_name].get("model_classes", []) + model_classes
                )
                if organization is not None:
                    repo_name = f"tiny-random-{base_arch_name}"
                    
                    if base_arch_name in COMPOSITE_MODELS:
                        repo_name = f"tiny-random-{COMPOSITE_MODELS[base_arch_name]}"
                    repo_id = f"{organization}/{repo_name}"
                    try:
                        commit_hash = hf_api.repo_info(repo_id, token=token).sha
                    except Exception:
                        
                        logger.warning(f"Failed to get information for {repo_id}.\n{traceback.format_exc()}")
                        del tiny_model_summary[base_arch_name]
                        continue
                    tiny_model_summary[base_arch_name]["sha"] = commit_hash

    return tiny_model_summary


def build_failed_report(results, include_warning=True):
    failed_results = {}
    for config_name in results:
        if "error" in results[config_name]:
            if config_name not in failed_results:
                failed_results[config_name] = {}
            failed_results[config_name] = {"error": results[config_name]["error"]}

        if include_warning and "warnings" in results[config_name]:
            if config_name not in failed_results:
                failed_results[config_name] = {}
            failed_results[config_name]["warnings"] = results[config_name]["warnings"]

        for framework in FRAMEWORKS:
            if framework not in results[config_name]:
                continue
            for arch_name in results[config_name][framework]:
                if "error" in results[config_name][framework][arch_name]:
                    if config_name not in failed_results:
                        failed_results[config_name] = {}
                    if framework not in failed_results[config_name]:
                        failed_results[config_name][framework] = {}
                    if arch_name not in failed_results[config_name][framework]:
                        failed_results[config_name][framework][arch_name] = {}
                    error = results[config_name][framework][arch_name]["error"]
                    failed_results[config_name][framework][arch_name]["error"] = error

    return failed_results


def build_simple_report(results):
    text = ""
    failed_text = ""
    for config_name in results:
        for framework in FRAMEWORKS:
            if framework not in results[config_name]:
                continue
            for arch_name in results[config_name][framework]:
                if "error" in results[config_name][framework][arch_name]:
                    result = results[config_name][framework][arch_name]["error"]
                    failed_text += f"{arch_name}: {result[0]}\n"
                else:
                    result = ("OK",)
                text += f"{arch_name}: {result[0]}\n"

    return text, failed_text


def update_tiny_model_summary_file(report_path):
    with open(os.path.join(report_path, "tiny_model_summary.json")) as fp:
        new_data = json.load(fp)
    with open("tests/utils/tiny_model_summary.json") as fp:
        data = json.load(fp)
    for key, value in new_data.items():
        if key not in data:
            data[key] = value
        else:
            for attr in ["tokenizer_classes", "processor_classes", "model_classes"]:
                
                data[key][attr].extend(value[attr])
            new_sha = value.get("sha", None)
            if new_sha is not None:
                data[key]["sha"] = new_sha

    updated_data = {}
    for key in sorted(data.keys()):
        updated_data[key] = {}
        for attr, value in data[key].items():
            
            updated_data[key][attr] = sorted(set(value)) if attr != "sha" else value

    with open(os.path.join(report_path, "updated_tiny_model_summary.json"), "w") as fp:
        json.dump(updated_data, fp, indent=4, ensure_ascii=False)


def create_tiny_models(
    output_path,
    all,
    model_types,
    models_to_skip,
    no_check,
    upload,
    organization,
    token,
    num_workers=1,
):
    clone_path = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
    if os.getcwd() != clone_path:
        raise ValueError(f"This script should be run from the root of the clone of `transformers` {clone_path}")

    report_path = os.path.join(output_path, "reports")
    os.makedirs(report_path)

    _pytorch_arch_mappings = [
        x
        for x in dir(transformers_module)
        if x.startswith("MODEL_") and x.endswith("_MAPPING") and x != "MODEL_NAMES_MAPPING"
    ]
    _tensorflow_arch_mappings = [
        x for x in dir(transformers_module) if x.startswith("TF_MODEL_") and x.endswith("_MAPPING")
    ]

    pytorch_arch_mappings = [getattr(transformers_module, x) for x in _pytorch_arch_mappings]
    tensorflow_arch_mappings = [getattr(transformers_module, x) for x in _tensorflow_arch_mappings]

    config_classes = CONFIG_MAPPING.values()
    if not all:
        config_classes = [CONFIG_MAPPING[model_type] for model_type in model_types]

    
    processor_type_map = {c: get_processor_types_from_config_class(c) for c in config_classes}

    to_create = {}
    for c in config_classes:
        processors = processor_type_map[c]
        models = get_architectures_from_config_class(c, pytorch_arch_mappings, models_to_skip)
        tf_models = get_architectures_from_config_class(c, tensorflow_arch_mappings, models_to_skip)
        if len(models) + len(tf_models) > 0:
            to_create[c] = {"processor": processors, "pytorch": models, "tensorflow": tf_models}

    results = {}
    if num_workers <= 1:
        for c, models_to_create in list(to_create.items()):
            print(f"Create models for {c.__name__} ...")
            result = build(c, models_to_create, output_dir=os.path.join(output_path, c.model_type))
            results[c.__name__] = result
            print("=" * 40)
    else:
        all_build_args = []
        for c, models_to_create in list(to_create.items()):
            all_build_args.append((c, models_to_create, os.path.join(output_path, c.model_type)))
        with multiprocessing.Pool() as pool:
            results = pool.starmap(build, all_build_args)
            results = {buid_args[0].__name__: result for buid_args, result in zip(all_build_args, results)}

    if upload:
        if organization is None:
            raise ValueError("The argument `organization` could not be `None`. No model is uploaded")

        to_upload = []
        for model_type in os.listdir(output_path):
            
            if model_type == "reports":
                continue
            for arch in os.listdir(os.path.join(output_path, model_type)):
                if arch == "processors":
                    continue
                to_upload.append(os.path.join(output_path, model_type, arch))
        to_upload = sorted(to_upload)

        upload_results = {}
        if len(to_upload) > 0:
            for model_dir in to_upload:
                try:
                    upload_model(model_dir, organization, token)
                except Exception as e:
                    error = f"Failed to upload {model_dir}. {e.__class__.__name__}: {e}"
                    logger.error(error)
                    upload_results[model_dir] = error

        with open(os.path.join(report_path, "failed_uploads.json"), "w") as fp:
            json.dump(upload_results, fp, indent=4)

    
    
    
    
    tiny_model_summary = build_tiny_model_summary(results, organization=organization, token=token)
    with open(os.path.join(report_path, "tiny_model_summary.json"), "w") as fp:
        json.dump(tiny_model_summary, fp, indent=4)

    with open(os.path.join(report_path, "tiny_model_creation_report.json"), "w") as fp:
        json.dump(results, fp, indent=4)

    
    
    failed_results = build_failed_report(results)
    with open(os.path.join(report_path, "failed_report.json"), "w") as fp:
        json.dump(failed_results, fp, indent=4)

    simple_report, failed_report = build_simple_report(results)
    
    
    with open(os.path.join(report_path, "simple_report.txt"), "w") as fp:
        fp.write(simple_report)

    
    with open(os.path.join(report_path, "simple_failed_report.txt"), "w") as fp:
        fp.write(failed_report)

    update_tiny_model_summary_file(report_path=os.path.join(output_path, "reports"))


if __name__ == "__main__":
    
    multiprocessing.set_start_method("spawn")

    def list_str(values):
        return values.split(",")

    parser = argparse.ArgumentParser()
    parser.add_argument("--all", action="store_true", help="Will create all tiny models.")
    parser.add_argument(
        "--no_check",
        action="store_true",
        help="If set, will not check the validity of architectures. Use with caution.",
    )
    parser.add_argument(
        "-m",
        "--model_types",
        type=list_str,
        help="Comma-separated list of model type(s) from which the tiny models will be created.",
    )
    parser.add_argument(
        "--models_to_skip",
        type=list_str,
        help=(
            "Comma-separated list of model class names(s) from which the tiny models won't be created.\nThis is usually "
            "the list of model classes that have their tiny versions already uploaded to the Hub."
        ),
    )
    parser.add_argument("--upload", action="store_true", help="If to upload the created tiny models to the Hub.")
    parser.add_argument(
        "--organization",
        default=None,
        type=str,
        help="The organization on the Hub to which the tiny models will be uploaded.",
    )
    parser.add_argument(
        "--token", default=None, type=str, help="A valid authentication token for HuggingFace Hub with write access."
    )
    parser.add_argument("output_path", type=Path, help="Path indicating where to store generated model.")
    parser.add_argument("--num_workers", default=1, type=int, help="The number of workers to run.")

    args = parser.parse_args()

    if not args.all and not args.model_types:
        raise ValueError("Please provide at least one model type or pass `--all` to export all architectures.")

    create_tiny_models(
        args.output_path,
        args.all,
        args.model_types,
        args.models_to_skip,
        args.no_check,
        args.upload,
        args.organization,
        args.token,
        args.num_workers,
    )