codingutf8 2022 eleutherai and the huggingface inc team this code is based on eleutherai s gptneox library and the gptneox and opt implementations in this library it has been modified from its original forms to accommodate minor architectural differences compared to gptneox and opt used by the meta ai team that trained the model licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license idefics model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername ideficspretrainedconfigarchivemap huggingfacem4idefics9b https huggingface cohuggingfacem4idefics9bblobmainconfig json huggingfacem4idefics80b https huggingface cohuggingfacem4idefics80bblobmainconfig json class ideficsvisionconfigpretrainedconfig r this is the configuration class to store the configuration of a ideficsmodel it is used to instantiate an idefics model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the idefics9b e g huggingfacem4idefics9bhttps huggingface cohuggingfacem4idefics9b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args hiddensize int optional defaults to 768 dimensionality of the encoder layers and the pooler layer elsewhere referred to as hiddensize imagesize int optional defaults to 224 the size resolution of each image intermediatesize int optional defaults to 5120 dimensionality of the intermediate i e feedforward layer in the transformer encoder patchsize int optional defaults to 14 the size resolution of each patch numhiddenlayers int optional defaults to 32 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 16 number of attention heads for each attention layer in the transformer encoder imagenumchannels int optional defaults to 3 number of image channels hiddenact str or function optional defaults to gelu the nonlinear activation function function or string in the encoder and pooler if string gelu relu selu and gelunew quickgelu are supported layernormeps float optional defaults to 1e5 the epsilon used by the layer normalization layers attentiondropout float optional defaults to 0 0 the dropout ratio for the attention probabilities initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices initializerfactor float optional defaults to 1 0 a factor for initializing all weight matrices should be kept to 1 0 used internally for initialization testing initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices modeltype idefics def init self useresamplerfalse resamplernlatents64 resamplerdepth6 resamplernheads16 resamplerheaddim96 qklayernormsperceiverfalse kwargs self useresampler useresampler self resamplernlatents resamplernlatents self resamplerdepth resamplerdepth self resamplernheads resamplernheads self resamplerheaddim resamplerheaddim self qklayernormsperceiver qklayernormsperceiver super initkwargs class ideficsconfigpretrainedconfig r this is the configuration class to store the configuration of a ideficsmodel it is used to instantiate an idefics model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the idefics9b e g huggingfacem4idefics9bhttps huggingface cohuggingfacem4idefics9b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args additionalvocabsize int optional defaults to 0 additional vocabulary size of the model typically for the special img token additional vocab tokens are always trainable whereas regular vocab tokens can be frozen or not vocabsize int optional defaults to 32000 vocabulary size of the idefics model defines the number of different tokens that can be represented by the inputsids passed when calling ideficsmodel hiddensize int optional defaults to 4096 dimension of the hidden representations intermediatesize int optional defaults to 11008 dimension of the mlp representations numhiddenlayers int optional defaults to 32 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 32 number of attention heads for each attention layer in the transformer encoder dropout float optional defaults to 0 0 the dropout probability for all fully connected layers in the embeddings encoder and pooler hiddenact str or function optional defaults to silu the nonlinear activation function function or string in the decoder initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices alphainitializer str optional defaults to zeros initialization type for the alphas alphasinitializerrange float optional defaults to 0 0 the standard deviation of the truncatednormalinitializer for initializing the alphas in the gated cross attention alphatype str optional defaults to float whether the gating alphas should be vectors or single floats rmsnormeps float optional defaults to 1e6 the epsilon used by the rms normalization layers usecache bool optional defaults to true whether or not the model should return the last keyvalues attentions not used by all models only relevant if config isdecodertrue padtokenid int optional defaults to 0 padding token id bostokenid int optional defaults to 1 beginning of stream token id eostokenid int optional defaults to 2 end of stream token id tiewordembeddingsbool optional defaults to false whether to tie weight embeddings crosslayerinterval int optional default to 1 interval for cross attention from text to image layers qklayernorms bool optional defaults to false whether to add layer norm after q and k freezetextlayers bool optional defaults to true whether to freeze text layers freezetextmoduleexceptions bool optional defaults to exceptions to freezing text layers when freezetextlayers is true freezelmhead bool optional defaults to false whether to freeze lm head freezevisionlayers bool optional defaults to true whether to freeze vision layers freezevisionmoduleexceptions bool optional defaults to exceptions to freezing vision layers when freezevisionlayers is true useresampler bool optional defaults to false whether to use the resampler visionconfig ideficsvisionconfig optional custom vision config or dict perceiverconfig ideficsperceiverconfig optional custom perceiver config or dict example python from transformers import ideficsmodel ideficsconfig initializing a idefics idefics9b style configuration configuration ideficsconfig initializing a model from the idefics9b style configuration model ideficsmodelconfiguration accessing the model configuration configuration model config modeltype idefics iscomposition false def init self vocabsize32000 additionalvocabsize0 hiddensize4096 intermediatesize11008 numhiddenlayers32 numattentionheads32 dropout0 0 hiddenactsilu initializerrange0 02 alphainitializerzeros alphasinitializerrange0 0 alphatypefloat rmsnormeps1e6 usecachetrue padtokenid0 bostokenid1 eostokenid2 tiewordembeddingsfalse crosslayerinterval1 qklayernormsfalse freezetextlayerstrue freezetextmoduleexceptions freezelmheadfalse freezevisionlayerstrue freezevisionmoduleexceptions useresamplerfalse visionconfignone perceiverconfignone kwargs self vocabsize vocabsize self additionalvocabsize additionalvocabsize self hiddensize hiddensize self intermediatesize intermediatesize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads self dropout dropout self hiddenact hiddenact self initializerrange initializerrange self alphainitializer alphainitializer self alphasinitializerrange alphasinitializerrange self alphatype alphatype self rmsnormeps rmsnormeps self usecache usecache self crosslayerinterval crosslayerinterval self qklayernorms qklayernorms self freezevisionlayers freezevisionlayers self freezetextlayers freezetextlayers self freezetextmoduleexceptions freezetextmoduleexceptions self freezevisionmoduleexceptions freezevisionmoduleexceptions self freezelmhead freezelmhead self useresampler useresampler if perceiverconfig is none self perceiverconfig ideficsperceiverconfig elif isinstanceperceiverconfig dict self perceiverconfig ideficsperceiverconfigperceiverconfig elif isinstanceperceiverconfig ideficsperceiverconfig self perceiverconfig perceiverconfig if visionconfig is none self visionconfig ideficsvisionconfig elif isinstancevisionconfig dict self visionconfig ideficsvisionconfigvisionconfig elif isinstancevisionconfig ideficsvisionconfig self visionconfig visionconfig super init padtokenidpadtokenid bostokenidbostokenid eostokenideostokenid tiewordembeddingstiewordembeddings kwargs important do not do any init argsbased checks in the constructor since pretrainedconfig fromdict first instantiates the class with the config dict and only then updates the config object with kwargs from frompretrained so during the instantiation of this object many attributes have default values and haven t yet been overridden do any required checks inside frompretrained once the superclass frompretrained was run coding utf 8 2022 eleutherai and the huggingface inc team this code is based on eleutherai s gpt neox library and the gpt neox and opt implementations in this library it has been modified from its original forms to accommodate minor architectural differences compared to gpt neox and opt used by the meta ai team that trained the model licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license idefics model configuration this is the configuration class to store the configuration of a ideficsmodel it is used to instantiate an idefics model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the idefics 9b e g huggingfacem4 idefics 9b https huggingface co huggingfacem4 idefics 9b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args hidden_size int optional defaults to 768 dimensionality of the encoder layers and the pooler layer elsewhere referred to as hidden_size image_size int optional defaults to 224 the size resolution of each image intermediate_size int optional defaults to 5120 dimensionality of the intermediate i e feed forward layer in the transformer encoder patch_size int optional defaults to 14 the size resolution of each patch num_hidden_layers int optional defaults to 32 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 16 number of attention heads for each attention layer in the transformer encoder image_num_channels int optional defaults to 3 number of image channels hidden_act str or function optional defaults to gelu the non linear activation function function or string in the encoder and pooler if string gelu relu selu and gelu_new quick_gelu are supported layer_norm_eps float optional defaults to 1e 5 the epsilon used by the layer normalization layers attention_dropout float optional defaults to 0 0 the dropout ratio for the attention probabilities initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices initializer_factor float optional defaults to 1 0 a factor for initializing all weight matrices should be kept to 1 0 used internally for initialization testing initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices this is the configuration class to store the configuration of a ideficsmodel it is used to instantiate an idefics model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the idefics 9b e g huggingfacem4 idefics 9b https huggingface co huggingfacem4 idefics 9b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args use_resampler bool optional defaults to false whether or not to use the resampler resampler_n_latents int optional defaults to number of latent embeddings to resample compress the input sequence to usually 128 resampler_depth int optional defaults to 6 depth of the perceiver resampler transformer w cross attention should be shallow 3 resampler_n_heads int optional defaults to 16 number of heads in each transformer block for multi headed self attention resampler_head_dim int optional defaults to 96 dimensionality of each head projection in the transformer block qk_layer_norms_perceiver bool optional defaults to false whether or not to use qk layer norms in perceiver this is the configuration class to store the configuration of a ideficsmodel it is used to instantiate an idefics model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the idefics 9b e g huggingfacem4 idefics 9b https huggingface co huggingfacem4 idefics 9b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args additional_vocab_size int optional defaults to 0 additional vocabulary size of the model typically for the special img token additional vocab tokens are always trainable whereas regular vocab tokens can be frozen or not vocab_size int optional defaults to 32000 vocabulary size of the idefics model defines the number of different tokens that can be represented by the inputs_ids passed when calling ideficsmodel hidden_size int optional defaults to 4096 dimension of the hidden representations intermediate_size int optional defaults to 11008 dimension of the mlp representations num_hidden_layers int optional defaults to 32 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 32 number of attention heads for each attention layer in the transformer encoder dropout float optional defaults to 0 0 the dropout probability for all fully connected layers in the embeddings encoder and pooler hidden_act str or function optional defaults to silu the non linear activation function function or string in the decoder initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices alpha_initializer str optional defaults to zeros initialization type for the alphas alphas_initializer_range float optional defaults to 0 0 the standard deviation of the truncated_normal_initializer for initializing the alphas in the gated cross attention alpha_type str optional defaults to float whether the gating alphas should be vectors or single floats rms_norm_eps float optional defaults to 1e 6 the epsilon used by the rms normalization layers use_cache bool optional defaults to true whether or not the model should return the last key values attentions not used by all models only relevant if config is_decoder true pad_token_id int optional defaults to 0 padding token id bos_token_id int optional defaults to 1 beginning of stream token id eos_token_id int optional defaults to 2 end of stream token id tie_word_embeddings bool optional defaults to false whether to tie weight embeddings cross_layer_interval int optional default to 1 interval for cross attention from text to image layers qk_layer_norms bool optional defaults to false whether to add layer norm after q and k freeze_text_layers bool optional defaults to true whether to freeze text layers freeze_text_module_exceptions bool optional defaults to exceptions to freezing text layers when freeze_text_layers is true freeze_lm_head bool optional defaults to false whether to freeze lm head freeze_vision_layers bool optional defaults to true whether to freeze vision layers freeze_vision_module_exceptions bool optional defaults to exceptions to freezing vision layers when freeze_vision_layers is true use_resampler bool optional defaults to false whether to use the resampler vision_config ideficsvisionconfig optional custom vision config or dict perceiver_config ideficsperceiverconfig optional custom perceiver config or dict example python from transformers import ideficsmodel ideficsconfig initializing a idefics idefics 9b style configuration configuration ideficsconfig initializing a model from the idefics 9b style configuration model ideficsmodel configuration accessing the model configuration configuration model config important do not do any __init__ args based checks in the constructor since pretrainedconfig from_dict first instantiates the class with the config dict and only then updates the config object with kwargs from from_pretrained so during the instantiation of this object many attributes have default values and haven t yet been overridden do any required checks inside from_pretrained once the superclass from_pretrained was run
from ...configuration_utils import PretrainedConfig
from ...utils import logging


logger = logging.get_logger(__name__)

IDEFICS_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "HuggingFaceM4/idefics-9b": "https://huggingface.co/HuggingFaceM4/idefics-9b/blob/main/config.json",
    "HuggingFaceM4/idefics-80b": "https://huggingface.co/HuggingFaceM4/idefics-80b/blob/main/config.json",
}


class IdeficsVisionConfig(PretrainedConfig):
    r

    model_type = "idefics"
    attribute_map = {
        "hidden_size": "embed_dim",
    }

    def __init__(
        self,
        embed_dim=768,
        image_size=224,
        intermediate_size=5120,
        patch_size=14,
        num_hidden_layers=32,
        num_attention_heads=16,
        num_channels=3,
        hidden_act="gelu",
        layer_norm_eps=1e-5,
        attention_dropout=0.0,
        initializer_range=0.02,
        initializer_factor=1.0,
        **kwargs,
    ):
        self.embed_dim = embed_dim
        self.image_size = image_size
        self.intermediate_size = intermediate_size
        self.patch_size = patch_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.num_channels = num_channels
        self.layer_norm_eps = layer_norm_eps
        self.attention_dropout = attention_dropout
        self.initializer_range = initializer_range
        self.initializer_factor = initializer_factor
        self.hidden_act = hidden_act

        super().__init__(**kwargs)


class IdeficsPerceiverConfig(PretrainedConfig):
    r

    model_type = "idefics"

    def __init__(
        self,
        use_resampler=False,
        resampler_n_latents=64,
        resampler_depth=6,
        resampler_n_heads=16,
        resampler_head_dim=96,
        qk_layer_norms_perceiver=False,
        **kwargs,
    ):
        self.use_resampler = use_resampler
        self.resampler_n_latents = resampler_n_latents
        self.resampler_depth = resampler_depth
        self.resampler_n_heads = resampler_n_heads
        self.resampler_head_dim = resampler_head_dim
        self.qk_layer_norms_perceiver = qk_layer_norms_perceiver

        super().__init__(**kwargs)


class IdeficsConfig(PretrainedConfig):
    r

    model_type = "idefics"
    is_composition = False

    def __init__(
        self,
        vocab_size=32000,
        additional_vocab_size=0,
        hidden_size=4096,
        intermediate_size=11008,
        num_hidden_layers=32,
        num_attention_heads=32,
        dropout=0.0,
        hidden_act="silu",
        initializer_range=0.02,
        alpha_initializer="zeros",
        alphas_initializer_range=0.0,
        alpha_type="float",
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=0,
        bos_token_id=1,
        eos_token_id=2,
        tie_word_embeddings=False,
        cross_layer_interval=1,
        qk_layer_norms=False,
        freeze_text_layers=True,
        freeze_text_module_exceptions=[],
        freeze_lm_head=False,
        freeze_vision_layers=True,
        freeze_vision_module_exceptions=[],
        use_resampler=False,
        vision_config=None,
        perceiver_config=None,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.additional_vocab_size = additional_vocab_size
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.dropout = dropout
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.alpha_initializer = alpha_initializer
        self.alphas_initializer_range = alphas_initializer_range
        self.alpha_type = alpha_type
        self.rms_norm_eps = rms_norm_eps
        self.use_cache = use_cache

        self.cross_layer_interval = cross_layer_interval
        self.qk_layer_norms = qk_layer_norms
        self.freeze_vision_layers = freeze_vision_layers

        self.freeze_text_layers = freeze_text_layers
        self.freeze_text_module_exceptions = freeze_text_module_exceptions
        self.freeze_vision_module_exceptions = freeze_vision_module_exceptions
        self.freeze_lm_head = freeze_lm_head

        self.use_resampler = use_resampler

        if perceiver_config is None:
            self.perceiver_config = IdeficsPerceiverConfig()
        elif isinstance(perceiver_config, dict):
            self.perceiver_config = IdeficsPerceiverConfig(**perceiver_config)
        elif isinstance(perceiver_config, IdeficsPerceiverConfig):
            self.perceiver_config = perceiver_config

        if vision_config is None:
            self.vision_config = IdeficsVisionConfig()
        elif isinstance(vision_config, dict):
            self.vision_config = IdeficsVisionConfig(**vision_config)
        elif isinstance(vision_config, IdeficsVisionConfig):
            self.vision_config = vision_config

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )