codingutf8 2022 eleutherai and the huggingface inc team this code is based on eleutherai s gptneox library and the gptneox and opt implementations in this library it has been modified from its original forms to accommodate minor architectural differences compared to gptneox and opt used by the meta ai team that trained the model licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license llama model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername llamapretrainedconfigarchivemap class llamaconfigpretrainedconfig r this is the configuration class to store the configuration of a llamamodel it is used to instantiate an llama model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the llama7b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocabsize int optional defaults to 32000 vocabulary size of the llama model defines the number of different tokens that can be represented by the inputsids passed when calling llamamodel hiddensize int optional defaults to 4096 dimension of the hidden representations intermediatesize int optional defaults to 11008 dimension of the mlp representations numhiddenlayers int optional defaults to 32 number of hidden layers in the transformer decoder numattentionheads int optional defaults to 32 number of attention heads for each attention layer in the transformer decoder numkeyvalueheads int optional this is the number of keyvalue heads that should be used to implement grouped query attention if numkeyvalueheadsnumattentionheads the model will use multi head attention mha if numkeyvalueheads1 the model will use multi query attention mqa otherwise gqa is used when converting a multihead checkpoint to a gqa checkpoint each group key and value head should be constructed by meanpooling all the original heads within that group for more details checkout this paperhttps arxiv orgpdf2305 13245 pdf if it is not specified will default to numattentionheads hiddenact str or function optional defaults to silu the nonlinear activation function function or string in the decoder maxpositionembeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with llama 1 supports up to 2048 tokens llama 2 up to 4096 codellama up to 16384 initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices rmsnormeps float optional defaults to 1e06 the epsilon used by the rms normalization layers usecache bool optional defaults to true whether or not the model should return the last keyvalues attentions not used by all models only relevant if config isdecodertrue padtokenid int optional padding token id bostokenid int optional defaults to 1 beginning of stream token id eostokenid int optional defaults to 2 end of stream token id pretrainingtp int optional defaults to 1 experimental feature tensor parallelism rank used during pretraining please refer to this documenthttps huggingface codocstransformersparallelism to understand more about it this value is necessary to ensure exact reproducibility of the pretraining results please refer to this issuehttps github compytorchpytorchissues76232 tiewordembeddings bool optional defaults to false whether to tie weight embeddings ropetheta float optional defaults to 10000 0 the base period of the rope embeddings ropescaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be a float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update maxpositionembeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit comrlocalllamacomments14mrgprdynamicallyscaledropefurtherincreases this is an experimental feature subject to breaking api changes in future versions attentionbias bool defaults to false optional defaults to false whether to use a bias in the query key value and output projection layers during selfattention attentiondropout float optional defaults to 0 0 the dropout ratio for the attention probabilities python from transformers import llamamodel llamaconfig initializing a llama llama7b style configuration configuration llamaconfig initializing a model from the llama7b style configuration model llamamodelconfiguration accessing the model configuration configuration model config modeltype llama keystoignoreatinference pastkeyvalues def init self vocabsize32000 hiddensize4096 intermediatesize11008 numhiddenlayers32 numattentionheads32 numkeyvalueheadsnone hiddenactsilu maxpositionembeddings2048 initializerrange0 02 rmsnormeps1e6 usecachetrue padtokenidnone bostokenid1 eostokenid2 pretrainingtp1 tiewordembeddingsfalse ropetheta10000 0 ropescalingnone attentionbiasfalse attentiondropout0 0 kwargs self vocabsize vocabsize self maxpositionembeddings maxpositionembeddings self hiddensize hiddensize self intermediatesize intermediatesize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads for backward compatibility if numkeyvalueheads is none numkeyvalueheads numattentionheads self numkeyvalueheads numkeyvalueheads self hiddenact hiddenact self initializerrange initializerrange self rmsnormeps rmsnormeps self pretrainingtp pretrainingtp self usecache usecache self ropetheta ropetheta self ropescaling ropescaling self ropescalingvalidation self attentionbias attentionbias self attentiondropout attentiondropout super init padtokenidpadtokenid bostokenidbostokenid eostokenideostokenid tiewordembeddingstiewordembeddings kwargs def ropescalingvalidationself if self ropescaling is none return if not isinstanceself ropescaling dict or lenself ropescaling 2 raise valueerror ropescaling must be a dictionary with with two fields type and factor fgot self ropescaling ropescalingtype self ropescaling gettype none ropescalingfactor self ropescaling getfactor none if ropescalingtype is none or ropescalingtype not in linear dynamic raise valueerror fropescaling s type field must be one of linear dynamic got ropescalingtype if ropescalingfactor is none or not isinstanceropescalingfactor float or ropescalingfactor 1 0 raise valueerrorfropescaling s factor field must be a float 1 got ropescalingfactor coding utf 8 2022 eleutherai and the huggingface inc team this code is based on eleutherai s gpt neox library and the gpt neox and opt implementations in this library it has been modified from its original forms to accommodate minor architectural differences compared to gpt neox and opt used by the meta ai team that trained the model licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license llama model configuration this is the configuration class to store the configuration of a llamamodel it is used to instantiate an llama model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the llama 7b configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocab_size int optional defaults to 32000 vocabulary size of the llama model defines the number of different tokens that can be represented by the inputs_ids passed when calling llamamodel hidden_size int optional defaults to 4096 dimension of the hidden representations intermediate_size int optional defaults to 11008 dimension of the mlp representations num_hidden_layers int optional defaults to 32 number of hidden layers in the transformer decoder num_attention_heads int optional defaults to 32 number of attention heads for each attention layer in the transformer decoder num_key_value_heads int optional this is the number of key_value heads that should be used to implement grouped query attention if num_key_value_heads num_attention_heads the model will use multi head attention mha if num_key_value_heads 1 the model will use multi query attention mqa otherwise gqa is used when converting a multi head checkpoint to a gqa checkpoint each group key and value head should be constructed by meanpooling all the original heads within that group for more details checkout this paper https arxiv org pdf 2305 13245 pdf if it is not specified will default to num_attention_heads hidden_act str or function optional defaults to silu the non linear activation function function or string in the decoder max_position_embeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with llama 1 supports up to 2048 tokens llama 2 up to 4096 codellama up to 16384 initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices rms_norm_eps float optional defaults to 1e 06 the epsilon used by the rms normalization layers use_cache bool optional defaults to true whether or not the model should return the last key values attentions not used by all models only relevant if config is_decoder true pad_token_id int optional padding token id bos_token_id int optional defaults to 1 beginning of stream token id eos_token_id int optional defaults to 2 end of stream token id pretraining_tp int optional defaults to 1 experimental feature tensor parallelism rank used during pretraining please refer to this document https huggingface co docs transformers parallelism to understand more about it this value is necessary to ensure exact reproducibility of the pretraining results please refer to this issue https github com pytorch pytorch issues 76232 tie_word_embeddings bool optional defaults to false whether to tie weight embeddings rope_theta float optional defaults to 10000 0 the base period of the rope embeddings rope_scaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be a float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update max_position_embeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit com r localllama comments 14mrgpr dynamically_scaled_rope_further_increases this is an experimental feature subject to breaking api changes in future versions attention_bias bool defaults to false optional defaults to false whether to use a bias in the query key value and output projection layers during self attention attention_dropout float optional defaults to 0 0 the dropout ratio for the attention probabilities python from transformers import llamamodel llamaconfig initializing a llama llama 7b style configuration configuration llamaconfig initializing a model from the llama 7b style configuration model llamamodel configuration accessing the model configuration configuration model config for backward compatibility validate the rope_scaling configuration
from ...configuration_utils import PretrainedConfig
from ...utils import logging


logger = logging.get_logger(__name__)

LLAMA_PRETRAINED_CONFIG_ARCHIVE_MAP = {}


class LlamaConfig(PretrainedConfig):
    r

    model_type = "llama"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(
        self,
        vocab_size=32000,
        hidden_size=4096,
        intermediate_size=11008,
        num_hidden_layers=32,
        num_attention_heads=32,
        num_key_value_heads=None,
        hidden_act="silu",
        max_position_embeddings=2048,
        initializer_range=0.02,
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=None,
        bos_token_id=1,
        eos_token_id=2,
        pretraining_tp=1,
        tie_word_embeddings=False,
        rope_theta=10000.0,
        rope_scaling=None,
        attention_bias=False,
        attention_dropout=0.0,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads

        
        if num_key_value_heads is None:
            num_key_value_heads = num_attention_heads

        self.num_key_value_heads = num_key_value_heads
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.pretraining_tp = pretraining_tp
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.rope_scaling = rope_scaling
        self._rope_scaling_validation()
        self.attention_bias = attention_bias
        self.attention_dropout = attention_dropout

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )

    def _rope_scaling_validation(self):
        
        if self.rope_scaling is None:
            return

        if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
            raise ValueError(
                "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
                f"got {self.rope_scaling}"
            )
        rope_scaling_type = self.rope_scaling.get("type", None)
        rope_scaling_factor = self.rope_scaling.get("factor", None)
        if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
            raise ValueError(
                f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
            )
        if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
            raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")