codingutf8 2023 eleutherai and the huggingface inc team this code is based on eleutherai s gptneox library and the gptneox and opt implementations in this library it has been modified from its original forms to accommodate minor architectural differences compared to gptneox and opt used by the meta ai team that trained the model licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license openllama model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername openllamapretrainedconfigarchivemap sjolopenllamav1 https huggingface cosjolopenllamav1blobmainconfig json class openllamaconfigpretrainedconfig r this is the configuration class to store the configuration of a openllamamodel it is used to instantiate an openllama model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the sjolopenllamav1https huggingface cosjolopenllamav1 configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocabsize int optional defaults to 32000 vocabulary size of the openllama model defines the number of different tokens that can be represented by the inputsids passed when calling openllamamodel hiddensize int optional defaults to 4096 dimension of the hidden representations intermediatesize int optional defaults to 11008 dimension of the mlp representations numhiddenlayers int optional defaults to 32 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 32 number of attention heads for each attention layer in the transformer encoder hiddenact str or function optional defaults to silu the nonlinear activation function function or string in the decoder maxpositionembeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with typically set this to something large just in case e g 512 or 1024 or 2048 initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices rmsnormeps float optional defaults to 1e12 the epsilon used by the rms normalization layers usecache bool optional defaults to true whether or not the model should return the last keyvalues attentions not used by all models only relevant if config isdecodertrue tiewordembeddingsbool optional defaults to false whether to tie weight embeddings ropescaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be a float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update maxpositionembeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit comrlocalllamacomments14mrgprdynamicallyscaledropefurtherincreases this is an experimental feature subject to breaking api changes in future versions example python from transformers import openllamamodel openllamaconfig initializing a openllama openllama7b style configuration configuration openllamaconfig initializing a model from the openllama7b style configuration model openllamamodelconfiguration accessing the model configuration configuration model config modeltype openllama def init self vocabsize100000 hiddensize4096 intermediatesize11008 numhiddenlayers32 numattentionheads32 hiddenactsilu maxpositionembeddings2048 initializerrange0 02 rmsnormeps1e6 usecachetrue padtokenid0 bostokenid1 eostokenid2 tiewordembeddingsfalse usememoryefficientattentiontrue hiddendropoutprob0 1 attentiondropoutprob0 1 usestableembeddingtrue sharedinputoutputembeddingtrue ropescalingnone kwargs self vocabsize vocabsize self maxpositionembeddings maxpositionembeddings self hiddensize hiddensize self intermediatesize intermediatesize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads self hiddenact hiddenact self initializerrange initializerrange self rmsnormeps rmsnormeps self usecache usecache self usememoryefficientattention kwargs pop usememorryefficientattention usememoryefficientattention self hiddendropoutprob hiddendropoutprob self attentiondropoutprob attentiondropoutprob self usestableembedding usestableembedding self sharedinputoutputembedding sharedinputoutputembedding self ropescaling ropescaling self ropescalingvalidation super init padtokenidpadtokenid bostokenidbostokenid eostokenideostokenid tiewordembeddingstiewordembeddings kwargs copied from transformers models llama configurationllama llamaconfig ropescalingvalidation def ropescalingvalidationself if self ropescaling is none return if not isinstanceself ropescaling dict or lenself ropescaling 2 raise valueerror ropescaling must be a dictionary with with two fields type and factor fgot self ropescaling ropescalingtype self ropescaling gettype none ropescalingfactor self ropescaling getfactor none if ropescalingtype is none or ropescalingtype not in linear dynamic raise valueerror fropescaling s type field must be one of linear dynamic got ropescalingtype if ropescalingfactor is none or not isinstanceropescalingfactor float or ropescalingfactor 1 0 raise valueerrorfropescaling s factor field must be a float 1 got ropescalingfactor coding utf 8 2023 eleutherai and the huggingface inc team this code is based on eleutherai s gpt neox library and the gpt neox and opt implementations in this library it has been modified from its original forms to accommodate minor architectural differences compared to gpt neox and opt used by the meta ai team that trained the model licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license open llama model configuration this is the configuration class to store the configuration of a openllamamodel it is used to instantiate an open llama model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the s jol open llama v1 https huggingface co s jol open llama v1 configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocab_size int optional defaults to 32000 vocabulary size of the open llama model defines the number of different tokens that can be represented by the inputs_ids passed when calling openllamamodel hidden_size int optional defaults to 4096 dimension of the hidden representations intermediate_size int optional defaults to 11008 dimension of the mlp representations num_hidden_layers int optional defaults to 32 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 32 number of attention heads for each attention layer in the transformer encoder hidden_act str or function optional defaults to silu the non linear activation function function or string in the decoder max_position_embeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with typically set this to something large just in case e g 512 or 1024 or 2048 initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices rms_norm_eps float optional defaults to 1e 12 the epsilon used by the rms normalization layers use_cache bool optional defaults to true whether or not the model should return the last key values attentions not used by all models only relevant if config is_decoder true tie_word_embeddings bool optional defaults to false whether to tie weight embeddings rope_scaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be a float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update max_position_embeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit com r localllama comments 14mrgpr dynamically_scaled_rope_further_increases this is an experimental feature subject to breaking api changes in future versions example python from transformers import openllamamodel openllamaconfig initializing a open llama open_llama 7b style configuration configuration openllamaconfig initializing a model from the open_llama 7b style configuration model openllamamodel configuration accessing the model configuration configuration model config copied from transformers models llama configuration_llama llamaconfig _rope_scaling_validation validate the rope_scaling configuration
from ....configuration_utils import PretrainedConfig
from ....utils import logging


logger = logging.get_logger(__name__)

OPEN_LLAMA_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "s-JoL/Open-Llama-V1": "https://huggingface.co/s-JoL/Open-Llama-V1/blob/main/config.json",
}


class OpenLlamaConfig(PretrainedConfig):
    r

    model_type = "open-llama"

    def __init__(
        self,
        vocab_size=100000,
        hidden_size=4096,
        intermediate_size=11008,
        num_hidden_layers=32,
        num_attention_heads=32,
        hidden_act="silu",
        max_position_embeddings=2048,
        initializer_range=0.02,
        rms_norm_eps=1e-6,
        use_cache=True,
        pad_token_id=0,
        bos_token_id=1,
        eos_token_id=2,
        tie_word_embeddings=False,
        use_memory_efficient_attention=True,
        hidden_dropout_prob=0.1,
        attention_dropout_prob=0.1,
        use_stable_embedding=True,
        shared_input_output_embedding=True,
        rope_scaling=None,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.rms_norm_eps = rms_norm_eps
        self.use_cache = use_cache
        self.use_memory_efficient_attention = kwargs.pop(
            "use_memorry_efficient_attention", use_memory_efficient_attention
        )
        self.hidden_dropout_prob = hidden_dropout_prob
        self.attention_dropout_prob = attention_dropout_prob
        self.use_stable_embedding = use_stable_embedding
        self.shared_input_output_embedding = shared_input_output_embedding
        self.rope_scaling = rope_scaling
        self._rope_scaling_validation()

        super().__init__(
            pad_token_id=pad_token_id,
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )

    
    def _rope_scaling_validation(self):
        
        if self.rope_scaling is None:
            return

        if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
            raise ValueError(
                "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
                f"got {self.rope_scaling}"
            )
        rope_scaling_type = self.rope_scaling.get("type", None)
        rope_scaling_factor = self.rope_scaling.get("factor", None)
        if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
            raise ValueError(
                f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
            )
        if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
            raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")