codingutf8 2022 eleutherai and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license gptneox model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername gptneoxpretrainedconfigarchivemap eleutheraigptneox20b https huggingface coeleutheraigptneox20bresolvemainconfig json see all gptneox models at https huggingface comodels filtergptneox class gptneoxconfigpretrainedconfig r this is the configuration class to store the configuration of a gptneoxmodel it is used to instantiate an gptneox model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the gptneox eleutheraigptneox20bhttps huggingface coeleutheraigptneox20b architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocabsize int optional defaults to 50432 vocabulary size of the gptneox model defines the number of different tokens that can be represented by the inputsids passed when calling gptneoxmodel hiddensize int optional defaults to 6144 dimension of the encoder layers and the pooler layer numhiddenlayers int optional defaults to 44 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 64 number of attention heads for each attention layer in the transformer encoder intermediatesize int optional defaults to 24576 dimension of the intermediate i e feedforward layer in the transformer encoder hiddenact str or function optional defaults to gelu the nonlinear activation function function or string in the encoder and pooler if string gelu relu selu and gelunew are supported rotarypct float optional defaults to 0 25 percentage of hidden dimensions to allocate to rotary embeddings rotaryembbase int optional defaults to 10000 base for computing rotary embeddings frequency attentiondropout float optional defaults to 0 0 the dropout ratio probability of the attention score hiddendropout float optional defaults to 0 0 the dropout ratio of 1 the word embeddings 2 the postattention hidden states and 3 the postmlp hidden states classifierdropout float optional defaults to 0 1 argument used when doing token classification used in the model gptneoxfortokenclassification the dropout ratio for the hidden layer maxpositionembeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with typically set this to something large just in case e g 512 or 1024 or 2048 initializerrange float optional defaults to 1e5 the standard deviation of the truncatednormalinitializer for initializing all weight matrices layernormeps float optional defaults to 1e12 the epsilon used by the layer normalization layers usecache bool optional defaults to true whether or not the model should return the last keyvalues attentions not used by all models only relevant if config isdecodertrue useparallelresidual bool optional defaults to true whether to use a parallel formulation in each transformer layer which can provide a slight training speedup at large scales e g 20b ropescaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be a float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update maxpositionembeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit comrlocalllamacomments14mrgprdynamicallyscaledropefurtherincreases this is an experimental feature subject to breaking api changes in future versions example python from transformers import gptneoxconfig gptneoxmodel initializing a gptneox gptneox20b style configuration configuration gptneoxconfig initializing a model with random weights from the gptneox20b style configuration model gptneoxmodelconfiguration doctest skip accessing the model configuration configuration model config doctest skip modeltype gptneox def init self vocabsize50432 hiddensize6144 numhiddenlayers44 numattentionheads64 intermediatesize24576 hiddenactgelu rotarypct0 25 rotaryembbase10000 attentiondropout0 0 hiddendropout0 0 classifierdropout0 1 maxpositionembeddings2048 initializerrange0 02 layernormeps1e5 usecachetrue bostokenid0 eostokenid2 tiewordembeddingsfalse useparallelresidualtrue ropescalingnone kwargs super initbostokenidbostokenid eostokenideostokenid kwargs self vocabsize vocabsize self maxpositionembeddings maxpositionembeddings self hiddensize hiddensize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads self intermediatesize intermediatesize self hiddenact hiddenact self rotarypct rotarypct self rotaryembbase rotaryembbase self attentiondropout attentiondropout self hiddendropout hiddendropout self classifierdropout classifierdropout self initializerrange initializerrange self layernormeps layernormeps self usecache usecache self tiewordembeddings tiewordembeddings self useparallelresidual useparallelresidual self ropescaling ropescaling self ropescalingvalidation if self hiddensize self numattentionheads 0 raise valueerror the hidden size is not divisble by the number of attention heads make sure to update them copied from transformers models llama configurationllama llamaconfig ropescalingvalidation def ropescalingvalidationself if self ropescaling is none return if not isinstanceself ropescaling dict or lenself ropescaling 2 raise valueerror ropescaling must be a dictionary with with two fields type and factor fgot self ropescaling ropescalingtype self ropescaling gettype none ropescalingfactor self ropescaling getfactor none if ropescalingtype is none or ropescalingtype not in linear dynamic raise valueerror fropescaling s type field must be one of linear dynamic got ropescalingtype if ropescalingfactor is none or not isinstanceropescalingfactor float or ropescalingfactor 1 0 raise valueerrorfropescaling s factor field must be a float 1 got ropescalingfactor coding utf 8 2022 eleutherai and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license gptneox model configuration see all gptneox models at https huggingface co models filter gpt_neox this is the configuration class to store the configuration of a gptneoxmodel it is used to instantiate an gptneox model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the gptneox eleutherai gpt neox 20b https huggingface co eleutherai gpt neox 20b architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocab_size int optional defaults to 50432 vocabulary size of the gptneox model defines the number of different tokens that can be represented by the inputs_ids passed when calling gptneoxmodel hidden_size int optional defaults to 6144 dimension of the encoder layers and the pooler layer num_hidden_layers int optional defaults to 44 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 64 number of attention heads for each attention layer in the transformer encoder intermediate_size int optional defaults to 24576 dimension of the intermediate i e feed forward layer in the transformer encoder hidden_act str or function optional defaults to gelu the non linear activation function function or string in the encoder and pooler if string gelu relu selu and gelu_new are supported rotary_pct float optional defaults to 0 25 percentage of hidden dimensions to allocate to rotary embeddings rotary_emb_base int optional defaults to 10000 base for computing rotary embeddings frequency attention_dropout float optional defaults to 0 0 the dropout ratio probability of the attention score hidden_dropout float optional defaults to 0 0 the dropout ratio of 1 the word embeddings 2 the post attention hidden states and 3 the post mlp hidden states classifier_dropout float optional defaults to 0 1 argument used when doing token classification used in the model gptneoxfortokenclassification the dropout ratio for the hidden layer max_position_embeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with typically set this to something large just in case e g 512 or 1024 or 2048 initializer_range float optional defaults to 1e 5 the standard deviation of the truncated_normal_initializer for initializing all weight matrices layer_norm_eps float optional defaults to 1e 12 the epsilon used by the layer normalization layers use_cache bool optional defaults to true whether or not the model should return the last key values attentions not used by all models only relevant if config is_decoder true use_parallel_residual bool optional defaults to true whether to use a parallel formulation in each transformer layer which can provide a slight training speedup at large scales e g 20b rope_scaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be a float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update max_position_embeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit com r localllama comments 14mrgpr dynamically_scaled_rope_further_increases this is an experimental feature subject to breaking api changes in future versions example python from transformers import gptneoxconfig gptneoxmodel initializing a gptneox gpt neox 20b style configuration configuration gptneoxconfig initializing a model with random weights from the gpt neox 20b style configuration model gptneoxmodel configuration doctest skip accessing the model configuration configuration model config doctest skip copied from transformers models llama configuration_llama llamaconfig _rope_scaling_validation validate the rope_scaling configuration
from ...configuration_utils import PretrainedConfig
from ...utils import logging


logger = logging.get_logger(__name__)

GPT_NEOX_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "EleutherAI/gpt-neox-20b": "https://huggingface.co/EleutherAI/gpt-neox-20b/resolve/main/config.json",
    
}


class GPTNeoXConfig(PretrainedConfig):
    r

    model_type = "gpt_neox"

    def __init__(
        self,
        vocab_size=50432,
        hidden_size=6144,
        num_hidden_layers=44,
        num_attention_heads=64,
        intermediate_size=24576,
        hidden_act="gelu",
        rotary_pct=0.25,
        rotary_emb_base=10000,
        attention_dropout=0.0,
        hidden_dropout=0.0,
        classifier_dropout=0.1,
        max_position_embeddings=2048,
        initializer_range=0.02,
        layer_norm_eps=1e-5,
        use_cache=True,
        bos_token_id=0,
        eos_token_id=2,
        tie_word_embeddings=False,
        use_parallel_residual=True,
        rope_scaling=None,
        **kwargs,
    ):
        super().__init__(bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.rotary_pct = rotary_pct
        self.rotary_emb_base = rotary_emb_base
        self.attention_dropout = attention_dropout
        self.hidden_dropout = hidden_dropout
        self.classifier_dropout = classifier_dropout
        self.initializer_range = initializer_range
        self.layer_norm_eps = layer_norm_eps
        self.use_cache = use_cache
        self.tie_word_embeddings = tie_word_embeddings
        self.use_parallel_residual = use_parallel_residual
        self.rope_scaling = rope_scaling
        self._rope_scaling_validation()

        if self.hidden_size % self.num_attention_heads != 0:
            raise ValueError(
                "The hidden size is not divisble by the number of attention heads! Make sure to update them!"
            )

    
    def _rope_scaling_validation(self):
        
        if self.rope_scaling is None:
            return

        if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
            raise ValueError(
                "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
                f"got {self.rope_scaling}"
            )
        rope_scaling_type = self.rope_scaling.get("type", None)
        rope_scaling_factor = self.rope_scaling.get("factor", None)
        if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
            raise ValueError(
                f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
            )
        if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
            raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")