codingutf8 2023 microsoft and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license phi model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername phipretrainedconfigarchivemap susnatophi1dev https huggingface cosusnatophi1devresolvemainconfig json susnatophi15dev https huggingface cosusnatophi15devresolvemainconfig json class phiconfigpretrainedconfig r this is the configuration class to store the configuration of a phimodel it is used to instantiate an phi model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the phi susnatophi1devhttps huggingface cosusnatophi1dev configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocabsize int optional defaults to 51200 vocabulary size of the phi model defines the number of different tokens that can be represented by the inputsids passed when calling phimodel hiddensize int optional defaults to 2048 dimension of the hidden representations intermediatesize int optional defaults to 8192 dimension of the mlp representations numhiddenlayers int optional defaults to 24 number of hidden layers in the transformer decoder numattentionheads int optional defaults to 32 number of attention heads for each attention layer in the transformer decoder residpdrop float optional defaults to 0 0 dropout probability for mlp outputs embdpdrop int optional defaults to 0 0 the dropout ratio for the embeddings attentiondropout float optional defaults to 0 0 the dropout ratio after computing the attention scores hiddenact str or function optional defaults to gelunew the nonlinear activation function function or string in the decoder maxpositionembeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with phi1 and phi1 5 supports up to 2048 tokens initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices layernormeps float optional defaults to 1e05 the epsilon used by the rms normalization layers usecache bool optional defaults to true whether or not the model should return the last keyvalues attentions not used by all models only relevant if config isdecodertrue whether to tie weight embeddings or not tiewordembeddings bool optional defaults to false whether to tie weight embeddings ropetheta float optional defaults to 10000 0 the base period of the rope embeddings ropescaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be an float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update maxpositionembeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit comrlocalpersimmoncomments14mrgprdynamicallyscaledropefurtherincreases this is an experimental feature subject to breaking api changes in future versions partialrotaryfactor float optional defaults to 0 5 percentage of the query and keys which will have rotary embedding qklayernorm bool optional defaults to false whether or not to normalize the queries and keys after projecting the hidden states bostokenid int optional defaults to 1 denotes beginning of sequences token id eostokenid int optional defaults to 2 denotes end of sequences token id example python from transformers import phimodel phiconfig initializing a phi1 style configuration configuration phiconfig frompretrainedsusnatophi1dev initializing a model from the configuration model phimodelconfiguration accessing the model configuration configuration model config modeltype phi keystoignoreatinference pastkeyvalues def init self vocabsize51200 hiddensize2048 intermediatesize8192 numhiddenlayers24 numattentionheads32 residpdrop0 0 embdpdrop0 0 attentiondropout0 0 hiddenactgelunew maxpositionembeddings2048 initializerrange0 02 layernormeps1e5 usecachetrue tiewordembeddingsfalse ropetheta10000 0 ropescalingnone partialrotaryfactor0 5 qklayernormfalse bostokenid1 eostokenid2 kwargs self vocabsize vocabsize self hiddensize hiddensize self intermediatesize intermediatesize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads self residpdrop residpdrop self embdpdrop embdpdrop self attentiondropout attentiondropout self hiddenact hiddenact self maxpositionembeddings maxpositionembeddings self initializerrange initializerrange self layernormeps layernormeps self usecache usecache self ropetheta ropetheta self ropescaling ropescaling self partialrotaryfactor partialrotaryfactor self qklayernorm qklayernorm self ropescalingvalidation super init bostokenidbostokenid eostokenideostokenid tiewordembeddingstiewordembeddings kwargs copied from transformers models llama configurationllama llamaconfig ropescalingvalidation def ropescalingvalidationself if self ropescaling is none return if not isinstanceself ropescaling dict or lenself ropescaling 2 raise valueerror ropescaling must be a dictionary with with two fields type and factor fgot self ropescaling ropescalingtype self ropescaling gettype none ropescalingfactor self ropescaling getfactor none if ropescalingtype is none or ropescalingtype not in linear dynamic raise valueerror fropescaling s type field must be one of linear dynamic got ropescalingtype if ropescalingfactor is none or not isinstanceropescalingfactor float or ropescalingfactor 1 0 raise valueerrorfropescaling s factor field must be a float 1 got ropescalingfactor coding utf 8 2023 microsoft and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license phi model configuration this is the configuration class to store the configuration of a phimodel it is used to instantiate an phi model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the phi susnato phi 1_dev https huggingface co susnato phi 1_dev configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocab_size int optional defaults to 51200 vocabulary size of the phi model defines the number of different tokens that can be represented by the inputs_ids passed when calling phimodel hidden_size int optional defaults to 2048 dimension of the hidden representations intermediate_size int optional defaults to 8192 dimension of the mlp representations num_hidden_layers int optional defaults to 24 number of hidden layers in the transformer decoder num_attention_heads int optional defaults to 32 number of attention heads for each attention layer in the transformer decoder resid_pdrop float optional defaults to 0 0 dropout probability for mlp outputs embd_pdrop int optional defaults to 0 0 the dropout ratio for the embeddings attention_dropout float optional defaults to 0 0 the dropout ratio after computing the attention scores hidden_act str or function optional defaults to gelu_new the non linear activation function function or string in the decoder max_position_embeddings int optional defaults to 2048 the maximum sequence length that this model might ever be used with phi 1 and phi 1 5 supports up to 2048 tokens initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices layer_norm_eps float optional defaults to 1e 05 the epsilon used by the rms normalization layers use_cache bool optional defaults to true whether or not the model should return the last key values attentions not used by all models only relevant if config is_decoder true whether to tie weight embeddings or not tie_word_embeddings bool optional defaults to false whether to tie weight embeddings rope_theta float optional defaults to 10000 0 the base period of the rope embeddings rope_scaling dict optional dictionary containing the scaling configuration for the rope embeddings currently supports two scaling strategies linear and dynamic their scaling factor must be an float greater than 1 the expected format is type strategy name factor scaling factor when using this flag don t update max_position_embeddings to the expected new maximum see the following thread for more information on how these scaling strategies behave https www reddit com r localpersimmon comments 14mrgpr dynamically_scaled_rope_further_increases this is an experimental feature subject to breaking api changes in future versions partial_rotary_factor float optional defaults to 0 5 percentage of the query and keys which will have rotary embedding qk_layernorm bool optional defaults to false whether or not to normalize the queries and keys after projecting the hidden states bos_token_id int optional defaults to 1 denotes beginning of sequences token id eos_token_id int optional defaults to 2 denotes end of sequences token id example python from transformers import phimodel phiconfig initializing a phi 1 style configuration configuration phiconfig from_pretrained susnato phi 1_dev initializing a model from the configuration model phimodel configuration accessing the model configuration configuration model config copied from transformers models llama configuration_llama llamaconfig _rope_scaling_validation validate the rope_scaling configuration
from ...configuration_utils import PretrainedConfig
from ...utils import logging


logger = logging.get_logger(__name__)

PHI_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "susnato/phi-1_dev": "https://huggingface.co/susnato/phi-1_dev/resolve/main/config.json",
    "susnato/phi-1_5_dev": "https://huggingface.co/susnato/phi-1_5_dev/resolve/main/config.json",
}


class PhiConfig(PretrainedConfig):
    r

    model_type = "phi"
    keys_to_ignore_at_inference = ["past_key_values"]

    def __init__(
        self,
        vocab_size=51200,
        hidden_size=2048,
        intermediate_size=8192,
        num_hidden_layers=24,
        num_attention_heads=32,
        resid_pdrop=0.0,
        embd_pdrop=0.0,
        attention_dropout=0.0,
        hidden_act="gelu_new",
        max_position_embeddings=2048,
        initializer_range=0.02,
        layer_norm_eps=1e-5,
        use_cache=True,
        tie_word_embeddings=False,
        rope_theta=10000.0,
        rope_scaling=None,
        partial_rotary_factor=0.5,
        qk_layernorm=False,
        bos_token_id=1,
        eos_token_id=2,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.resid_pdrop = resid_pdrop
        self.embd_pdrop = embd_pdrop
        self.attention_dropout = attention_dropout
        self.hidden_act = hidden_act
        self.max_position_embeddings = max_position_embeddings
        self.initializer_range = initializer_range
        self.layer_norm_eps = layer_norm_eps
        self.use_cache = use_cache
        self.rope_theta = rope_theta
        self.rope_scaling = rope_scaling
        self.partial_rotary_factor = partial_rotary_factor
        self.qk_layernorm = qk_layernorm
        self._rope_scaling_validation()

        super().__init__(
            bos_token_id=bos_token_id,
            eos_token_id=eos_token_id,
            tie_word_embeddings=tie_word_embeddings,
            **kwargs,
        )

    
    def _rope_scaling_validation(self):
        
        if self.rope_scaling is None:
            return

        if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
            raise ValueError(
                "`rope_scaling` must be a dictionary with with two fields, `type` and `factor`, "
                f"got {self.rope_scaling}"
            )
        rope_scaling_type = self.rope_scaling.get("type", None)
        rope_scaling_factor = self.rope_scaling.get("factor", None)
        if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
            raise ValueError(
                f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
            )
        if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
            raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")