codingutf8 2023 xuan ouyang shuohuan wang chao pang yu sun hao tian hua wu haifeng wang and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license erniem model configuration adapted from original paddlenlp repository https github compaddlepaddlepaddlenlpblobdeveloppaddlenlptransformerserniemconfiguration py from future import annotations from typing import dict from configurationutils import pretrainedconfig erniempretrainedconfigarchivemap susnatoerniembasepytorch https huggingface cosusnatoerniembasepytorchblobmainconfig json susnatoerniemlargepytorch https huggingface cosusnatoerniemlargepytorchblobmainconfig json class erniemconfigpretrainedconfig r this is the configuration class to store the configuration of a erniemmodel it is used to instantiate a erniem model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the erniem susnatoerniembasepytorchhttps huggingface cosusnatoerniembasepytorch architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocabsize int optional defaults to 250002 vocabulary size of inputsids in erniemmodel also is the vocab size of token embedding matrix defines the number of different tokens that can be represented by the inputsids passed when calling erniemmodel hiddensize int optional defaults to 768 dimensionality of the embedding layer encoder layers and pooler layer numhiddenlayers int optional defaults to 12 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 12 number of attention heads for each attention layer in the transformer encoder intermediatesize int optional defaults to 3072 dimensionality of the feedforward ff layer in the encoder input tensors to feedforward layers are firstly projected from hiddensize to intermediatesize and then projected back to hiddensize typically intermediatesize is larger than hiddensize hiddenact str optional defaults to gelu the nonlinear activation function in the feedforward layer gelu relu and any other torch supported activation functions are supported hiddendropoutprob float optional defaults to 0 1 the dropout probability for all fully connected layers in the embeddings and encoder attentionprobsdropoutprob float optional defaults to 0 1 the dropout probability used in multiheadattention in all encoder layers to drop some attention target actdropout float optional defaults to 0 0 this dropout probability is used in erniemencoderlayer after activation maxpositionembeddings int optional defaults to 512 the maximum value of the dimensionality of position encoding which dictates the maximum supported length of an input sequence layernormeps float optional defaults to 1e05 the epsilon used by the layer normalization layers classifierdropout float optional the dropout ratio for the classification head initializerrange float optional defaults to 0 02 the standard deviation of the normal initializer for initializing all weight matrices padtokenidint optional defaults to 1 the index of padding token in the token vocabulary a normalinitializer initializes weight matrices as normal distributions see erniempretrainedmodel initweights for how weights are initialized in erniemmodel coding utf 8 2023 xuan ouyang shuohuan wang chao pang yu sun hao tian hua wu haifeng wang and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license erniem model configuration adapted from original paddlenlp repository https github com paddlepaddle paddlenlp blob develop paddlenlp transformers ernie_m configuration py this is the configuration class to store the configuration of a erniemmodel it is used to instantiate a ernie m model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the ernie m susnato ernie m base_pytorch https huggingface co susnato ernie m base_pytorch architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocab_size int optional defaults to 250002 vocabulary size of inputs_ids in erniemmodel also is the vocab size of token embedding matrix defines the number of different tokens that can be represented by the inputs_ids passed when calling erniemmodel hidden_size int optional defaults to 768 dimensionality of the embedding layer encoder layers and pooler layer num_hidden_layers int optional defaults to 12 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 12 number of attention heads for each attention layer in the transformer encoder intermediate_size int optional defaults to 3072 dimensionality of the feed forward ff layer in the encoder input tensors to feed forward layers are firstly projected from hidden_size to intermediate_size and then projected back to hidden_size typically intermediate_size is larger than hidden_size hidden_act str optional defaults to gelu the non linear activation function in the feed forward layer gelu relu and any other torch supported activation functions are supported hidden_dropout_prob float optional defaults to 0 1 the dropout probability for all fully connected layers in the embeddings and encoder attention_probs_dropout_prob float optional defaults to 0 1 the dropout probability used in multiheadattention in all encoder layers to drop some attention target act_dropout float optional defaults to 0 0 this dropout probability is used in erniemencoderlayer after activation max_position_embeddings int optional defaults to 512 the maximum value of the dimensionality of position encoding which dictates the maximum supported length of an input sequence layer_norm_eps float optional defaults to 1e 05 the epsilon used by the layer normalization layers classifier_dropout float optional the dropout ratio for the classification head initializer_range float optional defaults to 0 02 the standard deviation of the normal initializer for initializing all weight matrices pad_token_id int optional defaults to 1 the index of padding token in the token vocabulary a normal_initializer initializes weight matrices as normal distributions see erniempretrainedmodel _init_weights for how weights are initialized in erniemmodel
from __future__ import annotations

from typing import Dict

from ...configuration_utils import PretrainedConfig


ERNIE_M_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "susnato/ernie-m-base_pytorch": "https://huggingface.co/susnato/ernie-m-base_pytorch/blob/main/config.json",
    "susnato/ernie-m-large_pytorch": "https://huggingface.co/susnato/ernie-m-large_pytorch/blob/main/config.json",
}


class ErnieMConfig(PretrainedConfig):
    r

    model_type = "ernie_m"
    attribute_map: Dict[str, str] = {"dropout": "classifier_dropout", "num_classes": "num_labels"}

    def __init__(
        self,
        vocab_size: int = 250002,
        hidden_size: int = 768,
        num_hidden_layers: int = 12,
        num_attention_heads: int = 12,
        intermediate_size: int = 3072,
        hidden_act: str = "gelu",
        hidden_dropout_prob: float = 0.1,
        attention_probs_dropout_prob: float = 0.1,
        max_position_embeddings: int = 514,
        initializer_range: float = 0.02,
        pad_token_id: int = 1,
        layer_norm_eps: float = 1e-05,
        classifier_dropout=None,
        is_decoder=False,
        act_dropout=0.0,
        **kwargs,
    ):
        super().__init__(pad_token_id=pad_token_id, **kwargs)
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.hidden_dropout_prob = hidden_dropout_prob
        self.attention_probs_dropout_prob = attention_probs_dropout_prob
        self.max_position_embeddings = max_position_embeddings
        self.initializer_range = initializer_range
        self.layer_norm_eps = layer_norm_eps
        self.classifier_dropout = classifier_dropout
        self.is_decoder = is_decoder
        self.act_dropout = act_dropout