codingutf8 2023 the intel aia team s and huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tvp model configuration import copy from configurationutils import pretrainedconfig from utils import logging from auto import configmapping logger logging getloggername tvppretrainedconfigarchivemap inteltvpbase https huggingface cointeltvpbaseresolvemainconfig json class tvpconfigpretrainedconfig r this is the configuration class to store the configuration of a tvpmodel it is used to instantiate an tvp model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the tvp inteltvpbasehttps huggingface cointeltvpbase architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args backboneconfig pretrainedconfig or dict optional the configuration of the backbone model distancelossweight float optional defaults to 1 0 the weight of distance loss durationlossweight float optional defaults to 0 1 the weight of duration loss visualpromptertype str optional defaults to framepad visual prompt type the type of padding framepad means padding on each frame should be one of framepad or framedownpad visualprompterapply str optional defaults to replace the way of applying visual prompt replace means use the value of prompt to change the original value in visual inputs should be one of replace or add or remove visualpromptsize int optional defaults to 96 the size of visual prompt maximgsize int optional defaults to 448 the maximum size of frame numframes int optional defaults to 48 the number of frames extracted from a video vocabsize int optional defaults to 30522 vocabulary size of the tvp text model defines the number of different tokens that can be represented by the inputsids passed when calling tvpmodel hiddensize int optional defaults to 768 dimensionality of the encoder layers intermediatesize int optional defaults to 3072 dimensionality of the intermediate i e feedforward layer in the transformer encoder numhiddenlayers int optional defaults to 12 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 12 number of attention heads for each attention layer in the transformer encoder maxpositionembeddings int optional defaults to 512 the maximum sequence length that this model might ever be used with typically set this to something large just in case e g 512 or 1024 or 2048 maxgridcolpositionembeddings int optional defaults to 100 the largest number of horizontal patches from a video frame maxgridrowpositionembeddings int optional defaults to 100 the largest number of vertical patches from a video frame hiddendropoutprob float optional defaults to 0 1 the dropout probability of hidden layers hiddenact str or function optional defaults to gelu the nonlinear activation function function or string in the encoder and pooler if string gelu relu selu and gelunew quickgelu are supported layernormeps float optional defaults to 1e12 the epsilon used by the layer normalization layers initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices attentionprobsdropoutprob float optional defaults to 0 1 the dropout probability of attention layers instantiate a tvpconfig or a derived class from a pretrained backbone model configuration args backboneconfig pretrainedconfig the backbone configuration returns tvpconfig an instance of a configuration object serializes this instance to a python dictionary override the default pretrainedconfig todict returns dictstr any dictionary of all the attributes that make up this configuration instance coding utf 8 2023 the intel aia team s and huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tvp model configuration this is the configuration class to store the configuration of a tvpmodel it is used to instantiate an tvp model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the tvp intel tvp base https huggingface co intel tvp base architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args backbone_config pretrainedconfig or dict optional the configuration of the backbone model distance_loss_weight float optional defaults to 1 0 the weight of distance loss duration_loss_weight float optional defaults to 0 1 the weight of duration loss visual_prompter_type str optional defaults to framepad visual prompt type the type of padding framepad means padding on each frame should be one of framepad or framedownpad visual_prompter_apply str optional defaults to replace the way of applying visual prompt replace means use the value of prompt to change the original value in visual inputs should be one of replace or add or remove visual_prompt_size int optional defaults to 96 the size of visual prompt max_img_size int optional defaults to 448 the maximum size of frame num_frames int optional defaults to 48 the number of frames extracted from a video vocab_size int optional defaults to 30522 vocabulary size of the tvp text model defines the number of different tokens that can be represented by the inputs_ids passed when calling tvpmodel hidden_size int optional defaults to 768 dimensionality of the encoder layers intermediate_size int optional defaults to 3072 dimensionality of the intermediate i e feed forward layer in the transformer encoder num_hidden_layers int optional defaults to 12 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 12 number of attention heads for each attention layer in the transformer encoder max_position_embeddings int optional defaults to 512 the maximum sequence length that this model might ever be used with typically set this to something large just in case e g 512 or 1024 or 2048 max_grid_col_position_embeddings int optional defaults to 100 the largest number of horizontal patches from a video frame max_grid_row_position_embeddings int optional defaults to 100 the largest number of vertical patches from a video frame hidden_dropout_prob float optional defaults to 0 1 the dropout probability of hidden layers hidden_act str or function optional defaults to gelu the non linear activation function function or string in the encoder and pooler if string gelu relu selu and gelu_new quick_gelu are supported layer_norm_eps float optional defaults to 1e 12 the epsilon used by the layer normalization layers initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices attention_probs_dropout_prob float optional defaults to 0 1 the dropout probability of attention layers instantiate a tvpconfig or a derived class from a pre trained backbone model configuration args backbone_config pretrainedconfig the backbone configuration returns tvpconfig an instance of a configuration object serializes this instance to a python dictionary override the default pretrainedconfig to_dict returns dict str any dictionary of all the attributes that make up this configuration instance
import copy

from ...configuration_utils import PretrainedConfig
from ...utils import logging
from ..auto import CONFIG_MAPPING


logger = logging.get_logger(__name__)


TVP_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "Intel/tvp-base": "https://huggingface.co/Intel/tvp-base/resolve/main/config.json",
}


class TvpConfig(PretrainedConfig):
    r

    model_type = "tvp"

    def __init__(
        self,
        backbone_config=None,
        distance_loss_weight=1.0,
        duration_loss_weight=0.1,
        visual_prompter_type="framepad",
        visual_prompter_apply="replace",
        visual_prompt_size=96,
        max_img_size=448,
        num_frames=48,
        vocab_size=30522,
        hidden_size=768,
        intermediate_size=3072,
        num_hidden_layers=12,
        num_attention_heads=12,
        max_position_embeddings=512,
        max_grid_col_position_embeddings=100,
        max_grid_row_position_embeddings=100,
        hidden_dropout_prob=0.1,
        hidden_act="gelu",
        layer_norm_eps=1e-12,
        initializer_range=0.02,
        attention_probs_dropout_prob=0.1,
        **kwargs,
    ):
        super().__init__(**kwargs)

        if backbone_config is None:
            logger.info("`backbone_config` is `None`. Initializing the config with the default `ResNet` backbone.")
            backbone_config = CONFIG_MAPPING["resnet"](out_features=["stage4"])
        elif isinstance(backbone_config, dict):
            backbone_model_type = backbone_config.get("model_type")
            config_class = CONFIG_MAPPING[backbone_model_type]
            backbone_config = config_class.from_dict(backbone_config)

        self.backbone_config = backbone_config
        self.distance_loss_weight = distance_loss_weight
        self.duration_loss_weight = duration_loss_weight
        self.visual_prompter_type = visual_prompter_type
        self.visual_prompter_apply = visual_prompter_apply
        self.visual_prompt_size = visual_prompt_size
        self.max_img_size = max_img_size
        self.num_frames = num_frames
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.max_position_embeddings = max_position_embeddings
        self.max_grid_col_position_embeddings = max_grid_col_position_embeddings
        self.max_grid_row_position_embeddings = max_grid_row_position_embeddings
        self.layer_norm_eps = layer_norm_eps
        self.hidden_dropout_prob = hidden_dropout_prob
        self.hidden_act = hidden_act
        self.initializer_range = initializer_range
        self.attention_probs_dropout_prob = attention_probs_dropout_prob

    @classmethod
    def from_backbone_config(cls, backbone_config: PretrainedConfig, **kwargs):
        
        return cls(backbone_config=backbone_config, **kwargs)

    def to_dict(self):
        
        output = copy.deepcopy(self.__dict__)
        if output["backbone_config"] is not None:
            output["backbone_config"] = self.backbone_config.to_dict()
        output["model_type"] = self.__class__.model_type
        return output