codingutf8 2023 murgelab and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tvlt model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername tvltpretrainedconfigarchivemap zinengtangtvltbase https huggingface cozinengtangtvltbaseblobmainconfig json class tvltconfigpretrainedconfig r this is the configuration class to store the configuration of a tvltmodel it is used to instantiate a tvlt model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the tvlt zinengtangtvltbasehttps huggingface cozinengtangtvltbase architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args imagesize int optional defaults to 224 the size resolution of each image spectrogramlength int optional defaults to 2048 the time length of each audio spectrogram frequencylength int optional defaults to 128 the frequency length of audio spectrogram imagepatchsize listint optional defaults to 16 16 the size resolution of each image patch audiopatchsize listint optional defaults to 16 16 the size resolution of each audio patch numimagechannels int optional defaults to 3 the number of input image channels numaudiochannels int optional defaults to 1 the number of input audio channels numframes int optional defaults to 8 the maximum number of frames for an input video hiddensize int optional defaults to 768 dimensionality of the encoder layers and the pooler layer numhiddenlayers int optional defaults to 12 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 12 number of attention heads for each attention layer in the transformer encoder intermediatesize int optional defaults to 3072 dimensionality of the intermediate i e feedforward layer in the transformer encoder hiddenact str or function optional defaults to gelu the nonlinear activation function function or string in the encoder and pooler if string gelu relu selu and gelunew are supported hiddendropoutprob float optional defaults to 0 0 the dropout probabilitiy for all fully connected layers in the embeddings encoder and pooler attentionprobsdropoutprob float optional defaults to 0 0 the dropout ratio for the attention probabilities initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices layernormeps float optional defaults to 1e06 the epsilon used by the layer normalization layers qkvbias bool optional defaults to true whether to add a bias to the queries keys and values usemeanpooling bool optional defaults to false whether to mean pool the final hidden states instead of using the final hidden state of the cls token decodernumattentionheads int optional defaults to 16 number of attention heads for each attention layer in the decoder decoderhiddensize int optional defaults to 512 dimensionality of the decoder decodernumhiddenlayers int optional defaults to 8 number of hidden layers in the decoder decoderintermediatesize int optional defaults to 2048 dimensionality of the intermediate i e feedforward layer in the decoder pixelmaskratio float optional defaults to 0 75 image patch masking ratio audiomaskratio float optional defaults to 0 15 audio patch masking ratio audiomasktype str optional defaults to framelevel audio patch masking type choose between framelevel and patchlevel taskmatching bool optional defaults to true whether to use vision audio matching task in pretraining taskmae bool optional defaults to true whether to use the masked autoencoder mae in pretraining losstype str optional defaults to classification loss types including regression and classification example python from transformers import tvltconfig tvltmodel initializing a tvlt zinengtangtvltbase style configuration configuration tvltconfig initializing a model with random weights from the zinengtangtvltbase style configuration model tvltmodelconfiguration accessing the model configuration configuration model config modeltype tvlt def init self imagesize224 spectrogramlength2048 frequencylength128 imagepatchsize16 16 audiopatchsize16 16 numimagechannels3 numaudiochannels1 numframes8 hiddensize768 numhiddenlayers12 numattentionheads12 intermediatesize3072 hiddenactgelu hiddendropoutprob0 0 attentionprobsdropoutprob0 0 initializerrange0 02 layernormeps1e6 qkvbiastrue usemeanpoolingfalse decodernumattentionheads16 decoderhiddensize512 decodernumhiddenlayers8 decoderintermediatesize2048 pixelmaskratio0 75 audiomaskratio0 15 audiomasktypeframelevel taskmatchingtrue taskmaetrue losstypeclassification kwargs super initkwargs if audiomasktype not in framelevel patchlevel raise valueerror audiomasktype must be one of two acceptable strategies framelevel patchlevel fgot audiomasktype self imagesize imagesize self spectrogramlength spectrogramlength self frequencylength frequencylength self imagepatchsize imagepatchsize self audiopatchsize audiopatchsize self numimagechannels numimagechannels self numaudiochannels numaudiochannels self numframes numframes self hiddensize hiddensize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads self intermediatesize intermediatesize self hiddenact hiddenact self hiddendropoutprob hiddendropoutprob self attentionprobsdropoutprob attentionprobsdropoutprob self initializerrange initializerrange self layernormeps layernormeps self qkvbias qkvbias self usemeanpooling usemeanpooling self decodernumattentionheads decodernumattentionheads self decoderhiddensize decoderhiddensize self decodernumhiddenlayers decodernumhiddenlayers self decoderintermediatesize decoderintermediatesize self pixelmaskratio pixelmaskratio self audiomaskratio audiomaskratio self audiomasktype audiomasktype self taskmatching taskmatching self taskmae taskmae self losstype losstype coding utf 8 2023 murge lab and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license tvlt model configuration this is the configuration class to store the configuration of a tvltmodel it is used to instantiate a tvlt model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the tvlt zinengtang tvlt base https huggingface co zinengtang tvlt base architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args image_size int optional defaults to 224 the size resolution of each image spectrogram_length int optional defaults to 2048 the time length of each audio spectrogram frequency_length int optional defaults to 128 the frequency length of audio spectrogram image_patch_size list int optional defaults to 16 16 the size resolution of each image patch audio_patch_size list int optional defaults to 16 16 the size resolution of each audio patch num_image_channels int optional defaults to 3 the number of input image channels num_audio_channels int optional defaults to 1 the number of input audio channels num_frames int optional defaults to 8 the maximum number of frames for an input video hidden_size int optional defaults to 768 dimensionality of the encoder layers and the pooler layer num_hidden_layers int optional defaults to 12 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 12 number of attention heads for each attention layer in the transformer encoder intermediate_size int optional defaults to 3072 dimensionality of the intermediate i e feed forward layer in the transformer encoder hidden_act str or function optional defaults to gelu the non linear activation function function or string in the encoder and pooler if string gelu relu selu and gelu_new are supported hidden_dropout_prob float optional defaults to 0 0 the dropout probabilitiy for all fully connected layers in the embeddings encoder and pooler attention_probs_dropout_prob float optional defaults to 0 0 the dropout ratio for the attention probabilities initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices layer_norm_eps float optional defaults to 1e 06 the epsilon used by the layer normalization layers qkv_bias bool optional defaults to true whether to add a bias to the queries keys and values use_mean_pooling bool optional defaults to false whether to mean pool the final hidden states instead of using the final hidden state of the cls token decoder_num_attention_heads int optional defaults to 16 number of attention heads for each attention layer in the decoder decoder_hidden_size int optional defaults to 512 dimensionality of the decoder decoder_num_hidden_layers int optional defaults to 8 number of hidden layers in the decoder decoder_intermediate_size int optional defaults to 2048 dimensionality of the intermediate i e feed forward layer in the decoder pixel_mask_ratio float optional defaults to 0 75 image patch masking ratio audio_mask_ratio float optional defaults to 0 15 audio patch masking ratio audio_mask_type str optional defaults to frame level audio patch masking type choose between frame level and patch level task_matching bool optional defaults to true whether to use vision audio matching task in pretraining task_mae bool optional defaults to true whether to use the masked auto encoder mae in pretraining loss_type str optional defaults to classification loss types including regression and classification example python from transformers import tvltconfig tvltmodel initializing a tvlt zinengtang tvlt base style configuration configuration tvltconfig initializing a model with random weights from the zinengtang tvlt base style configuration model tvltmodel configuration accessing the model configuration configuration model config
from ...configuration_utils import PretrainedConfig
from ...utils import logging


logger = logging.get_logger(__name__)

TVLT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "ZinengTang/tvlt-base": "https://huggingface.co/ZinengTang/tvlt-base/blob/main/config.json",
}


class TvltConfig(PretrainedConfig):
    r

    model_type = "tvlt"

    def __init__(
        self,
        image_size=224,
        spectrogram_length=2048,
        frequency_length=128,
        image_patch_size=[16, 16],
        audio_patch_size=[16, 16],
        num_image_channels=3,
        num_audio_channels=1,
        num_frames=8,
        hidden_size=768,
        num_hidden_layers=12,
        num_attention_heads=12,
        intermediate_size=3072,
        hidden_act="gelu",
        hidden_dropout_prob=0.0,
        attention_probs_dropout_prob=0.0,
        initializer_range=0.02,
        layer_norm_eps=1e-6,
        qkv_bias=True,
        use_mean_pooling=False,
        decoder_num_attention_heads=16,
        decoder_hidden_size=512,
        decoder_num_hidden_layers=8,
        decoder_intermediate_size=2048,
        pixel_mask_ratio=0.75,
        audio_mask_ratio=0.15,
        audio_mask_type="frame-level",
        task_matching=True,
        task_mae=True,
        loss_type="classification",
        **kwargs,
    ):
        super().__init__(**kwargs)

        if audio_mask_type not in ("frame-level", "patch_level"):
            raise ValueError(
                "audio_mask_type must be one of two acceptable strategies - {'frame_level', 'patch-level') "
                f"got {audio_mask_type}"
            )

        self.image_size = image_size
        self.spectrogram_length = spectrogram_length
        self.frequency_length = frequency_length
        self.image_patch_size = image_patch_size
        self.audio_patch_size = audio_patch_size
        self.num_image_channels = num_image_channels
        self.num_audio_channels = num_audio_channels
        self.num_frames = num_frames

        self.hidden_size = hidden_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.intermediate_size = intermediate_size
        self.hidden_act = hidden_act
        self.hidden_dropout_prob = hidden_dropout_prob
        self.attention_probs_dropout_prob = attention_probs_dropout_prob
        self.initializer_range = initializer_range
        self.layer_norm_eps = layer_norm_eps
        self.qkv_bias = qkv_bias
        self.use_mean_pooling = use_mean_pooling

        self.decoder_num_attention_heads = decoder_num_attention_heads
        self.decoder_hidden_size = decoder_hidden_size
        self.decoder_num_hidden_layers = decoder_num_hidden_layers
        self.decoder_intermediate_size = decoder_intermediate_size
        self.pixel_mask_ratio = pixel_mask_ratio
        self.audio_mask_ratio = audio_mask_ratio
        self.audio_mask_type = audio_mask_type

        self.task_matching = task_matching
        self.task_mae = task_mae
        self.loss_type = loss_type