codingutf8 2023 the kakao enterprise s and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache orglicenseslicense2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license vits model configuration from configurationutils import pretrainedconfig from utils import logging logger logging getloggername vitspretrainedconfigarchivemap facebookmmsttseng https huggingface cofacebookmmsttsengresolvemainconfig json class vitsconfigpretrainedconfig r this is the configuration class to store the configuration of a vitsmodel it is used to instantiate a vits model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the vits facebookmmsttsenghttps huggingface cofacebookmmsttseng architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocabsize int optional defaults to 38 vocabulary size of the vits model defines the number of different tokens that can be represented by the inputsids passed to the forward method of vitsmodel hiddensize int optional defaults to 192 dimensionality of the text encoder layers numhiddenlayers int optional defaults to 6 number of hidden layers in the transformer encoder numattentionheads int optional defaults to 2 number of attention heads for each attention layer in the transformer encoder windowsize int optional defaults to 4 window size for the relative positional embeddings in the attention layers of the transformer encoder usebias bool optional defaults to true whether to use bias in the key query value projection layers in the transformer encoder ffndim int optional defaults to 768 dimensionality of the intermediate i e feedforward layer in the transformer encoder layerdrop float optional defaults to 0 1 the layerdrop probability for the encoder see the layerdrop papersee https arxiv orgabs1909 11556 for more details ffnkernelsize int optional defaults to 3 kernel size of the 1d convolution layers used by the feedforward network in the transformer encoder flowsize int optional defaults to 192 dimensionality of the flow layers spectrogrambins int optional defaults to 513 number of frequency bins in the target spectrogram hiddenact str or function optional defaults to relu the nonlinear activation function function or string in the encoder and pooler if string gelu relu selu and gelunew are supported hiddendropout float optional defaults to 0 1 the dropout probability for all fully connected layers in the embeddings and encoder attentiondropout float optional defaults to 0 1 the dropout ratio for the attention probabilities activationdropout float optional defaults to 0 1 the dropout ratio for activations inside the fully connected layer initializerrange float optional defaults to 0 02 the standard deviation of the truncatednormalinitializer for initializing all weight matrices layernormeps float optional defaults to 1e05 the epsilon used by the layer normalization layers usestochasticdurationprediction bool optional defaults to true whether to use the stochastic duration prediction module or the regular duration predictor numspeakers int optional defaults to 1 number of speakers if this is a multispeaker model speakerembeddingsize int optional defaults to 0 number of channels used by the speaker embeddings is zero for singlespeaker models upsampleinitialchannel int optional defaults to 512 the number of input channels into the hifigan upsampling network upsamplerates tupleint or listint optional defaults to 8 8 2 2 a tuple of integers defining the stride of each 1d convolutional layer in the hifigan upsampling network the length of upsamplerates defines the number of convolutional layers and has to match the length of upsamplekernelsizes upsamplekernelsizes tupleint or listint optional defaults to 16 16 4 4 a tuple of integers defining the kernel size of each 1d convolutional layer in the hifigan upsampling network the length of upsamplekernelsizes defines the number of convolutional layers and has to match the length of upsamplerates resblockkernelsizes tupleint or listint optional defaults to 3 7 11 a tuple of integers defining the kernel sizes of the 1d convolutional layers in the hifigan multireceptive field fusion mrf module resblockdilationsizes tupletupleint or listlistint optional defaults to 1 3 5 1 3 5 1 3 5 a nested tuple of integers defining the dilation rates of the dilated 1d convolutional layers in the hifigan multireceptive field fusion mrf module leakyreluslope float optional defaults to 0 1 the angle of the negative slope used by the leaky relu activation depthseparablechannels int optional defaults to 2 number of channels to use in each depthseparable block depthseparablenumlayers int optional defaults to 3 number of convolutional layers to use in each depthseparable block durationpredictorflowbins int optional defaults to 10 number of channels to map using the unonstrained rational spline in the duration predictor model durationpredictortailbound float optional defaults to 5 0 value of the tail bin boundary when computing the unconstrained rational spline in the duration predictor model durationpredictorkernelsize int optional defaults to 3 kernel size of the 1d convolution layers used in the duration predictor model durationpredictordropout float optional defaults to 0 5 the dropout ratio for the duration predictor model durationpredictornumflows int optional defaults to 4 number of flow stages used by the duration predictor model durationpredictorfilterchannels int optional defaults to 256 number of channels for the convolution layers used in the duration predictor model priorencodernumflows int optional defaults to 4 number of flow stages used by the prior encoder flow model priorencodernumwavenetlayers int optional defaults to 4 number of wavenet layers used by the prior encoder flow model posteriorencodernumwavenetlayers int optional defaults to 16 number of wavenet layers used by the posterior encoder model wavenetkernelsize int optional defaults to 5 kernel size of the 1d convolution layers used in the wavenet model wavenetdilationrate int optional defaults to 1 dilation rates of the dilated 1d convolutional layers used in the wavenet model wavenetdropout float optional defaults to 0 0 the dropout ratio for the wavenet layers speakingrate float optional defaults to 1 0 speaking rate larger values give faster synthesised speech noisescale float optional defaults to 0 667 how random the speech prediction is larger values create more variation in the predicted speech noisescaleduration float optional defaults to 0 8 how random the duration prediction is larger values create more variation in the predicted durations samplingrate int optional defaults to 16000 the sampling rate at which the output audio waveform is digitalized expressed in hertz hz example python from transformers import vitsmodel vitsconfig initializing a facebookmmsttseng style configuration configuration vitsconfig initializing a model with random weights from the facebookmmsttseng style configuration model vitsmodelconfiguration accessing the model configuration configuration model config modeltype vits def init self vocabsize38 hiddensize192 numhiddenlayers6 numattentionheads2 windowsize4 usebiastrue ffndim768 layerdrop0 1 ffnkernelsize3 flowsize192 spectrogrambins513 hiddenactrelu hiddendropout0 1 attentiondropout0 1 activationdropout0 1 initializerrange0 02 layernormeps1e5 usestochasticdurationpredictiontrue numspeakers1 speakerembeddingsize0 upsampleinitialchannel512 upsamplerates8 8 2 2 upsamplekernelsizes16 16 4 4 resblockkernelsizes3 7 11 resblockdilationsizes1 3 5 1 3 5 1 3 5 leakyreluslope0 1 depthseparablechannels2 depthseparablenumlayers3 durationpredictorflowbins10 durationpredictortailbound5 0 durationpredictorkernelsize3 durationpredictordropout0 5 durationpredictornumflows4 durationpredictorfilterchannels256 priorencodernumflows4 priorencodernumwavenetlayers4 posteriorencodernumwavenetlayers16 wavenetkernelsize5 wavenetdilationrate1 wavenetdropout0 0 speakingrate1 0 noisescale0 667 noisescaleduration0 8 samplingrate16000 kwargs self vocabsize vocabsize self hiddensize hiddensize self numhiddenlayers numhiddenlayers self numattentionheads numattentionheads self windowsize windowsize self usebias usebias self ffndim ffndim self layerdrop layerdrop self ffnkernelsize ffnkernelsize self flowsize flowsize self spectrogrambins spectrogrambins self hiddenact hiddenact self hiddendropout hiddendropout self attentiondropout attentiondropout self activationdropout activationdropout self initializerrange initializerrange self layernormeps layernormeps self usestochasticdurationprediction usestochasticdurationprediction self numspeakers numspeakers self speakerembeddingsize speakerembeddingsize self upsampleinitialchannel upsampleinitialchannel self upsamplerates upsamplerates self upsamplekernelsizes upsamplekernelsizes self resblockkernelsizes resblockkernelsizes self resblockdilationsizes resblockdilationsizes self leakyreluslope leakyreluslope self depthseparablechannels depthseparablechannels self depthseparablenumlayers depthseparablenumlayers self durationpredictorflowbins durationpredictorflowbins self durationpredictortailbound durationpredictortailbound self durationpredictorkernelsize durationpredictorkernelsize self durationpredictordropout durationpredictordropout self durationpredictornumflows durationpredictornumflows self durationpredictorfilterchannels durationpredictorfilterchannels self priorencodernumflows priorencodernumflows self priorencodernumwavenetlayers priorencodernumwavenetlayers self posteriorencodernumwavenetlayers posteriorencodernumwavenetlayers self wavenetkernelsize wavenetkernelsize self wavenetdilationrate wavenetdilationrate self wavenetdropout wavenetdropout self speakingrate speakingrate self noisescale noisescale self noisescaleduration noisescaleduration self samplingrate samplingrate if lenupsamplekernelsizes lenupsamplerates raise valueerror fthe length of upsamplekernelsizes lenupsamplekernelsizes must match the length of fupsamplerates lenupsamplerates super initkwargs coding utf 8 2023 the kakao enterprise s and the huggingface inc team licensed under the apache license version 2 0 the license you may not use this file except in compliance with the license you may obtain a copy of the license at http www apache org licenses license 2 0 unless required by applicable law or agreed to in writing software distributed under the license is distributed on an as is basis without warranties or conditions of any kind either express or implied see the license for the specific language governing permissions and limitations under the license vits model configuration this is the configuration class to store the configuration of a vitsmodel it is used to instantiate a vits model according to the specified arguments defining the model architecture instantiating a configuration with the defaults will yield a similar configuration to that of the vits facebook mms tts eng https huggingface co facebook mms tts eng architecture configuration objects inherit from pretrainedconfig and can be used to control the model outputs read the documentation from pretrainedconfig for more information args vocab_size int optional defaults to 38 vocabulary size of the vits model defines the number of different tokens that can be represented by the inputs_ids passed to the forward method of vitsmodel hidden_size int optional defaults to 192 dimensionality of the text encoder layers num_hidden_layers int optional defaults to 6 number of hidden layers in the transformer encoder num_attention_heads int optional defaults to 2 number of attention heads for each attention layer in the transformer encoder window_size int optional defaults to 4 window size for the relative positional embeddings in the attention layers of the transformer encoder use_bias bool optional defaults to true whether to use bias in the key query value projection layers in the transformer encoder ffn_dim int optional defaults to 768 dimensionality of the intermediate i e feed forward layer in the transformer encoder layerdrop float optional defaults to 0 1 the layerdrop probability for the encoder see the layerdrop paper see https arxiv org abs 1909 11556 for more details ffn_kernel_size int optional defaults to 3 kernel size of the 1d convolution layers used by the feed forward network in the transformer encoder flow_size int optional defaults to 192 dimensionality of the flow layers spectrogram_bins int optional defaults to 513 number of frequency bins in the target spectrogram hidden_act str or function optional defaults to relu the non linear activation function function or string in the encoder and pooler if string gelu relu selu and gelu_new are supported hidden_dropout float optional defaults to 0 1 the dropout probability for all fully connected layers in the embeddings and encoder attention_dropout float optional defaults to 0 1 the dropout ratio for the attention probabilities activation_dropout float optional defaults to 0 1 the dropout ratio for activations inside the fully connected layer initializer_range float optional defaults to 0 02 the standard deviation of the truncated_normal_initializer for initializing all weight matrices layer_norm_eps float optional defaults to 1e 05 the epsilon used by the layer normalization layers use_stochastic_duration_prediction bool optional defaults to true whether to use the stochastic duration prediction module or the regular duration predictor num_speakers int optional defaults to 1 number of speakers if this is a multi speaker model speaker_embedding_size int optional defaults to 0 number of channels used by the speaker embeddings is zero for single speaker models upsample_initial_channel int optional defaults to 512 the number of input channels into the hifi gan upsampling network upsample_rates tuple int or list int optional defaults to 8 8 2 2 a tuple of integers defining the stride of each 1d convolutional layer in the hifi gan upsampling network the length of upsample_rates defines the number of convolutional layers and has to match the length of upsample_kernel_sizes upsample_kernel_sizes tuple int or list int optional defaults to 16 16 4 4 a tuple of integers defining the kernel size of each 1d convolutional layer in the hifi gan upsampling network the length of upsample_kernel_sizes defines the number of convolutional layers and has to match the length of upsample_rates resblock_kernel_sizes tuple int or list int optional defaults to 3 7 11 a tuple of integers defining the kernel sizes of the 1d convolutional layers in the hifi gan multi receptive field fusion mrf module resblock_dilation_sizes tuple tuple int or list list int optional defaults to 1 3 5 1 3 5 1 3 5 a nested tuple of integers defining the dilation rates of the dilated 1d convolutional layers in the hifi gan multi receptive field fusion mrf module leaky_relu_slope float optional defaults to 0 1 the angle of the negative slope used by the leaky relu activation depth_separable_channels int optional defaults to 2 number of channels to use in each depth separable block depth_separable_num_layers int optional defaults to 3 number of convolutional layers to use in each depth separable block duration_predictor_flow_bins int optional defaults to 10 number of channels to map using the unonstrained rational spline in the duration predictor model duration_predictor_tail_bound float optional defaults to 5 0 value of the tail bin boundary when computing the unconstrained rational spline in the duration predictor model duration_predictor_kernel_size int optional defaults to 3 kernel size of the 1d convolution layers used in the duration predictor model duration_predictor_dropout float optional defaults to 0 5 the dropout ratio for the duration predictor model duration_predictor_num_flows int optional defaults to 4 number of flow stages used by the duration predictor model duration_predictor_filter_channels int optional defaults to 256 number of channels for the convolution layers used in the duration predictor model prior_encoder_num_flows int optional defaults to 4 number of flow stages used by the prior encoder flow model prior_encoder_num_wavenet_layers int optional defaults to 4 number of wavenet layers used by the prior encoder flow model posterior_encoder_num_wavenet_layers int optional defaults to 16 number of wavenet layers used by the posterior encoder model wavenet_kernel_size int optional defaults to 5 kernel size of the 1d convolution layers used in the wavenet model wavenet_dilation_rate int optional defaults to 1 dilation rates of the dilated 1d convolutional layers used in the wavenet model wavenet_dropout float optional defaults to 0 0 the dropout ratio for the wavenet layers speaking_rate float optional defaults to 1 0 speaking rate larger values give faster synthesised speech noise_scale float optional defaults to 0 667 how random the speech prediction is larger values create more variation in the predicted speech noise_scale_duration float optional defaults to 0 8 how random the duration prediction is larger values create more variation in the predicted durations sampling_rate int optional defaults to 16000 the sampling rate at which the output audio waveform is digitalized expressed in hertz hz example python from transformers import vitsmodel vitsconfig initializing a facebook mms tts eng style configuration configuration vitsconfig initializing a model with random weights from the facebook mms tts eng style configuration model vitsmodel configuration accessing the model configuration configuration model config
from ...configuration_utils import PretrainedConfig
from ...utils import logging


logger = logging.get_logger(__name__)

VITS_PRETRAINED_CONFIG_ARCHIVE_MAP = {
    "facebook/mms-tts-eng": "https://huggingface.co/facebook/mms-tts-eng/resolve/main/config.json",
}


class VitsConfig(PretrainedConfig):
    r

    model_type = "vits"

    def __init__(
        self,
        vocab_size=38,
        hidden_size=192,
        num_hidden_layers=6,
        num_attention_heads=2,
        window_size=4,
        use_bias=True,
        ffn_dim=768,
        layerdrop=0.1,
        ffn_kernel_size=3,
        flow_size=192,
        spectrogram_bins=513,
        hidden_act="relu",
        hidden_dropout=0.1,
        attention_dropout=0.1,
        activation_dropout=0.1,
        initializer_range=0.02,
        layer_norm_eps=1e-5,
        use_stochastic_duration_prediction=True,
        num_speakers=1,
        speaker_embedding_size=0,
        upsample_initial_channel=512,
        upsample_rates=[8, 8, 2, 2],
        upsample_kernel_sizes=[16, 16, 4, 4],
        resblock_kernel_sizes=[3, 7, 11],
        resblock_dilation_sizes=[[1, 3, 5], [1, 3, 5], [1, 3, 5]],
        leaky_relu_slope=0.1,
        depth_separable_channels=2,
        depth_separable_num_layers=3,
        duration_predictor_flow_bins=10,
        duration_predictor_tail_bound=5.0,
        duration_predictor_kernel_size=3,
        duration_predictor_dropout=0.5,
        duration_predictor_num_flows=4,
        duration_predictor_filter_channels=256,
        prior_encoder_num_flows=4,
        prior_encoder_num_wavenet_layers=4,
        posterior_encoder_num_wavenet_layers=16,
        wavenet_kernel_size=5,
        wavenet_dilation_rate=1,
        wavenet_dropout=0.0,
        speaking_rate=1.0,
        noise_scale=0.667,
        noise_scale_duration=0.8,
        sampling_rate=16_000,
        **kwargs,
    ):
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
        self.window_size = window_size
        self.use_bias = use_bias
        self.ffn_dim = ffn_dim
        self.layerdrop = layerdrop
        self.ffn_kernel_size = ffn_kernel_size
        self.flow_size = flow_size
        self.spectrogram_bins = spectrogram_bins
        self.hidden_act = hidden_act
        self.hidden_dropout = hidden_dropout
        self.attention_dropout = attention_dropout
        self.activation_dropout = activation_dropout
        self.initializer_range = initializer_range
        self.layer_norm_eps = layer_norm_eps
        self.use_stochastic_duration_prediction = use_stochastic_duration_prediction
        self.num_speakers = num_speakers
        self.speaker_embedding_size = speaker_embedding_size
        self.upsample_initial_channel = upsample_initial_channel
        self.upsample_rates = upsample_rates
        self.upsample_kernel_sizes = upsample_kernel_sizes
        self.resblock_kernel_sizes = resblock_kernel_sizes
        self.resblock_dilation_sizes = resblock_dilation_sizes
        self.leaky_relu_slope = leaky_relu_slope
        self.depth_separable_channels = depth_separable_channels
        self.depth_separable_num_layers = depth_separable_num_layers
        self.duration_predictor_flow_bins = duration_predictor_flow_bins
        self.duration_predictor_tail_bound = duration_predictor_tail_bound
        self.duration_predictor_kernel_size = duration_predictor_kernel_size
        self.duration_predictor_dropout = duration_predictor_dropout
        self.duration_predictor_num_flows = duration_predictor_num_flows
        self.duration_predictor_filter_channels = duration_predictor_filter_channels
        self.prior_encoder_num_flows = prior_encoder_num_flows
        self.prior_encoder_num_wavenet_layers = prior_encoder_num_wavenet_layers
        self.posterior_encoder_num_wavenet_layers = posterior_encoder_num_wavenet_layers
        self.wavenet_kernel_size = wavenet_kernel_size
        self.wavenet_dilation_rate = wavenet_dilation_rate
        self.wavenet_dropout = wavenet_dropout
        self.speaking_rate = speaking_rate
        self.noise_scale = noise_scale
        self.noise_scale_duration = noise_scale_duration
        self.sampling_rate = sampling_rate

        if len(upsample_kernel_sizes) != len(upsample_rates):
            raise ValueError(
                f"The length of `upsample_kernel_sizes` ({len(upsample_kernel_sizes)}) must match the length of "
                f"`upsample_rates` ({len(upsample_rates)})"
            )

        super().__init__(**kwargs)