<jupyter_start><jupyter_text>FastPitch: Voice Modification with Custom Transformations  Model overview The [FastPitch](https://arxiv.org/abs/2006.06873) model is based on the [FastSpeech](https://arxiv.org/abs/1905.09263) model. Similarly to [FastSpeech2](https://arxiv.org/abs/2006.04558), which has been developed concurrently, it learns to predict the pitch contour and conditions the generation on such contour.The simple mechanism of predicting the pitch on grapheme-level (rather than frame-level, as FastSpeech2 does) allows to easily alter the pitch during synthesis. FastPitch can thus change the perceived emotional state of the speaker, or slightly emphasise certain lexical units.  Requirements Run the notebook inside the container. By default the container forwards port `8888`.```bash scripts/docker/interactive.sh inside the containercd notebooksjupyter notebook --ip='*' --port=8888```Please refer the Requirement section in `README.md` for more details and running outside the container.<jupyter_code>import os
assert os.getcwd().split('/')[-1] == 'notebooks'<jupyter_output><empty_output><jupyter_text>Generate audio samples Training a FastPitch model from scrath takes 3 to 27 hours depending on the type and number of GPUs, performance numbers can be found in Section "Training performance results" in `README.md`. Therefore, to save the time of running this notebook, we recommend to download the pretrained FastPitch checkpoints on NGC for inference.You can find FP32 checkpoint at [NGC](https://ngc.nvidia.com/catalog/models/nvidia:fastpitch_pyt_fp32_ckpt_v1/files) , and AMP (Automatic Mixed Precision) checkpoint at [NGC](https://ngc.nvidia.com/catalog/models/nvidia:fastpitch_pyt_amp_ckpt_v1/files).To synthesize audio, you will need a WaveGlow model, which generates waveforms based on mel-spectrograms generated by FastPitch.You can download a pre-trained WaveGlow AMP model at [NGC](https://ngc.nvidia.com/catalog/models/nvidia:waveglow256pyt_fp16).<jupyter_code>! mkdir -p output
! MODEL_DIR='../pretrained_models' ../scripts/download_fastpitch.sh
! MODEL_DIR='../pretrained_models' ../scripts/download_waveglow.sh<jupyter_output><empty_output><jupyter_text>You can perform inference using the respective checkpoints that are passed as `--fastpitch` and `--waveglow` arguments. Next, you will use FastPitch model to generate audio samples for input text, including the basic version and the variations i npace, fade out, and pitch transforms, etc.<jupyter_code>import IPython

# store paths in aux variables
fastp = '../pretrained_models/fastpitch/nvidia_fastpitch_200518.pt'
waveg = '../pretrained_models/waveglow/waveglow_1076430_14000_amp.pt'
flags = f'--cuda --fastpitch {fastp} --waveglow {waveg} --wn-channels 256'<jupyter_output><empty_output><jupyter_text>1. Basic speech synthesis You need to create an input file with some text, or just input the text in the below cell:<jupyter_code>%%writefile text.txt
This is a sample sentence you can synthesize using this wonderful model!
# Basic synthesis
!python ../inference.py {flags} -i text.txt -o output/original --pace 0.75 > /dev/null

IPython.display.Audio("output/original/audio_0.wav")<jupyter_output><empty_output><jupyter_text>2. 'Low - high, odd - even' speech transformation<jupyter_code>%%writefile ../pitch_transform.py
import torch
import numpy as np

def pitch_transform_custom(pitch, pitch_lens):
    """Apply a custom pitch transformation to predicted pitch values.

    Odd - even sentence transformation.
    This sample modification decreses the pitch for even words
    and increses the pitch for odd words in the sentence.

    PARAMS
    ------
    pitch: torch.Tensor (bs, max_len)
        Predicted pitch values for each lexical unit, padded to max_len (in Hz).
    pitch_lens: torch.Tensor (bs, max_len)
        Number of lexical units in each utterance.

    RETURNS
    -------
    pitch: torch.Tensor
        Modified pitch (in Hz).
    """
    
    sentence = 'This is a sample sentence you can synthesize using this wonderful model!'
    sep_sums = np.cumsum(np.asarray([c == ' ' for c in sentence]))
    transform = np.where(sep_sums % 2 == 0, 0.6, 1.2)
    transform = torch.tensor(transform, dtype=torch.float32, device=pitch.device)

    return pitch * transform
# Synthesis with pace 0.75 and odd - even sentence transformation
!python ../inference.py {flags} -i text.txt -o output/custom --pitch-transform-custom --pace 0.75 > /dev/null

IPython.display.Audio("output/custom/audio_0.wav")<jupyter_output><empty_output><jupyter_text>3. 'Really' speech transformation<jupyter_code>%%writefile text.txt
Really? It sounds nothing like that.
# Basic synthesis
!python ../inference.py {flags} -i text.txt -o output/original_really > /dev/null

IPython.display.Audio("output/original_really/audio_0.wav")
%%writefile ../pitch_transform.py
import torch

def pitch_transform_custom(pitch, pitch_lens):
    
    sentence = "Really? I wouldn't be so sure."
    
    # Put emphasis on `lly?` in 'Really?'
    for i in range(len('Rea'), len('Really?')):
        pitch[0][i] = 280 + (i - 3) * 20

    return pitch
# Synthesis with 'really' question transformation and pace 0.9
!python ../inference.py {flags} -i text.txt -o output/custom_really_question \
    --pitch-transform-custom --pace 0.9 > /dev/null

IPython.display.Audio("output/custom_really_question/audio_0.wav")
%%writefile ../pitch_transform.py
import torch

def pitch_transform_custom(pitch, pitch_lens):
    
    sentence = 'Really? It does not sound like that!'
    
    # Fixed 'really' word adjustment
    for i in range(len('Really?')):
        pitch[0][i] = 215 - i * 10

    return pitch * torch.tensor(0.8)
# Synthesis with 'really' sceptical transformation and pace 0.9
!python ../inference.py {flags} -i text.txt -o output/custom_really_sceptical \
    --pitch-transform-custom --pace 0.9 > /dev/null

IPython.display.Audio("output/custom_really_sceptical/audio_0.wav")<jupyter_output><empty_output><jupyter_text>4. 'Right' speech transformation<jupyter_code>%%writefile text.txt
It's obvious... right?
# Basic synthesis
!python ../inference.py {flags} -i text.txt -o output/original_right > /dev/null

IPython.display.Audio("output/original_right/audio_0.wav")
%%writefile ../pitch_transform.py
import torch

def pitch_transform_custom(pitch, pitch_lens):
            
    pitch[0][-6] = 180  # R
    pitch[0][-5] = 260  # i
    pitch[0][-4] = 360  # g
    pitch[0][-3] = 360  # h
    pitch[0][-2] = 380  # t
    pitch[0][-1] = 400  # ?

    return pitch * torch.tensor(0.9)
# Synthesis with 'right' question transformation
!python ../inference.py {flags} -i text.txt -o output/custom_right_question \
    --pitch-transform-custom > /dev/null

IPython.display.Audio("output/custom_right_question/audio_0.wav")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11107854630715124, 'fraction_numerical': 0.022860492379835874, 'mean_word_length': 4.303030303030303, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 36, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '170850', 'license': 'MIT', 'n_tokens_mistral': 2308, 'n_tokens_neox': 2084, 'n_words': 647, 'path': 'SpeechSynthesis/FastPitch/notebooks/FastPitch_voice_modification_custom.ipynb', 'repo_name': 'eba472/mongolian_tts', 'size': '6824'}