<jupyter_start><jupyter_text>Force Alignment using CTCForced alignment is a technique to take an orthographic transcription of an audio file and generate a time-aligned version. In this example, I am going to use Malay and Singlish CTC models. This tutorial is available as an IPython notebook at [malaya-speech/example/force-alignment-ctc](https://github.com/huseinzol05/malaya-speech/tree/master/example/force-alignment-ctc).     This module is not language independent, so it not save to use on different languages. Pretrained models trained on hyperlocal languages.<jupyter_code>import malaya_speech
import numpy as np
from malaya_speech import Pipeline
import IPython.display as ipd
import matplotlib.pyplot as plt
from malaya_speech.utils.aligner import plot_alignments<jupyter_output><empty_output><jupyter_text>List available CTC models<jupyter_code>malaya_speech.force_alignment.available_ctc()<jupyter_output><empty_output><jupyter_text>Load CTC Aligner model```pythondef deep_ctc(    model: str = 'hubert-conformer', quantized: bool = False, **kwargs):    """    Load Encoder-CTC ASR model.    Parameters    ----------    model : str, optional (default='hubert-conformer')        Model architecture supported. Allowed values:        * ``'hubert-conformer-tiny'`` - Finetuned HuBERT Conformer TINY.        * ``'hubert-conformer'`` - Finetuned HuBERT Conformer.        * ``'hubert-conformer-large'`` - Finetuned HuBERT Conformer LARGE.        * ``'hubert-conformer-large-3mixed'`` - Finetuned HuBERT Conformer LARGE for (Malay + Singlish + Mandarin) languages.        * ``'best-rq-conformer-tiny'`` - Finetuned BEST-RQ Conformer TINY.        * ``'best-rq-conformer'`` - Finetuned BEST-RQ Conformer.        * ``'best-rq-conformer-large'`` - Finetuned BEST-RQ Conformer LARGE.    quantized : bool, optional (default=False)        if True, will load 8-bit quantized model.        Quantized model not necessary faster, totally depends on the machine.    Returns    -------    result : malaya_speech.model.wav2vec.Wav2Vec2_Aligner class    """```<jupyter_code>model = malaya_speech.force_alignment.deep_ctc(model = 'hubert-conformer')
mixed_model = malaya_speech.force_alignment.deep_ctc(model = 'hubert-conformer-large-3mixed')<jupyter_output><empty_output><jupyter_text>Load sample  Malay samples<jupyter_code>malay1, sr = malaya_speech.load('speech/example-speaker/shafiqah-idayu.wav')
malay2, sr = malaya_speech.load('speech/example-speaker/haqkiem.wav')
texts = ['nama saya <NAME>',
        'sebagai pembangkang yang matang dan sejahtera pas akan menghadapi pilihan raya umum dan tidak menumbang kerajaan dari pintu belakang']
ipd.Audio(malay2, rate = sr)<jupyter_output><empty_output><jupyter_text>Singlish samples<jupyter_code>import json
import os
from glob import glob

with open('speech/imda/output.json') as fopen:
    data = json.load(fopen)
    
data
wavs = glob('speech/imda/*.WAV')
wavs
y, sr = malaya_speech.load(wavs[0])
ipd.Audio(y, rate = sr)<jupyter_output><empty_output><jupyter_text>Predict```pythondef predict(self, input, transcription: str, sample_rate: int = 16000):    """    Transcribe input, will return a string.    Parameters    ----------    input: np.array        np.array or malaya_speech.model.frame.Frame.    transcription: str        transcription of input audio.    sample_rate: int, optional (default=16000)        sample rate for `input`.    Returns    -------    result: Dict[chars_alignment, words_alignment, alignment]    """```  Predict MalayOur original text is: 'sebagai pembangkang yang matang dan sejahtera pas akan menghadapi pilihan raya umum dan tidak menumbang kerajaan dari pintu belakang'<jupyter_code>results = model.predict(malay2, texts[1])
results.keys()
results['words_alignment']<jupyter_output><empty_output><jupyter_text>Plot alignment```pythondef plot_alignments(    alignment,    subs_alignment,    words_alignment,    waveform,    separator: str = ' ',    sample_rate: int = 16000,    figsize: tuple = (16, 9),    plot_score_char: bool = False,    plot_score_word: bool = True,):    """    plot alignment.    Parameters    ----------    alignment: np.array        usually `alignment` output.    subs_alignment: list        usually `chars_alignment` or `subwords_alignment` output.    words_alignment: list        usually `words_alignment` output.    waveform: np.array        input audio.    separator: str, optional (default=' ')        separator between words, only useful if `subs_alignment` is character based.    sample_rate: int, optional (default=16000)    figsize: tuple, optional (default=(16, 9))        figure size for matplotlib `figsize`.    plot_score_char: bool, optional (default=False)        plot score on top of character plots.    plot_score_word: bool, optional (default=True)        plot score on top of word plots.    """```<jupyter_code>plot_alignments(alignment = results['alignment'],
                subs_alignment = results['chars_alignment'],
                words_alignment = results['words_alignment'],
                waveform = malay2,
                separator = ' ',
                sample_rate = 16000,
                figsize = (16, 9))<jupyter_output><empty_output><jupyter_text>Predict SinglishOur original text is: '<NAME> yeo yong boon and tay kheng soon'<jupyter_code>results = mixed_model.predict(y, data[os.path.split(wavs[0])[1]])
results.keys()
results['words_alignment']
results['chars_alignment']
plot_alignments(alignment = results['alignment'],
                subs_alignment = results['chars_alignment'],
                words_alignment = results['words_alignment'],
                waveform = y,
                separator = ' ',
                sample_rate = 16000,
                figsize = (16, 9))<jupyter_output><empty_output><jupyter_text>What if we give wrong transcription?<jupyter_code>results = mixed_model.predict(y, 'husein sangat comel')
plot_alignments(alignment = results['alignment'],
                subs_alignment = results['chars_alignment'],
                words_alignment = results['words_alignment'],
                waveform = y,
                separator = ' ',
                sample_rate = 16000,
                figsize = (16, 9))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.109375, 'fraction_numerical': 0.010792525773195876, 'mean_word_length': 3.7073540561031084, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '139670', 'license': 'MIT', 'n_tokens_mistral': 2013, 'n_tokens_neox': 1859, 'n_words': 471, 'path': 'example/force-alignment-ctc/force-alignment-ctc.ipynb', 'repo_name': 'techthiyanes/malaya-speech', 'size': '6237'}