<jupyter_start><jupyter_text>Voice Activity Detection This tutorial is available as an IPython notebook at [malaya-speech/example/vad](https://github.com/huseinzol05/malaya-speech/tree/master/example/vad).     This module is language independent, so it save to use on different languages. Pretrained models trained on multilanguages.     This is an application of malaya-speech Pipeline, read more about malaya-speech Pipeline at [malaya-speech/example/pipeline](https://github.com/huseinzol05/malaya-speech/tree/master/example/pipeline).      DatasetTrained on MUSAN and selected Youtube videos gathered at https://github.com/huseinzol05/malaya-speech/tree/master/data/noise<jupyter_code>import malaya_speech
import numpy as np
from malaya_speech import Pipeline
y, sr = malaya_speech.load('speech/7021-79759-0004.wav')
len(y), sr<jupyter_output><empty_output><jupyter_text>Load WebRTC```pythondef webrtc(    aggressiveness: int = 3,    sample_rate: int = 16000,    minimum_amplitude: int = 100,):    """    Load WebRTC VAD model.    Parameters    ----------    aggressiveness: int, optional (default=3)        an integer between 0 and 3.        0 is the least aggressive about filtering out non-speech, 3 is the most aggressive.    sample_rate: int, optional (default=16000)        sample rate for samples.    minimum_amplitude: int, optional (default=100)        abs(minimum_amplitude) to assume a sample is a voice activity. Else, automatically False.    Returns    -------    result : malaya_speech.model.webrtc.WEBRTC class    """``` WebRTC developed by Google, reportedly one of the best available, being fast, modern and free. Read more about pywebrtc at https://github.com/wiseman/py-webrtcvad.**For WebRTC, we need to pass integer signal**.Install `webrtcvad` by simply using PIP,```bashpip3 install webrtcvad```<jupyter_code>y_int = malaya_speech.astype.float_to_int(y)
vad = malaya_speech.vad.webrtc(minimum_amplitude = int(np.quantile(np.abs(y_int), 0.2)))<jupyter_output><empty_output><jupyter_text>List available deep model<jupyter_code>malaya_speech.vad.available_model()<jupyter_output><empty_output><jupyter_text>Load deep modelDeep model trained on different noises, augmented noises, procedural noises, overlapped noises and actual voices.```pythondef deep_model(model: str = 'marblenet-factor1', quantized: bool = False, **kwargs):    """    Load VAD model.    Parameters    ----------    model : str, optional (default='vggvox-v2')        Model architecture supported. Allowed values:        * ``'vggvox-v1'`` - finetuned VGGVox V1.        * ``'vggvox-v2'`` - finetuned VGGVox V2.        * ``'speakernet'`` - finetuned SpeakerNet.        * ``'marblenet-factor1'`` - Pretrained MarbleNet * factor 1.        * ``'marblenet-factor3'`` - Pretrained MarbleNet * factor 3.        * ``'marblenet-factor5'`` - Pretrained MarbleNet * factor 5.    quantized : bool, optional (default=False)        if True, will load 8-bit quantized model.        Quantized model not necessary faster, totally depends on the machine.    Returns    -------    result : malaya_speech.supervised.classification.load function    """```<jupyter_code>model_factor1 = malaya_speech.vad.deep_model(model = 'marblenet-factor1')
model_factor3 = malaya_speech.vad.deep_model(model = 'marblenet-factor3')<jupyter_output><empty_output><jupyter_text>How to detect Voice Activity In order to use available Malaya-Speech VAD models, we need to split our audio sample into really small chunks. For Google WebRTC, we need to split by every 10, 20 or 30 ms. For deep learning,- `vggvox-v1`, `vggvox-v2` and `speakernet`, we trained on 30 ms, 90 ms.- `marblenet`, we trained on 50 ms.To split an audio sample by giving time split, use, [malaya_speech.utils.generator.frames](https://github.com/huseinzol05/malaya-speech/blob/master/malaya_speech/utils/generator.pyL7)```pythondef frames(    audio,    frame_duration_ms: int = 30,    sample_rate: int = 16000,    append_ending_trail: bool = True,):    """    Generates audio frames from PCM audio data.    Takes the desired frame duration in milliseconds, the PCM data, and the sample rate.    Parameters    ----------    audio: np.array / list    frame_duration_ms: int, optional (default=30)    sample_rate: int, optional (default=16000)    append_ending_trail: bool, optional (default=True)        if True, will append last trail and this last trail might not same length as `frame_duration_ms`.    Returns    -------    result: List[malaya_speech.model.frame.FRAME]    """```<jupyter_code>%%time

frames_int = list(malaya_speech.utils.generator.frames(y_int, 30, sr))
frames = list(malaya_speech.utils.generator.frames(y, 30, sr))
%%time

frames_webrtc = [(frame, vad(frame)) for frame in frames_int]
%%time

frames = list(malaya_speech.utils.generator.frames(y, 50, sr))
frames_deep_factor1 = [(frame, model_factor1(frame)) for frame in frames]
%%time

frames_deep_factor3 = [(frame, model_factor3(frame)) for frame in frames]<jupyter_output>CPU times: user 848 ms, sys: 57.5 ms, total: 906 ms
Wall time: 667 ms<jupyter_text>**To speedup, you can do pipeline batching to utilize parallel processing or use GPU or both**.  Batching pipeline<jupyter_code>p = Pipeline()
pipeline = (
    p.batching(5)
    .foreach_map(model_factor1.predict)
    .flatten()
)
p.visualize()
%%time

result = p.emit(frames)
result.keys()
frames_deep_factor1_batch = [(frame, result['flatten'][no]) for no, frame in enumerate(frames)]
len(frames_deep_factor1) == len(frames_deep_factor1_batch)<jupyter_output><empty_output><jupyter_text>get probabilityDeep learning model able to calculate probability, simply use `model.predict_proba`, and make sure the input is an array or a list,<jupyter_code>%%time

probs = [(frame, model_factor1.predict_proba([frame])) for frame in frames]
probs[:5]<jupyter_output>CPU times: user 735 ms, sys: 51.1 ms, total: 786 ms
Wall time: 603 ms<jupyter_text>VAD Visualization We can visualize our audio sample to annotate which parts are active speech or not.<jupyter_code>malaya_speech.extra.visualization.visualize_vad(y, frames_webrtc, sr)
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_factor1, sr)
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_factor3, sr)
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_factor1_batch, sr)<jupyter_output><empty_output><jupyter_text>Groupby VAD To group small samples into larger samples, use `malaya_speech.utils.group.group_frames`,```pythondef group_frames(frames):    """    Group multiple frames based on label.    Parameters    ----------    frames: List[Tuple[Frame, label]]    Returns    -------    result : List[Tuple[Frame, label]]    """```<jupyter_code>grouped_vad = malaya_speech.utils.group.group_frames(frames_webrtc)
grouped_deep = malaya_speech.utils.group.group_frames(frames_deep_factor1)
len(frames), len(grouped_vad), len(grouped_deep)<jupyter_output><empty_output><jupyter_text>Harder example In the real world, not all samples recorded in clean cut studio, full with background noises, music and etc. So those are not the real voice activities. Lets take noise sound,<jupyter_code>y, sr = malaya_speech.load('speech/noise/n98.wav')
len(y), sr
%%time

y_int = malaya_speech.astype.float_to_int(y)
frames_int = list(malaya_speech.utils.generator.frames(y_int, 30, sr))
%%time

vad = malaya_speech.vad.webrtc(minimum_amplitude = int(np.quantile(np.abs(y_int), 0.2)))
frames_webrtc = [(frame, vad(frame)) for frame in frames_int]
malaya_speech.extra.visualization.visualize_vad(y, frames_webrtc, sr)
%%time

frames = list(malaya_speech.utils.generator.frames(y, 50, sr))
result = p.emit(frames)
frames_deep_batch = [(frame, result['flatten'][no]) for no, frame in enumerate(frames)]
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_batch, sr)
%%time

frames_deep_factor3 = [(frame, model_factor3(frame)) for frame in frames]
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_factor3, sr)<jupyter_output><empty_output><jupyter_text>As you can see, the deep model able to know this sample mostly noises!<jupyter_code>y, sr = malaya_speech.load('speech/podcast/2x5%20Ep%2010.wav')
y = y[: sr * 10]
len(y), sr
y_int = malaya_speech.utils.astype.float_to_int(y)

frames = list(malaya_speech.utils.generator.frames(y, 30, sr))
frames_int = list(malaya_speech.utils.generator.frames(y_int, 30, sr))
%%time

vad = malaya_speech.vad.webrtc(minimum_amplitude = int(np.quantile(np.abs(y_int), 0.2)))
frames_webrtc = [(frame, vad(frame)) for frame in frames_int]
malaya_speech.extra.visualization.visualize_vad(y, frames_webrtc, sr)
%%time

frames = list(malaya_speech.utils.generator.frames(y, 50, sr))
result = p.emit(frames)
frames_deep_batch = [(frame, result['flatten'][no]) for no, frame in enumerate(frames)]
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_batch, sr)<jupyter_output><empty_output><jupyter_text>Groupby based on time thresholdGroup multiple frames based on label and threshold to stop.```pythondef group_frames_threshold(frames, threshold_to_stop: float = 0.3):    """    Group multiple frames based on label and threshold to stop.    Parameters    ----------    frames: List[Tuple[Frame, label]]    threshold_to_stop: float, optional (default = 0.3)        If `threshold_to_stop` is 0.3, means that, length same label samples must at least 0.3 second.    Returns    -------    result : List[Tuple[Frame, label]]    """```<jupyter_code>%%time

result = p.emit(frames)
frames_deep_batch = [(frame, result['flatten'][no]) for no, frame in enumerate(frames)]
grouped_deep = malaya_speech.utils.group.group_frames(frames_deep_batch)
grouped_deep_threshold = malaya_speech.utils.group.group_frames_threshold(grouped_deep, 0.05)
malaya_speech.extra.visualization.visualize_vad(y, grouped_deep_threshold, sr)
%%time

frames_deep_factor3 = [(frame, model_factor3(frame)) for frame in frames]
grouped_deep = malaya_speech.utils.group.group_frames(frames_deep_batch)
grouped_deep_threshold = malaya_speech.utils.group.group_frames_threshold(grouped_deep, 0.05)
malaya_speech.extra.visualization.visualize_vad(y, grouped_deep_threshold, sr)<jupyter_output><empty_output><jupyter_text>Load Quantized deep modelTo load 8-bit quantized model, simply pass `quantized = True`, default is `False`.We can expect slightly accuracy drop from quantized model, and not necessary faster than normal 32-bit float model, totally depends on machine.<jupyter_code>quantized_model = malaya_speech.vad.deep_model(model = 'marblenet-factor1', quantized = True)
%%time

frames_deep_factor1 = [(frame, model_factor1(frame)) for frame in frames]
%%time

frames_deep_factor1_quantized = [(frame, quantized_model(frame)) for frame in frames]
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_factor1, sr)
malaya_speech.extra.visualization.visualize_vad(y, frames_deep_factor1_quantized, sr)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11547640903249495, 'fraction_numerical': 0.017532586744997247, 'mean_word_length': 5.401292596944771, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '227366', 'license': 'MIT', 'n_tokens_mistral': 3814, 'n_tokens_neox': 3507, 'n_words': 899, 'path': 'example/vad/load-vad.ipynb', 'repo_name': 'ishine/malaya-speech', 'size': '10894'}