<jupyter_start><jupyter_text>Kesalahan Tatabahasa This tutorial is available as an IPython notebook at [Malaya/example/tatabahasa](https://github.com/huseinzol05/Malaya/tree/master/example/tatabahasa).     This module only trained on standard language structure, so it is not save to use it for local language structure.<jupyter_code>%%time

import malaya
from pprint import pprint<jupyter_output>CPU times: user 4.87 s, sys: 865 ms, total: 5.73 s
Wall time: 6.34 s<jupyter_text>ModelCommon Seq2Seq model, P(yt | X, yt-1), one step decoder will generate `yt`, and this required encoder output and output from last step decoder, `yt-1`.So we improve the model to general tags also, P(yt, zt | X, yt-1, zt-1), one step decoder will generate `yt` and `zt`, and this required encoder output and outputs from last step decoder, `yt-1` and `zt-1`.We named this model as `TransformerTag`. There is no paper produced for this model, feel free to write a paper about it, check out our implementation at https://github.com/huseinzol05/malaya/tree/master/session/tatabahasa  List available Transformer Tag models<jupyter_code>malaya.tatabahasa.available_transformer()<jupyter_output>INFO:root:tested on 10k kesalahan tatabahasa texts.<jupyter_text>Supported kesalahan tatabahasaFor full description, check out https://tatabahasabm.tripod.com/tata/salahtata.htm<jupyter_code>malaya.tatabahasa.describe()<jupyter_output><empty_output><jupyter_text>Right now we only able to predict up to 15 different kesalahan tatabahasa, hopefully in the future we can scale this up.  Load Transformer Tag model<jupyter_code>model = malaya.tatabahasa.transformer(model = 'base')
quantized_model = malaya.tatabahasa.transformer(model = 'base', quantized = True)<jupyter_output>WARNING:root:Load quantized model will cause accuracy drop.<jupyter_text>Predict using greedy decoder```pythondef greedy_decoder(self, strings: List[str]):    """    Fix kesalahan tatatabahasa.    Parameters    ----------    strings : List[str]    Returns    -------    result: List[str]    """```For TransformerTag model, right now only supported `greedy_decoder` method. Randomly picked string in bahasa melayu wikipedia.<jupyter_code># https://ms.wikipedia.org/wiki/Bola_sepak
string = 'Pada amnya, hanya penjaga gol sahaja yang dibenarkan menyentuh bola dengan tangan di dalam kawasan golnya'
model.greedy_decoder([string])<jupyter_output><empty_output><jupyter_text>Now assumed we have kesalahan frasa nama, from `penjaga gol` become `gol penjaga`.<jupyter_code># https://ms.wikipedia.org/wiki/Bola_sepak
string = 'Pada amnya, hanya gol penjaga sahaja yang dibenarkan menyentuh bola dengan tangan di dalam kawasan golnya'
model.greedy_decoder([string])
string = 'Sani mendapat markah yang tertinggi sekali.'
string1 = 'Hassan ialah peserta yang termuda sekali dalam pertandingan itu.'
model.greedy_decoder([string, string1])
string = 'Dia kata kepada saya.'
model.greedy_decoder([string])
import pickle

with open('tests/dataset-tatabahasa.pkl', 'rb') as fopen:
    test_set = pickle.load(fopen)
    
len(test_set)
def get_xy(row):
    x, y, tag = [], [], []

    for i in range(len(row[0])):
        t = [row[0][i][0]]
        y.extend(t)
        t = [row[1][i][0]]
        x.extend(t)
        tag.extend([row[1][i][1]] * len(t))

    return ' '.join(x), ' '.join(y), tag
x, y, t = get_xy(test_set[0])
x, y, t
model.greedy_decoder([x])
quantized_model.greedy_decoder([x])
x, y, t = get_xy(test_set[-1])
x, y, t
model.greedy_decoder([x])
x, y, t = get_xy(test_set[-2])
x, y, t
model.greedy_decoder([x])
x, y, t = get_xy(test_set[-3])
x, y, t
model.greedy_decoder([x])
quantized_model.greedy_decoder([x])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11882799782962561, 'fraction_numerical': 0.010851871947911014, 'mean_word_length': 5.13477537437604, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '402985', 'license': 'MIT', 'n_tokens_mistral': 1360, 'n_tokens_neox': 1280, 'n_words': 363, 'path': 'docs/load-tatabahasa.ipynb', 'repo_name': 'ahmed3991/malaya', 'size': '3686'}