<jupyter_start><jupyter_text>GPT2Malaya provided Pretrained GTP2 model, specific to Malay, we called it GTP2-Bahasa. This interface not able us to use it to do custom training.GPT2-Bahasa was pretrained on ~0.9 billion words, and below is the list of dataset we trained,1. [dumping wikipedia (222MB)](https://github.com/huseinzol05/Malaya-Datasetwikipedia-1).2. [local news (257MB)](https://github.com/huseinzol05/Malaya-Datasetpublic-news).3. [local parliament text (45MB)](https://github.com/huseinzol05/Malaya-Datasetparliament).4. [IIUM Confession (74MB)](https://github.com/huseinzol05/Malaya-Datasetiium-confession).5. [Wattpad (74MB)](https://github.com/huseinzol05/Malaya-Datasetwattpad).6. [Academia PDF (42MB)](https://github.com/huseinzol05/Malaya-Datasetacademia-pdf).7. [Common-Crawl (3GB)](https://github.com/huseinzol05/malaya-datasetcommon-crawl). If you want to download pretrained model for GPT2-Bahasa and use it for custom transfer-learning, you can download it here, https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/gpt2, some notebooks to help you get started.**Here we hope these models are not use to finetune for spreading fake news**. Or you can simply use [Transformers](https://huggingface.co/models?filter=malay&search=gpt2) to try GPT2-Bahasa models from Malaya, simply check available models from here, https://huggingface.co/models?filter=malay&search=gpt2<jupyter_code>from IPython.core.display import Image, display

display(Image('gpt2.png', width=500))<jupyter_output><empty_output><jupyter_text>load modelGPT2-Bahasa only available `117M` and `345M` models.1. `117M` size around 442MB.2. `345M` is around 1.2GB.<jupyter_code>model = malaya.generator.gpt2(model = '117M')
string = 'ceritanya sebegini, aku bangun pagi baca surat khabar berita harian, tetiba aku nampak cerita seram, '
print(model.generate(string))
model = malaya.generator.gpt2(model = '345M')
string = 'ceritanya sebegini, aku bangun pagi baca surat khabar berita harian, tetiba aku nampak cerita seram, '
print(model.generate(string))<jupyter_output>ceritanya sebegini, aku bangun pagi baca surat khabar berita harian, tetiba aku nampak cerita seram, omputeh-uteh cerita lama-lama, seram tak boleh bayang
Sebelum kejadian, dalam 2 jam aku buat panggilan polis , lepas tu kira la sendiri nak ke lokasi.
Tengok cerita lama..
Sekarang ni, apa yang aku lalui, kita yang jaga diri, kita yang jaga kesihatan dan juga kita yang jaga minda dalam hidup.
Maka, inilah jalan penyelesaian terbaiknya.
Jangan lupakan manusia
Orang yang paling ditakuti untuk berjaya dalam hidup, tidak akan jumpa yang tersayang!
Jangan rosakkan masa depannya, ingatlah apa yang kita nak buat, walaupun pahit untuk ditelan.
Jangan lupakan orang lain - masa depan mereka.
Jangan lupakan orang - masa itulah kita yang lebih dicintai.
Jangan lupakan orang - orang yang kita sayang, mereka bukan orang yang tersayang!
Jangan lupakan orang - orang yang kita cinta, mereka cinta pada kita.
Jangan lupakan diri - diri kita - yang kita punya, yang kita tinggal adalah masa lalu kita.
Janga[...]<jupyter_text>Wordvector augmentationLet say you have a very limited labelled corpus, and you want to add more, but labelling is very costly.So, text augmentation! You can use wordvector to replace words with similar semantics!```pythondef wordvector_augmentation(    string,    wordvector,    threshold = 0.5,    top_n = 5,    soft = False,    cleaning_function = None,):    """    augmenting a string using wordvector.    Parameters    ----------    string: str    wordvector: object        wordvector interface object.    threshold: float, optional (default=0.5)        random selection for a word.    soft: bool, optional (default=False)        if True, a word not in the dictionary will be replaced with nearest jarowrinkler ratio.        if False, it will throw an exception if a word not in the dictionary.    top_n: int, (default=5)        number of nearest neighbors returned.    cleaning_function: function, (default=None)        function to clean text.    Returns    -------    result: list    """```<jupyter_code>string = 'saya suka makan ayam dan ikan'
vocab_wiki, embedded_wiki = malaya.wordvector.load_wiki()
word_vector_wiki = malaya.wordvector.load(embedded_wiki, vocab_wiki)
augmented = malaya.generator.wordvector_augmentation(string,
                                  word_vector_wiki,
                                  soft=True)
augmented
text = 'Perdana Menteri berkata, beliau perlu memperoleh maklumat terperinci berhubung isu berkenaan sebelum kerajaan dapat mengambil sebarang tindakan lanjut. Bagaimanapun, beliau yakin masalah itu dapat diselesaikan dan pentadbiran kerajaan boleh berfungsi dengan baik.'
augmented = malaya.generator.wordvector_augmentation(text,
                                  word_vector_wiki,
                                  soft=True)
augmented<jupyter_output><empty_output><jupyter_text>Transformer augmentationProblem with wordvector, it just replaced a word for near synonym without understood the whole sentence context, so, Transformer comes to the rescue!```pythondef transformer_augmentation(    string,    model,    threshold = 0.5,    top_p = 0.8,    top_k = 100,    temperature = 0.8,    top_n = 5,    cleaning_function = None,):    """    augmenting a string using transformer + nucleus sampling / top-k sampling.    Parameters    ----------    string: str    model: object        transformer interface object. Right now only supported BERT.    threshold: float, optional (default=0.5)        random selection for a word.    top_p: float, optional (default=0.8)        cumulative sum of probabilities to sample a word. If top_n bigger than 0, the model will use nucleus sampling, else top-k sampling.    top_k: int, optional (default=100)        k for top-k sampling.    temperature: float, optional (default=0.8)        logits * temperature.    top_n: int, (default=5)        number of nearest neighbors returned.    cleaning_function: function, (default=None)        function to clean text.    Returns    -------    result: list    """```<jupyter_code>model = malaya.transformer.load(model = 'albert')
augmented = malaya.generator.transformer_augmentation(text, model)
augmented<jupyter_output><empty_output><jupyter_text>ngramsYou can generate ngrams pretty easy using this interface,```pythondef ngrams(    sequence,    n: int,    pad_left = False,    pad_right = False,    left_pad_symbol = None,    right_pad_symbol = None,):    """    generate ngrams.    Parameters    ----------    sequence : List[str]        list of tokenize words.    n : int        ngram size    Returns    -------    ngram: list    """```<jupyter_code>string = 'saya suka makan ayam'

list(malaya.generator.ngrams(string.split(), n = 2))
list(malaya.generator.ngrams(string.split(), n = 2, pad_left = True, pad_right = True))
list(malaya.generator.ngrams(string.split(), n = 2, pad_left = True, pad_right = True,
                            left_pad_symbol = 'START'))
list(malaya.generator.ngrams(string.split(), n = 2, pad_left = True, pad_right = True,
                            left_pad_symbol = 'START', right_pad_symbol = 'END'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09415855354659249, 'fraction_numerical': 0.015855354659248956, 'mean_word_length': 3.8261744966442954, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '433166', 'license': 'MIT', 'n_tokens_mistral': 2399, 'n_tokens_neox': 2255, 'n_words': 709, 'path': 'example/generator/load-generator.ipynb', 'repo_name': 'biranchi2018/Malaya', 'size': '7190'}