<jupyter_start><jupyter_text>Source Code<jupyter_code>%reload_ext autoreload
%autoreload 2 
%matplotlib inline
! pip install sentencepiece
from fastai.text import *
import pandas as pd
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Preparing the data  Language model Note that language models can use a lot of GPU, so you may need to decrease batchsize here.<jupyter_code>bs=128<jupyter_output><empty_output><jupyter_text>Now let's grab the full dataset for what follows.<jupyter_code>path = Path("/tf/data/datasets/raw/raw_java/data00m_god-r")
path.ls()
(path/'train').ls()<jupyter_output><empty_output><jupyter_text>The reviews are in a training and test set following an imagenet structure. The only difference is that there is an `unsup` folder on top of `train` and `test` that contains the unlabelled data.We're not going to train a model that classifies the reviews from scratch. Like in computer vision, we'll use a model pretrained on a bigger dataset (a cleaned subset of wikipedia called [wikitext-103](https://einstein.ai/research/blog/the-wikitext-long-term-dependency-language-modeling-dataset)). That model has been trained to guess what the next word is, its input being all the previous words. It has a recurrent structure and a hidden state that is updated each time it sees a new word. This hidden state thus contains information about the sentence up to that point.We are going to use that 'knowledge' of the English language to build our classifier, but first, like for computer vision, we need to fine-tune the pretrained model to our particular dataset. Because the English of the reviews left by people on IMDB isn't the same as the English of wikipedia, we'll need to adjust the parameters of our model by a little bit. Plus there might be some words that would be extremely common in the reviews dataset but would be barely present in wikipedia, and therefore might not be part of the vocabulary the model was trained on. This is where the unlabelled data is going to be useful to us, as we can use it to fine-tune our model. Let's create our data object with the data block API (next line takes a few minutes).<jupyter_code>processor = [
    OpenFileProcessor(),
    SPProcessor(lang="en", pre_rules=[], post_rules=[],
                max_sentence_len=30_720,
                sp_model = "/tf/main/nbs/poc/model.model",
                sp_vocab = "/tf/main/nbs/poc/model.vocab"
               )
]
data_lm = (
    TextList.from_folder(
        path, extensions={".java"}, processor = processor
    )
    #Inputs: all the text files in path
    .filter_by_folder(include=['sm_train', 'sm_valid'])
    #We may have other temp folders that contain text files
    #so we only keep what's in train and test
    .split_by_folder(train='sm_train', valid='sm_valid')
    .label_for_lm() 
    #We want to do a language model so we label accordingly
    .databunch(bs=bs)
)
data_lm.save('data_lm.pkl')
# data_lm = (TextList.from_folder(path, extensions={".java"},
#                                 processor = [OpenFileProcessor(),
#                                              SPProcessor(lang="en", pre_rules=[], post_rules=[],
#                                                          max_sentence_len=40_960,
#                                                          sp_model = "/tf/src/prototypes/model.model",
#                                                          sp_vocab = "/tf/src/prototypes/model.vocab"
#                                                         )])
#            #Inputs: all the text files in path
#             .filter_by_folder(include=['sm_train', 'sm_valid']) 
# #            #We may have other temp folders that contain text files so we only keep what's in train and test
#             .split_by_folder(train='sm_train', valid='sm_valid')
#            #We randomly split and keep 10% (10,000 reviews) for validation
#             .label_for_lm()           
#            #We want to do a language model so we label accordingly
#             .databunch(bs=bs))
# data_lm.save('data_lm.pkl')<jupyter_output><empty_output><jupyter_text>We have to use a special kind of `TextDataBunch` for the language model, that ignores the labels (that's why we put 0 everywhere), will shuffle the texts at each epoch before concatenating them all together (only for training, we don't shuffle for the validation set) and will send batches that read that text in order with targets that are the next word in the sentence.The line before being a bit long, we want to load quickly the final ids by using the following cell.<jupyter_code>data_lm = load_data(path, 'data_lm.pkl', bs=bs)
len(data_lm.vocab.itos)
data_lm.show_batch()<jupyter_output><empty_output><jupyter_text>We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file).<jupyter_code>learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3, pretrained=False)
learn = language_model_learner(data_lm, TransformerXL, drop_mult=0.3)
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(1, 1.5e-2, moms=(0.8,0.75))
len(data_lm.train_ds)
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(1, 5e-2, moms=(0.8,0.75), pct_start = 0.02)
learn.recorder.plot_losses()
learn.save('fit_head')
learn.load('fit_head');<jupyter_output><empty_output><jupyter_text>To complete the fine-tuning, we can then unfeeze and launch a new training.<jupyter_code>learn.unfreeze()
learn.fit_one_cycle(10, 5e-4, moms=(0.8,0.7), pct_start = 0.02)
learn.unfreeze()
learn.fit_one_cycle(10, 5e-4, moms=(0.8,0.7), pct_start = 0.02)
learn.save('fine_tuned')
! curl -X POST -H 'Content-type: application/json' --data '{"text":"from: semeru tower 2\nstatus: finished training TransformerXL"}' https://hooks.slack.com/services/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
learn.recorder.plot_metrics()<jupyter_output><empty_output><jupyter_text>How good is our model? Well let's try to see what it predicts after a few given words.<jupyter_code>learn.load('fine_tuned');
TEXT = "public String get"
N_WORDS = 40
N_SENTENCES = 2
print("\n".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))<jupyter_output><empty_output><jupyter_text>We have to save not only the model, but also its encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word.<jupyter_code>learn.save_encoder('fine_tuned_enc')<jupyter_output><empty_output><jupyter_text>Classifier Now, we'll create a new data object that only grabs the labelled data and keeps those labels. Again, this line takes a bit of time.<jupyter_code>path = untar_data(URLs.IMDB)
data_clas = (TextList.from_folder(
    path, extensions={".java"}, vocab=data_lm.vocab,
    processor = [OpenFileProcessor(), SPProcessor(lang="en",
                                                  sp_model = "/tf/src/prototypes/model.model",
                                                  sp_vocab = "/tf/src/prototypes/model.vocab"
                                                 )]
)
             .filter_by_folder(include=['sm_train', 'sm_valid']) 
             #grab all the text files in path
             .split_by_folder(train='sm_train', valid='sm_valid')
             #split by train and valid folder (that only keeps 'train' and 'test' so no need to filter)
             .label_from_folder(classes=['before', 'after'])
             #label them all with their folders
             .databunch(bs=bs))

data_clas.save('data_clas.pkl')
path = Path("/tf/data/datasets/raw/security_c++")
path.ls()
bs = 64
ItemList??
data_clas = TextList.from_csv(path, 'security-training.csv',
                                       text_cols = 'code', label_cols = 'label', tokenizer = SPProcessor(lang="en",
                                                  sp_model = "/tf/src/prototypes/model.model",
                                                  sp_vocab = "/tf/src/prototypes/model.vocab"
                                                 ))
TextList.from_csv?
data_clas = (
    TextList.from_csv(
        path, 'security-training.csv',
        cols=['code', 'label'],
        processor=SPProcessor(
            lang="en",
            sp_model = "/tf/main/nbs/poc/model.model",
            sp_vocab = "/tf/main/nbs/poc/model.vocab"
        )
    )
    .split_by_rand_pct()
    .label_from_df(cols=['label'])
    .databunch(bs=bs)
)
data_clas.vocab = data_lm.vocab
data_clas.show_batch()
data_clas.save('data_clas.pkl')
data_clas = load_data(path, 'data_clas.pkl', bs=bs)
data_clas.show_batch()
len(data_clas.train_ds)
path = Path("/tf/data/datasets/raw/raw_java/data00m_god-r")<jupyter_output><empty_output><jupyter_text>We can then create a model to classify those reviews and load the encoder we saved before.<jupyter_code>learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, pretrained=False)
learn.load_encoder(path/'models/fine_tuned_enc')
learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(1, 2e-1, moms=(0.8,0.7))
learn.recorder.plot_metrics()
learn.recorder.plot()
learn.save('first')
learn.load('first');
learn.freeze_to(-2)

learn.lr_find()
learn.recorder.plot()
learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2), moms=(0.8,0.7))
learn.save('second')
learn.load('second');
learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))
learn.save('third')
learn.load('third');
learn.unfreeze()
learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3), moms=(0.8,0.7))
learn.predict("I really loved that movie, it was awesome!")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1096938775510204, 'fraction_numerical': 0.013979591836734693, 'mean_word_length': 3.278044522042776, 'pattern_counts': {'":': 1, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '603057', 'license': 'MIT', 'n_tokens_mistral': 3167, 'n_tokens_neox': 3048, 'n_words': 877, 'path': 'main/nbs/poc/fastai/transformer_xl_src.ipynb', 'repo_name': 'jason424217/Artificial-Code-Gen', 'size': '9800'}