<jupyter_start><jupyter_text>Assignment 3, Part 1: BERT Loss Model Welcome to the part 1 of testing the models for this week's assignment. We will perform decoding using the BERT Loss model. In this notebook we'll use an input, mask (hide) random word(s) in it and see how well we get the "Target" answer(s).  IMPORTANT- As you cannot save the changes you make to this colab, you have to make a copy of this notebook in your own drive and run that. You can do so by going to `File -> Save a copy in Drive`. Close this colab and open the copy which you have made in your own drive.- Go to this [google drive folder](https://drive.google.com/drive/folders/1rOZsbEzcpMRVvgrRULRh1JPFpkIG_JOz?usp=sharing) named `NLP C4 W3 Colabs & Data`. In the folder, next to its name use the drop down menu to select `"Add shortcut to Drive" -> "My Drive" and then press ADD SHORTCUT`. This should add a shortcut to the folder `NLP C4 W3 Colabs & Data` within your own google drive. Please make sure this happens, as you'll be reading the data for this notebook from this folder.- Make sure your runtime is GPU (_not_ CPU or TPU). And if it is an option, make sure you are using _Python 3_. You can select these settings by going to `Runtime -> Change runtime type -> Select the above mentioned settings and then press SAVE` **Note: Restarting the runtime maybe required**.Colab will tell you if the restarting is necessary -- you can do this from the:Runtime > Restart Runtimeoption in the dropdown.  Outline- [Part 0: Downloading and loading dependencies](0)- [Part 1: Mounting your drive for data accessibility](1)- [Part 2: Getting things ready](2)- [Part 3: Part 3: BERT Loss](3)    - [3.1 Decoding](3.1)  Part 0: Downloading and loading dependenciesUncomment the code cell below and run it to download some dependencies that you will need. You need to download them once every time you open the colab. You can ignore the `kfac` error.<jupyter_code>!pip -q install trax==1.3.4
import pickle
import string
import ast
import numpy as np
import trax 
from trax.supervised import decoding
import textwrap 
# Will come handy later.
wrapper = textwrap.TextWrapper(width=70)<jupyter_output>INFO:tensorflow:tokens_length=568 inputs_length=512 targets_length=114 noise_density=0.15 mean_noise_span_length=3.0<jupyter_text>Part 1: Mounting your drive for data accessibilityRun the code cell below and follow the instructions to mount your drive. The data is the same as used in the coursera version of the assignment.<jupyter_code>from google.colab import drive
drive.mount('/content/drive/', force_remount=True)<jupyter_output>Mounted at /content/drive/<jupyter_text>Part 2: Getting things ready Run the code cell below to ready some functions which will later help us in decoding. The code and the functions are the same as the ones you previsouly ran on the coursera version of the assignment.<jupyter_code># again, we change dir to 'NLP C4 W3 Colabs & Data'
example_jsons = list(map(ast.literal_eval, open("/content/drive/My Drive/NLP C4 W3 Colabs & Data/data.txt")))

natural_language_texts = [example_json['text'] for example_json in example_jsons]

PAD, EOS, UNK = 0, 1, 2
 
def detokenize(np_array):
  return trax.data.detokenize(
      np_array,
      vocab_type='sentencepiece',
      vocab_file='sentencepiece.model',
      vocab_dir='/content/drive/My Drive/NLP C4 W3 Colabs & Data/')
 
def tokenize(s):
  # The trax.data.tokenize function operates on streams,
  # that's why we have to create 1-element stream with iter
  # and later retrieve the result with next.
  return next(trax.data.tokenize(
      iter([s]),
      vocab_type='sentencepiece',
      vocab_file='sentencepiece.model',
      vocab_dir='/content/drive/My Drive/NLP C4 W3 Colabs & Data/'))
 
vocab_size = trax.data.vocab_size(
    vocab_type='sentencepiece',
    vocab_file='sentencepiece.model',
    vocab_dir='/content/drive/My Drive/NLP C4 W3 Colabs & Data/')

def get_sentinels(vocab_size):
    sentinels = {}

    for i, char in enumerate(reversed(string.ascii_letters), 1):

        decoded_text = detokenize([vocab_size - i]) 
        
        # Sentinels, ex: <Z> - <a>
        sentinels[decoded_text] = f'<{char}>'
        
    return sentinels

sentinels = get_sentinels(vocab_size)   


def pretty_decode(encoded_str_list, sentinels=sentinels):
    # If already a string, just do the replacements.
    if isinstance(encoded_str_list, (str, bytes)):
        for token, char in sentinels.items():
            encoded_str_list = encoded_str_list.replace(token, char)
        return encoded_str_list
  
    # We need to decode and then prettyfy it.
    return pretty_decode(detokenize(encoded_str_list))


inputs_targets_pairs = []

# here you are reading already computed input/target pairs from a file
with open ('/content/drive/My Drive/NLP C4 W3 Colabs & Data/inputs_targets_pairs_file.txt', 'rb') as fp:
    inputs_targets_pairs = pickle.load(fp)  


def display_input_target_pairs(inputs_targets_pairs):
    for i, inp_tgt_pair in enumerate(inputs_targets_pairs, 1):
      inps, tgts = inp_tgt_pair
      inps, tgts = pretty_decode(inps), pretty_decode(tgts)
      print(f'[{i}]\n'
            f'inputs:\n{wrapper.fill(text=inps)}\n\n'
            f'targets:\n{wrapper.fill(text=tgts)}\n\n\n\n')      
display_input_target_pairs(inputs_targets_pairs)<jupyter_output>[1]
inputs:
Beginners BBQ <Z> Taking <Y> in Missoula! <X> want to get better <W>
making delicious <V>? You will have the opportunity, put this on <U>
calendar now <T> Thursday, September 22nd<S> World Class BBQ Champion,
<NAME> from Lonestar Smoke Rangers. He<R> be <Q> a beginner<P>
class for everyone <O> wants to<N> better <M> their <L> skills. He
will teach you <K> you need to know <J> compete in  <I> KCBS BBQ
competition, including techniques, recipes,<H>s, meat selection<G>
trimming, plus smoker <F> information. The cost to be in the class is
$35 per person<E> for spectator<D> is free. Included in the cost will
be either a t-shirt or apron and you will<C> tasting samples of each
meat that <B>.

targets:
<Z> Class <Y> Place <X> Do you <W> at <V> BBQ <U> your <T>.<S> join<R>
will <Q> teaching<P> level <O> who<N> get <M> with <L> culinary <K>
everything <J> to <I>a<H> timeline<G> and <F> and fire<E>, and<D>s
it<C> be <B> is prepared




[2]
inputs:
Discussion <Z> ' <Y> X Lion (10.[...]<jupyter_text>Part 3: BERT LossWe will not train the encoder which you have built in the assignment (coursera version). Training it could easily cost you a few days depending on which GPUs/TPUs you are using. Very few people train the full transformer from scratch. Instead, what the majority of people do, they load in a pretrained model, and they fine tune it on a specific task. That is exactly what you are about to do. Let's start by initializing and then loading in the model. Initialize the model from the saved checkpoint.<jupyter_code># Initializing the model
model = trax.models.Transformer(
    d_ff = 4096,
    d_model = 1024,
    max_len = 2048,
    n_heads = 16,
    dropout = 0.1,
    input_vocab_size = 32000,
    n_encoder_layers = 24,
    n_decoder_layers = 24,
    mode='predict')  # Change to 'eval' for slow decoding.
# Now load in the model
# this takes about 1 minute
shape11 = trax.shapes.ShapeDtype((1, 1), dtype=np.int32)  # Needed in predict mode.
model.init_from_file('/content/drive/My Drive/NLP C4 W3 Colabs & Data/models/model.pkl.gz',
                     weights_only=True, input_signature=(shape11, shape11))
# Uncomment to see the transformer's structure.
print(model)<jupyter_output>Serial_in2_out2[
  Select[0,1,1]_in2_out3
  Branch_out2[
    []
    PaddingMask(0)
  ]
  Cache_in2_out2[
    Serial_in2_out2[
      Embedding_32000_1024
      Dropout
      PositionalEncoding
      Serial_in2_out2[
        Branch_in2_out3[
          None
          Serial_in2_out2[
            LayerNorm
            Serial_in2_out2[
              Dup_out2
              Dup_out2
              Serial_in4_out2[
                Parallel_in3_out3[
                  Dense_1024
                  Dense_1024
                  Dense_1024
                ]
                PureAttention_in4_out2
                Dense_1024
              ]
            ]
            Dropout
          ]
        ]
        Add_in2
      ]
      Serial[
        Branch_out2[
          None
          Serial[
            LayerNorm
            Dense_4096
            Relu
            Dropout
            Dense_1024
            Dropout
          ]
        ]
        Add_in2
      ]
      Serial_in2_out2[
        Branch_in2_out3[
 [...]<jupyter_text>3.1 DecodingNow you will use one of the `inputs_targets_pairs` for input and as target. Next you will use the `pretty_decode` to output the input and target. The code to perform all of this has been provided below.<jupyter_code># # using the 3rd example
# c4_input = inputs_targets_pairs[2][0]
# c4_target = inputs_targets_pairs[2][1]

# using the 1st example
c4_input = inputs_targets_pairs[0][0]
c4_target = inputs_targets_pairs[0][1]

print('pretty_decoded input: \n\n', pretty_decode(c4_input))
print('\npretty_decoded target: \n\n', pretty_decode(c4_target))
print('\nc4_input:\n\n', c4_input)
print('\nc4_target:\n\n', c4_target)
print(len(c4_target))
print(len(pretty_decode(c4_target)))<jupyter_output>pretty_decoded input: 

 Beginners BBQ <Z> Taking <Y> in Missoula! <X> want to get better <W> making delicious <V>? You will have the opportunity, put this on <U> calendar now <T> Thursday, September 22nd<S> World Class BBQ Champion, <NAME> from Lonestar Smoke Rangers. He<R> be <Q> a beginner<P> class for everyone <O> wants to<N> better <M> their <L> skills. He will teach you <K> you need to know <J> compete in  <I> KCBS BBQ competition, including techniques, recipes,<H>s, meat selection<G> trimming, plus smoker <F> information. The cost to be in the class is $35 per person<E> for spectator<D> is free. Included in the cost will be either a t-shirt or apron and you will<C> tasting samples of each meat that <B>.

pretty_decoded target: 

 <Z> Class <Y> Place <X> Do you <W> at <V> BBQ <U> your <T>.<S> join<R> will <Q> teaching<P> level <O> who<N> get <M> with <L> culinary <K> everything <J> to <I>a<H> timeline<G> and <F> and fire<E>, and<D>s it<C> be <B> is prepared

c4_input:

 [1284[...]<jupyter_text>Run the cell below to decode<jupyter_code># Faster decoding: (still - maybe lower max_length to 20 for speed)
# Temperature is a parameter for sampling.
#   # * 0.0: same as argmax, always pick the most probable token
#   # * 1.0: sampling from the distribution (can sometimes say random things)
#   # * values inbetween can trade off diversity and quality, try it out!
output = decoding.autoregressive_sample(model, inputs=np.array(c4_input)[None, :],
                                        temperature=0.0, max_length=50)
print(wrapper.fill(pretty_decode(output[0])))<jupyter_output><Z> Class <Y> Place <X> Do you <W> at <V> BBQ <U> your <T>! This<S>,
is the first class taught by<R> will <Q> teaching<P> BBQ <O> who<N>
get <M> at <L> BBQ <K> everything <J> to <I>a<H> judging, judging<jupyter_text>Note: As you can see the RAM is almost full, it is because the model and the decoding is memory heavy. Running it the second time might give you an answer that makes no sense, or repetitive words. If that happens restart the runtime (see how to at the start of the notebook) and run all the cells again.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09274299187843857, 'fraction_numerical': 0.019561610339708324, 'mean_word_length': 3.5571030640668524, 'pattern_counts': {'":': 0, '<': 149, '<?xml version=': 0, '>': 154, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '60569', 'license': 'MIT', 'n_tokens_mistral': 3875, 'n_tokens_neox': 3679, 'n_words': 1364, 'path': 'NLP/Attention/3/Copy_of_C4W3_Colab_BERT_Loss_Model.ipynb', 'repo_name': 'verneh/DataSci', 'size': '11459'}