<jupyter_start><jupyter_text>GPT Tokenizer files--- Learning ObjectivesThe goal of this lab is to examine the difference between BPE and GPTBPE Tokenizer.Later on, we will use the observations from this notebook to train a GPTBPE Tokenizer with our own raw text data.We will load and verify the GPTBPE Tokenizer and make sure the output tokens and token ids are as expected.  Let's review the source code of [GPT2 tokenizer](https://huggingface.co/transformers/_modules/transformers/tokenization_gpt2.html)    This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will    be encoded differently whether it is at the beginning of the sentence (without space) or not:             from transformers import GPT2Tokenizer         tokenizer = GPT2Tokenizer.from_pretrained("gpt2")                 tokenizer(" Hello world")['input_ids']        [18435, 995]We expect our custom tokenizer, which we will train later in lab 2,  will exhibit the same behavior of [treating spaces like parts of the tokens](https://huggingface.co/transformers/model_doc/gpt2.htmltransformers.GPT2Tokenizer). Install necessary python libraries.<jupyter_code>!pip install tokenizers transformers ipywidgets
!jupyter nbextension enable --py widgetsnbextension<jupyter_output><empty_output><jupyter_text>Next, we will proceed to fetch the pretrained GPT Tokenizer files, namely the vocab and merge files, will ideally looks like. We can later on use these observations to validate our custom trained GPTBPE tokenizer and the corresponding vocab.json and merges.txt file, in order to ensure the custom trained GPTBPE tokenizer will tokenze as expected.<jupyter_code>!wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json
!wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt<jupyter_output><empty_output><jupyter_text>Examine the vocab and merge files, observe the presence of Ġ character.Ġ = space + 256, this character is used as a control letter.<jupyter_code>import json
import random
with open('gpt2-vocab.json') as ip_file:
    o = json.load(ip_file)
    take=20
    rn=random.randint(0,len(o)-1)
    print("noted that the Ġ = space + 256 is the control letter")
    print(list(o.keys())[rn:rn+take])            
!tail -n 5 gpt2-merges.txt<jupyter_output><empty_output><jupyter_text>The following code block will load a default GPT2Tokenizer from HuggingFace's **_transformer_** library, verify the following:            from transformers import GPT2Tokenizer            tokenizer = GPT2Tokenizer.from_pretrained("gpt2")                    tokenizer(" Hello world")['input_ids']            expected token ids for " Hello world" is [18435, 995]Note: The HuggingFace's **transformer** library does not have functions to train the GPTBPE tokenizer, but it can load a pre-trained tokenizer given valid files. For training the GPTBPE Tokenizer, we will need to use another library called **tokenizers**, also from HuggingFace.<jupyter_code>from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

print('\n notice the **SPACE** in front of ** Hello world** \n')
sample_text=" Hello world"
print(sample_text)
out=tokenizer.tokenize(sample_text)
print("tokens:",out)
ids=tokenizer(sample_text)['input_ids']
print("ids:",ids)
## expected output :
## [18435, 995]<jupyter_output><empty_output><jupyter_text>Below is the expected outputs :             Hello world        tokens: ['ĠHello', 'Ġworld']        ids: [18435, 995]Observe the presence of the **Ġ** character.  In the next code block, we will examine how HuggingFace's **_tokenizers_** library loads a pretrained tokenizer given gpt2-vocab.json and merges.txt files. We will verify that, the usage of the `use_gpt` flag will result in the same tokenization behavior, i.e the presence of the **Ġ** character. We will also double check that the token IDs are identical to HuggingFace's **_transformer_** loaded `tokenizer = GPT2Tokenizer.from_pretrained("gpt2")` when applying tokenization to the exact same text ` Hello world`. Setting `use_gpt` to True will evoke the following :         tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()        tokenizer.decoder = ByteLevelDecoder()        This is the expected tokenizer behavior for GPTBPE Tokenizer. This GPTBPE tokenizer will load the vocab.json and merges.txt files and tokenize as expected. Whereas setting the `use_gpt` flag to False, will result in a normal BPE Tokenizer and the tokenization will behave differently.<jupyter_code>from tokenizers import Tokenizer, models, pre_tokenizers, trainers
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.models import BPE
import json

def load_tokenizer(vocab_file,merge_file, use_gpt):
    tokenizer = Tokenizer(BPE())
    tokenizer.model = BPE.from_file(vocab_file, merge_file)
    with open(vocab_file, 'r') as f2:
        vocab = json.loads(f2.read())  
    if use_gpt:
        tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()
        tokenizer.decoder = ByteLevelDecoder()
    return tokenizer , vocab
vocab_file='./gpt2-vocab.json'
merge_file='./gpt2-merges.txt'
tokenizers_gpt,_=load_tokenizer(vocab_file,merge_file,True)
sample_text=' Hello world' 
output=tokenizers_gpt.encode(sample_text)
ids=output.ids
tokens=output.tokens
#print(tokens ,'\n')
print("tokens: ",tokens)
print("ids:",ids)

tokenizers_bpe,_=load_tokenizer(vocab_file,merge_file, False)
sample_text=' Hello world'
output=tokenizers_bpe.encode(sample_text)
ids=output.ids
tokens=output.tokens
print("---"*10)
print('\nnotice the difference when using BPE as tokenizer instead of GPT2BPE tokenizer')
print("tokens: ",tokens)
print("ids:",ids)<jupyter_output><empty_output><jupyter_text>Below is the expected outputs :        tokens:  ['ĠHello', 'Ġworld']        ids: [18435, 995]        ------------------------------        notice the difference when using BPE as tokenizer instead of GPT2BPE tokenizer        tokens:  ['H', 'ellow', 'orld']        ids: [39, 5037, 1764] What did we observed? By setting the `use_gpt` flag to True in HuggingFace's **_tokenizers_** library when loading the same gpt2-vocab.json and merges.txt gives us the expected behavior of GPTBPE tokenization. We can further verify that by applying tokenization to the exact same text ` Hello world`, the result of the tokenizer, with the`use_gpt` flag = True, will match the result of the HuggingFace's  **_transformer_** library loaded gpt2 tokenizer.Whereas setting the `use_gpt` flag = False would result in a different behavior. Therefore, we will enforce having :    tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()    tokenizer.decoder = ByteLevelDecoder()When training our own GPTBPETokenizer with our own raw text data in Lab 2.     We will now move the gpt-vocab.json and gpt2-merges.txt to the correct data folder as a preparation for the next step.<jupyter_code>!mv gpt2-vocab.json ../dataset/EN/50k/
!mv gpt2-merges.txt ../dataset/EN/50k/
!ls ../dataset/EN/50k/<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09021167779514136, 'fraction_numerical': 0.014916891603921012, 'mean_word_length': 4.663716814159292, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '630634', 'license': 'Apache-2.0', 'n_tokens_mistral': 2161, 'n_tokens_neox': 2093, 'n_words': 737, 'path': 'ai/Megatron/English/Python/jupyter_notebook/Lab1-4_GPT_vocab_merge_files.ipynb', 'repo_name': 'programmah/gpubootcamp', 'size': '7039'}