<jupyter_start><jupyter_text>First install HuggingFace `datasets`:<jupyter_code>!pip install datasets<jupyter_output><empty_output><jupyter_text>Or use *conda*:<jupyter_code>!conda install -c huggingface -c conda-forge datasets<jupyter_output><empty_output><jupyter_text>If this doesn't work, install from source:<jupyter_code>!git clone https://github.com/huggingface/datasets.git
!cd datsets
!pip install -e .<jupyter_output><empty_output><jupyter_text>---<jupyter_code>from datasets import load_dataset<jupyter_output><empty_output><jupyter_text>We then load the **Italian** part of the [**OSCAR**](https://huggingface.co/datasets/oscar) dataset. This is a *huge* dataset so download can take a long time:<jupyter_code>dataset = load_dataset('oscar', 'unshuffled_deduplicated_it')
dataset
len(dataset['train'])
dataset['train']
dataset['train'].features
dataset['train'][0]<jupyter_output><empty_output><jupyter_text>Now we save this data to file as several *plaintext* files.<jupyter_code>from tqdm.auto import tqdm

text_data = []
file_count = 0

for sample in tqdm(dataset['train']):
    sample = sample['text'].replace('\n', '')
    text_data.append(sample)
    if len(text_data) == 10_000:
        with open(f'../../data/text/oscar_it/text_{file_count}.txt', 'w', encoding='utf-8') as fp:
            fp.write('\n'.join(text_data))
        text_data = []
        file_count += 1<jupyter_output>100%|██████████| 28522082/28522082 [33:39<00:00, 14126.15it/s]<jupyter_text>Now we get a list of paths to each file in our *oscar_it* directory.<jupyter_code>from pathlib import Path

paths = [str(x) for x in Path('../../data/text/oscar_it').glob('**/*.txt')]

paths[-5:]<jupyter_output><empty_output><jupyter_text>Now we move onto training the tokenizer. We use a byte-level Byte-pair encoding (BPE) tokenizer. This allows us to build the vocabulary from an alphabet of single bytes, meaning all words will be decomposable into tokens.<jupyter_code>from tokenizers import ByteLevelBPETokenizer

tokenizer = ByteLevelBPETokenizer()
tokenizer.train(files=paths[:5], vocab_size=52_000, min_frequency=2,
                special_tokens=['[PAD]', '[CLS]', '[SEP]', '[MASK]', '[UNK]'])<jupyter_output><empty_output><jupyter_text>We can now save our tokenizer to file, we'll be giving our model a traditional Italian name - filiBERTo:<jupyter_code>import os

os.mkdir('./filiberto')

tokenizer.save_model('./filiberto', 'filiberto')<jupyter_output><empty_output><jupyter_text>Now we have two files that outline our new filiBERTo tokenizer:* the *vocab.json* - a mapping file between tokens to token IDs* and *merges.txt* - which describes which characters/set of characters can be decomposed/composed smaller/larger tokensTo begin using our tokenizer like we would usually use a `from_pretrained` tokenizer we do this:<jupyter_code>from tokenizers.implementations import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing

# initialize the tokenizer using the tokenizer we initialized and saved to file
tokenizer = ByteLevelBPETokenizer(
    './filiberto/filiberto-vocab.json',
    './filiberto/filiberto-merges.txt'
)

# set [CLS] and [SEP] to be added to start-end of sequences
tokenizer._tokenizer.post_processor = BertProcessing(
    ('[SEP]', tokenizer.token_to_id('[SEP]')),
    ('[CLS]', tokenizer.token_to_id('[CLS]'))
)

# truncate anything more than 512 characters in length
tokenizer.enable_truncation(max_length=512)

# test our tokenizer on a simple sentence
tokens = tokenizer.encode('ciao, come va?')
print(tokens)
tokens.ids<jupyter_output><empty_output><jupyter_text>We can see here that our **CLS** token is now placed at the beginning of our sequences using token ID *1*. At the end of the sequence we see the **SEP** token represented by *2*.<jupyter_code>52e3<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12549019607843137, 'fraction_numerical': 0.016209150326797386, 'mean_word_length': 5.994515539305302, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 44, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '348086', 'license': 'MIT', 'n_tokens_mistral': 1334, 'n_tokens_neox': 1280, 'n_words': 334, 'path': 'course/bert_from_scratch/00.ipynb', 'repo_name': 'chiragnsolanki/transformers', 'size': '3825'}