<jupyter_start><jupyter_text>How to use the BOW^2 (Bag of Whole-Words) Tokenizer (Standalone)The BOW^2 tokenizer gives you two main additions to the HuggingFace tokenizer output: a list of unique words, and a mapping between the input_ids and the corresponding unique words. This can then be used to aggregate sub-word representations into unique whole-words.<jupyter_code>from bow2_tokenizer import BOW2Tokenizer
from transformers import AutoTokenizer

#
# use create_global_id=False to get simple ids (good for debugging)
#
tok = BOW2Tokenizer(AutoTokenizer.from_pretrained("distilbert-base-uncased"), create_global_id=False)
tokenized_data = tok.tokenize("This is a sentence with redundant words, superlongandconnectedwords as a sentences. Usually a passage contains some words multiple times :)").data

print("Input ids:",len(tokenized_data["input_ids"][0]),"tokens:", tokenized_data["input_ids"][0])
print("Unique input ids:",len(tokenized_data["unique_input_ids"]),"tokens:", tokenized_data["unique_input_ids"])
print("Input ids to unique_words map:",len(tokenized_data["input_ids_to_words_map"][0]),"tokens:", tokenized_data["input_ids_to_words_map"][0])
print("Unique whole words:",len(tokenized_data["unique_words"][0]),"tokens:", tokenized_data["unique_words"][0])

from bow2_tokenizer import BOW2Tokenizer
from transformers import AutoTokenizer

#
# use create_global_id=True (default) to get simple globally usable hashed ids - 32 bits (good for matching queries + documents from different input data loader processes)
#
tok = BOW2Tokenizer(AutoTokenizer.from_pretrained("distilbert-base-uncased"),create_global_id=True)
tokenized_data = tok.tokenize("This is a sentence with redundant words, superlongandconnectedwords as a sentences. Usually a passage contains some words multiple times :)")

print("Input ids:",len(tokenized_data["input_ids"][0]),"tokens:", tokenized_data["input_ids"][0])
print("Unique input ids:",len(tokenized_data["unique_input_ids"]),"tokens:", tokenized_data["unique_input_ids"])
print("Input ids to unique_words map:",len(tokenized_data["input_ids_to_words_map"][0]),"tokens:", tokenized_data["input_ids_to_words_map"][0])
print("Unique whole words:",len(tokenized_data["unique_words"][0]),"whole words:", tokenized_data["unique_words"][0])<jupyter_output>Input ids: 29 tokens: tensor([  101,  2023,  2003,  1037,  6251,  2007, 21707,  2616,  1010,  3565,
        10052,  5685, 24230, 22104,  2004,  1037, 11746,  1012,  2788,  1037,
         6019,  3397,  2070,  2616,  3674,  2335,  1024,  1007,   102])
Unique input ids: 26 tokens: tensor([  101,   102,  1007,  1010,  1012,  1024,  1037,  2003,  2004,  2007,
         2023,  2070,  2335,  2616,  2788,  3397,  3565,  3674,  5685,  6019,
         6251, 10052, 11746, 21707, 22104, 24230])
Input ids to unique_words map: 29 tokens: tensor([         0,   43431710, 1241338362,  310482890, 3608744322, 1672844550,
        3176364122, 4285244379, 3288479184, 3315150416, 3315150416, 3315150416,
        3315150416, 3315150416, 2141175796,  310482890, 2870031442,  720286925,
        1373811150,  310482890, 1370523144, 4213238718, 3496850598, 4285244379,
        1719568959,  991622699, 2248649959,  499474106,          0])
Unique whole words: 20 tokens: tensor([  43431710, 1241338362,  310482890, 36087443[...]<jupyter_text>How to aggregate sub-word representations to BOW^2 (Bag of Whole-Words) representationsWe have the additional tokenizer infos + a tensor of encoded input_id (BERT output) -> the following code shows how to do the aggregated transformation (which is part of the ColBERTer model). This can be easily applied in any BERT based model, that does something with the output vectors on a word-level afterwards.**Important note:** We also create a new mask for the resulting tensor -> because we have a new shape with different padding (not visible in this example as we only have one sentence in the batch)<jupyter_code>import torch

# this is the BERT model output, with a toy example of 1 sentence in the batch and 8 dimensions
subword_vecs = torch.ones((1,len(tokenized_data["input_ids"][0]),8), dtype=torch.float32)

#
# dynamic aggregation of subword vectors to whole word vectors
#
aggregation_mask = (tokenized_data["unique_words"].unsqueeze(-1) == tokenized_data["input_ids_to_words_map"].unsqueeze(1)).unsqueeze(-1)
aggregated_vecs = (subword_vecs.unsqueeze(1).expand(-1, aggregation_mask.shape[1], -1, -1)*aggregation_mask).sum(2)

# mean pooling
aggregated_vecs = aggregated_vecs / aggregation_mask.float().sum(-2)

# create a new mask (!) this is important for batched inputs, we need to swap the mask as well (because we have a new tensor shape)
wholeword_mask = tokenized_data["unique_words"] > 0
wholeword_vecs = aggregated_vecs

print("Input shape:",subword_vecs.shape)
print("Resulting shape:",wholeword_vecs.shape)

print("Result:",wholeword_vecs) # should be all ones -> we take the mean of the subword vectors per whole word, comment out the pooling line to see how many occurrences there are :)<jupyter_output>Input shape: torch.Size([1, 29, 8])
Resulting shape: torch.Size([1, 20, 8])
Result: tensor([[[1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1., 1., 1.]]])
{'fraction_non_alphanumeric': 0.1534603811434303, 'fraction_numerical': 0.1240387830157138, 'mean_word_length': 3.8800978792822187, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '655523', 'license': 'Apache-2.0', 'n_tokens_mistral': 2551, 'n_tokens_neox': 2037, 'n_words': 662, 'path': 'bow2_tokenizer_example.ipynb', 'repo_name': 'sebastian-hofstaetter/colberter', 'size': '5982'}