<jupyter_start><jupyter_text>How does ALBEF handle tokenization?The captions in the input JSON fed into ALBEF are raw text. They are loaded into the [`pretrain_dataset`](https://sourcegraph.com/github.com/salesforce/ALBEF/-/blob/dataset/caption_dataset.py?L97) class, where the `pre_caption` function does some basic preprocessing. It's then wrapped by a `create_dataset` function that doesn't alter the text data.So, in summary, the dataset that comes out of `create_dataset` has `str` typed text data, not integer input ids.The dataset then gets passed into a `create_loader` function, which also does not modify the text data. The tokenization happens in the [`train`](https://sourcegraph.com/github.com/salesforce/ALBEF@9e9a5e952f72374c15cea02d3c34013554c86513/-/blob/Pretrain.py?L59) function.```pythontext_input = tokenizer(text, padding='longest', truncation=True, max_length=25, return_tensors="pt").to(device)```<jupyter_code>text = [
    "this",
    "this is",
    "this is a",
    "this is a random",
    "this is a random image"
]
text_input = tokenizer(text, padding='longest', truncation=True, max_length=25, return_tensors="pt")
text_input<jupyter_output><empty_output><jupyter_text>How does sentence-pair tokenization look?<jupyter_code>text_input = tokenizer.batch_encode_plus(list(zip(text, text)), padding='longest', truncation=True, max_length=50, return_tensors="pt")
text_input<jupyter_output><empty_output><jupyter_text>The input ALBEF gets is a dictionary containing `input_ids` and the `attention_mask`. The attention mask covers all the non-pad tokens. In sentence-pair mode, there is no padding in between the sentences. The sentences are just separated by the `[SEP]` symbol.   Constructing a sentence pair from ViT / word embeddingsThe ViT output sequence is always the same length, and if stacked together, has no ragged edges. The sentences cannot be stacked together without padding.<jupyter_code>image = torch.rand(3, 224, 224).unsqueeze(0)
vit_out = vm(image)
vit_out.shape
text = [
    "this",
    "this is",
    "this is a",
    "this is a random",
    "this is a random image"
]
# Make a batch the same size as the amount of text.
img_batch = torch.vstack([vit_out] * len(text))
img_batch.shape
# Tokenize the text.
text_batch = tokenizer(text, padding='longest', truncation=True, max_length=25, return_tensors="pt", add_special_tokens=False)
text_batch
prefix = torch.ones(batch_size, 1, 1) * tokenizer.cls_token_id
separator = torch.ones(batch_size, 1, 1) * tokenizer.sep_token_id
eos = torch.ones(batch_size, 1, 1) * tokenizer.sep_token_id<jupyter_output><empty_output><jupyter_text>The `input_embeds` keyword, which we will be passing data into, is filled from the `word_embeddings` layer if not provided (https://github.com/huggingface/transformers/blob/v4.15.0/src/transformers/models/bert/modeling_bert.pyL214).<jupyter_code>lang_input_embeds = lm.embeddings.word_embeddings(text_batch['input_ids'])
prefix_embeds = lm.embeddings.word_embeddings(prefix.long()).squeeze(1)
sep_embeds = lm.embeddings.word_embeddings(separator.long()).squeeze(1)
eos_embeds = lm.embeddings.word_embeddings(eos.long()).squeeze(1)
model_input = torch.cat([prefix_embeds, img_batch, sep_embeds, lang_input_embeds, eos_embeds], dim=1)
lm.bert(inputs_embeds=model_input)<jupyter_output><empty_output><jupyter_text>Computing the  token_type_id mask<jupyter_code>sentence_pair = tokenizer.encode_plus(
    text="this is some text",
    text_pair="a second sentence, longer than the first",
    padding='longest', 
    truncation=True, 
    max_length=25, 
    return_tensors="pt", 
    add_special_tokens=True
)
print(sentence_pair.input_ids)
print(sentence_pair.input_ids.shape)
print(sentence_pair.attention_mask)
print(sentence_pair.attention_mask.shape)
print(sentence_pair.token_type_ids)
print(sentence_pair.token_type_ids.shape)<jupyter_output>tensor([[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]])
torch.Size([1, 15])
{'fraction_non_alphanumeric': 0.11308316430020283, 'fraction_numerical': 0.0218052738336714, 'mean_word_length': 6.346368715083799, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '143542', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1403, 'n_tokens_neox': 1315, 'n_words': 334, 'path': 'notebooks/1.Explore-Single-Stream.ipynb', 'repo_name': 'codezakh/ALBEF', 'size': '3944'}