<jupyter_start><jupyter_text>Important Notes:1. ProtT5-XL-BFD has both encoder and decoder, for feature extraction we only load and use the encoder part.2. Loading only the encoder part, reduces the inference speed and the GPU memory requirements by half.2. In order to use ProtT5-XL-BFD encoder, you must install the latest huggingface transformers version from their GitHub repo, or at least the bellow mentioned commit.3. If you are intersted in both the encoder and decoder, you should use T5Model rather than T5EncoderModel. Extracting protein sequences' features using ProtT5-XL-BFD pretrained-model **1. Load necessry libraries including huggingface transformers**<jupyter_code>!pip install -q SentencePiece git+https://github.com/huggingface/transformers.git@40ecaf0c2b1c0b3894e9abf619f32472c5a3b3ca
import torch
from transformers import T5EncoderModel, T5Tokenizer
import re
import numpy as np
import gc<jupyter_output><empty_output><jupyter_text>2. Load the vocabulary and ProtT5-XL-BFD Model<jupyter_code>tokenizer = T5Tokenizer.from_pretrained("Rostlab/prot_t5_xl_bfd", do_lower_case=False )
model = T5EncoderModel.from_pretrained("Rostlab/prot_t5_xl_bfd")
gc.collect()<jupyter_output><empty_output><jupyter_text>3. Load the model into the GPU if avilabile and switch to inference mode<jupyter_code>device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
model = model.eval()<jupyter_output><empty_output><jupyter_text>4. Create or load sequences and map rarely occured amino acids (U,Z,O,B) to (X)<jupyter_code>sequences_Example = ["A E T C Z A O","S K T Z P"]
sequences_Example = [re.sub(r"[UZOB]", "X", sequence) for sequence in sequences_Example]<jupyter_output><empty_output><jupyter_text>5. Tokenize, encode sequences and load it into the GPU if possibile<jupyter_code>ids = tokenizer.batch_encode_plus(sequences_Example, add_special_tokens=True, padding=True)
input_ids = torch.tensor(ids['input_ids']).to(device)
attention_mask = torch.tensor(ids['attention_mask']).to(device)<jupyter_output><empty_output><jupyter_text>6. Extracting sequences' features and load it into the CPU if needed<jupyter_code>with torch.no_grad():
    embedding = model(input_ids=input_ids,attention_mask=attention_mask)
embedding = embedding.last_hidden_state.cpu().numpy()<jupyter_output><empty_output><jupyter_text>7. Remove padding (\) and special tokens (\) that is added by ProtT5-XL-BFD model<jupyter_code>features = [] 
for seq_num in range(len(embedding)):
    seq_len = (attention_mask[seq_num] == 1).sum()
    seq_emd = embedding[seq_num][:seq_len-1]
    features.append(seq_emd)
print(features)<jupyter_output>[array([[ 0.2906924 , -0.20804922, -0.21934754, ...,  0.33846605,
         0.46031427, -0.16942444],
       [ 0.20362833, -0.12995909, -0.17139852, ...,  0.1064226 ,
        -0.37738243,  0.05401208],
       [ 0.03736684, -0.085066  , -0.23616493, ...,  0.25152582,
         0.13338618,  0.03063769],
       ...,
       [ 0.4778798 , -0.2302942 , -0.10682338, ...,  0.4057171 ,
         0.52513736,  0.1917527 ],
       [ 0.0799041 ,  0.07094701, -0.08554685, ...,  0.2494752 ,
         0.3653832 , -0.45506272],
       [ 0.09110155,  0.17046824,  0.42079163, ...,  0.25626677,
         0.02010923, -0.11016797]], dtype=float32), array([[ 0.28857493, -0.11107822, -0.13360327, ..., -0.06594554,
         0.00528727, -0.21770114],
       [ 0.13953964, -0.12703417,  0.06635736, ..., -0.02377463,
        -0.28750962,  0.09930848],
       [ 0.22406411,  0.02809858,  0.04245748, ...,  0.14432026,
         0.20209198, -0.22417574],
       [ 0.5806172 , -0.16730031, -0.1463695 , ...,  0.36016148,
     [...]
{'fraction_non_alphanumeric': 0.1299126637554585, 'fraction_numerical': 0.15283842794759825, 'mean_word_length': 4.228245363766049, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '586827', 'license': 'AFL-3.0', 'n_tokens_mistral': 1690, 'n_tokens_neox': 1360, 'n_words': 291, 'path': 'Embedding/PyTorch/Advanced/ProtT5-XL-BFD.ipynb', 'repo_name': 'taneishi/ProtTrans', 'size': '3664'}