<jupyter_start><jupyter_text>Extracting protein sequences' features using ProtBert pretrained-model  1. Load necessry libraries including huggingface transformers<jupyter_code>!pip install -q transformers
import torch
from transformers import BertModel, BertTokenizer
import re
import os
import requests
from tqdm.auto import tqdm<jupyter_output><empty_output><jupyter_text>2. Load the vocabulary and ProtBert Model<jupyter_code>tokenizer = BertTokenizer.from_pretrained("Rostlab/prot_bert", do_lower_case=False)
model = BertModel.from_pretrained("Rostlab/prot_bert")<jupyter_output><empty_output><jupyter_text>3. Load the model into the GPU if avilabile and switch to inference mode<jupyter_code>device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
model = model.eval()<jupyter_output><empty_output><jupyter_text>4. Create or load sequences and map rarely occured amino acids (U,Z,O,B) to (X)<jupyter_code>sequences_Example = ["A E T C Z A O","S K T Z P"]
sequences_Example = [re.sub(r"[UZOB]", "X", sequence) for sequence in sequences_Example]<jupyter_output><empty_output><jupyter_text>5. Tokenize, encode sequences and load it into the GPU if possibile<jupyter_code>ids = tokenizer.batch_encode_plus(sequences_Example, add_special_tokens=True, pad_to_max_length=True)
input_ids = torch.tensor(ids['input_ids']).to(device)
attention_mask = torch.tensor(ids['attention_mask']).to(device)<jupyter_output><empty_output><jupyter_text>6. Extracting sequences' features and load it into the CPU if needed<jupyter_code>with torch.no_grad():
    embedding = model(input_ids=input_ids,attention_mask=attention_mask)[0]
embedding = embedding.cpu().numpy()<jupyter_output><empty_output><jupyter_text>7. Remove padding ([PAD]) and special tokens ([CLS],[SEP]) that is added by Bert model<jupyter_code>features = [] 
for seq_num in range(len(embedding)):
    seq_len = (attention_mask[seq_num] == 1).sum()
    seq_emd = embedding[seq_num][1:seq_len-1]
    features.append(seq_emd)
print(features)<jupyter_output>[array([[ 0.09233951,  0.13914399, -0.05241546, ..., -0.13947977,
        -0.0428006 ,  0.07431364],
       [ 0.11505969,  0.01998261, -0.08627468, ..., -0.00946233,
        -0.1872724 ,  0.13169006],
       [ 0.04012022,  0.05467906, -0.0424584 , ...,  0.02054804,
        -0.03819017,  0.1011811 ],
       ...,
       [ 0.08853199,  0.0228724 , -0.05121122, ..., -0.0673336 ,
        -0.03013714,  0.04913732],
       [ 0.10792391,  0.09771117, -0.05832333, ..., -0.12765658,
        -0.06493297,  0.12894136],
       [ 0.0545795 ,  0.03635893, -0.07821195, ..., -0.03016043,
        -0.06015311,  0.08903892]], dtype=float32), array([[ 0.0880546 ,  0.10885102,  0.06212992, ...,  0.0144938 ,
        -0.03327068, -0.0362049 ],
       [-0.03332657,  0.03442192,  0.0809955 , ...,  0.0008738 ,
        -0.02733596,  0.05783575],
       [ 0.07896441,  0.10343076, -0.01508527, ...,  0.06711898,
        -0.01733149,  0.12912118],
       [ 0.01805276,  0.07072446, -0.03629201, ..., -0.03547218,
     [...]
{'fraction_non_alphanumeric': 0.1458538184201901, 'fraction_numerical': 0.1717469682071452, 'mean_word_length': 4.120805369127517, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '512542', 'license': 'AFL-3.0', 'n_tokens_mistral': 1479, 'n_tokens_neox': 1173, 'n_words': 216, 'path': 'Embedding/PyTorch/Advanced/ProtBert.ipynb', 'repo_name': 'taneishi/ProtTrans', 'size': '3051'}