<jupyter_start><jupyter_text>Extracting protein sequences' features using ProtBert-BFD pretrained-model  1. Load necessry libraries including huggingface transformers<jupyter_code>!pip install -q transformers
import torch
from transformers import AutoTokenizer, AutoModel, pipeline
import re
import numpy as np
import os
import requests
from tqdm.auto import tqdm<jupyter_output><empty_output><jupyter_text>2. Load the vocabulary and ProtBert-BFD Model<jupyter_code>tokenizer = AutoTokenizer.from_pretrained("Rostlab/prot_bert_bfd", do_lower_case=False )
model = AutoModel.from_pretrained("Rostlab/prot_bert_bfd")<jupyter_output><empty_output><jupyter_text>3. Load the model into the GPU if avilabile<jupyter_code>fe = pipeline('feature-extraction', model=model, tokenizer=tokenizer,device=0 )<jupyter_output><empty_output><jupyter_text>4. Create or load sequences and map rarely occured amino acids (U,Z,O,B) to (X)<jupyter_code>sequences_Example = ["A E T C Z A O","S K T Z P"]
sequences_Example = [re.sub(r"[UZOB]", "X", sequence) for sequence in sequences_Example]<jupyter_output><empty_output><jupyter_text>5. Extracting sequences' features and covert the output to numpy if needed<jupyter_code>embedding = fe(sequences_Example)
embedding = np.array(embedding)
print(embedding)<jupyter_output>[[[ 0.07873335  0.02530318  0.10022405 ... -0.11921909 -0.08556543
    0.0157964 ]
  [ 0.05551133 -0.10461304 -0.03253962 ...  0.05091606  0.04318975
    0.10181108]
  [ 0.13895561 -0.046583    0.02193631 ...  0.06942613  0.14762992
    0.06503808]
  ...
  [ 0.08129995 -0.1092955  -0.03022903 ...  0.08717731  0.02061446
    0.05156654]
  [ 0.06197417 -0.06417818 -0.02039655 ... -0.02796507  0.0884005
    0.07532689]
  [ 0.09792715 -0.02395107 -0.07244891 ... -0.02299493  0.03639549
    0.0507541 ]]

 [[ 0.11757391 -0.01604314  0.12586999 ... -0.0931114  -0.09125263
   -0.01659234]
  [-0.06304268 -0.23687428 -0.07115868 ... -0.03852162 -0.00322069
   -0.05244054]
  [ 0.01905588 -0.105173   -0.02930211 ... -0.00238627 -0.09289714
    0.02722595]
  ...
  [-0.00986713 -0.15825894 -0.08714887 ... -0.10954074 -0.03796846
   -0.03043361]
  [-0.01160221 -0.16065687 -0.11397075 ... -0.07776542 -0.00063999
   -0.02496129]
  [ 0.002604   -0.1685513  -0.06161089 ... -0.09357076 -0.0614622
   -0.02[...]<jupyter_text>Optional: Remove padding ([PAD]) and special tokens ([CLS],[SEP]) that is added by ProtBert-BFD model<jupyter_code>features = [] 

for seq_num in range(len(embedding)):
    seq_len = len(sequences_Example[seq_num].replace(" ", ""))
    start_Idx = 1
    end_Idx = seq_len+1
    seq_emd = embedding[seq_num][start_Idx:end_Idx]
    features.append(seq_emd)
print(features)<jupyter_output>[array([[ 0.05551133, -0.10461304, -0.03253962, ...,  0.05091606,
         0.04318975,  0.10181108],
       [ 0.13895561, -0.046583  ,  0.02193631, ...,  0.06942613,
         0.14762992,  0.06503808],
       [ 0.14610603, -0.08092842, -0.12500416, ..., -0.03651231,
         0.02485525,  0.07977536],
       ...,
       [ 0.02349902, -0.01549769, -0.05685329, ..., -0.01342281,
         0.01704315,  0.06431052],
       [ 0.08129995, -0.1092955 , -0.03022903, ...,  0.08717731,
         0.02061446,  0.05156654],
       [ 0.06197417, -0.06417818, -0.02039655, ..., -0.02796507,
         0.0884005 ,  0.07532689]]), array([[-0.06304268, -0.23687428, -0.07115868, ..., -0.03852162,
        -0.00322069, -0.05244054],
       [ 0.01905588, -0.105173  , -0.02930211, ..., -0.00238627,
        -0.09289714,  0.02722595],
       [ 0.07721861, -0.1703198 , -0.13987812, ..., -0.08390203,
         0.03587941, -0.01317161],
       [ 0.00872737, -0.1771819 , -0.05856298, ..., -0.09918059,
        -0.06392955,[...]
{'fraction_non_alphanumeric': 0.15911542610571736, 'fraction_numerical': 0.31283710895361383, 'mean_word_length': 3.053551912568306, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '335378', 'license': 'AFL-3.0', 'n_tokens_mistral': 2229, 'n_tokens_neox': 1578, 'n_words': 266, 'path': 'Embedding/PyTorch/Basic/ProtBert-BFD.ipynb', 'repo_name': 'taneishi/ProtTrans', 'size': '3708'}