---
metrics:
- matthews_correlation
- f1
tags:
- biology
- medical
---
# DNABERT-2 (modified)

Modified to configure the use of flash attention.

## Below are works from the original repository and jaandoui.

This version of DNABERT2 has been changed to be able to output the attention too, for attention analysis. 

**To the author of DNABERT2, feel free to use those modifications.**

Use  ```--model_name_or_path jaandoui/DNABERT2-AttentionExtracted``` instead of the original repository to have access to the attention.

Most of the modifications were done in Bert_Layer.py. 
It has been modified especially for fine tuning and hasn't been tried for pretraining.
Before or next to each modification, you can find ```"JAANDOUI"``` so to see al modifications, search for ```"JAANDOUI"```.
```"JAANDOUI TODO"``` means that if that part is going to be used, maybe something might be missing.

Now in ```Trainer``` (or ```CustomTrainer``` if overwritten) in ```compute_loss(..)``` when defining the model:
        ```outputs = model(**inputs, return_dict=True, output_attentions=True)```
activate the extraction of attention: ```output_attentions=True``` (and ```return_dict=True``` (optional)).
You can now extract the attention in ```outputs.attentions```
Note than the output has a third dimension, mostly of value 12, referring to the layer ```outputs.attentions[-1]``` refers to the attention of the last layer.
Read more about model outputs here: https://huggingface.co/docs/transformers/v4.40.2/en/main_classes/output#transformers.utils.ModelOutput

I'm also not using Triton, therefore cannot guarantee that it will work with it. 

I also read that there were some problems with extracting attention when using Flash Attention here: https://github.com/huggingface/transformers/issues/28903
Not sure if that is relevant for us, since it's about Mistral models.

I'm still exploring this attention, please don't take it as if it works 100%. I'll update the repository when I'm sure.

The official link to DNABERT2 [DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
](https://arxiv.org/pdf/2306.15006.pdf).

READ ME OF THE OFFICIAL DNABERT2:
We sincerely appreciate the MosaicML team for the [MosaicBERT](https://openreview.net/forum?id=5zipcfLC2Z) implementation, which serves as the base of DNABERT-2 development. 

DNABERT-2 is a transformer-based genome foundation model trained on multi-species genome. 

To load the model from huggingface:
```
import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True)
model = AutoModel.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True)
```

To calculate the embedding of a dna sequence
```
dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC"
inputs = tokenizer(dna, return_tensors = 'pt')["input_ids"]
hidden_states = model(inputs)[0] # [1, sequence_length, 768]

# embedding with mean pooling
embedding_mean = torch.mean(hidden_states[0], dim=0)
print(embedding_mean.shape) # expect to be 768

# embedding with max pooling
embedding_max = torch.max(hidden_states[0], dim=0)[0]
print(embedding_max.shape) # expect to be 768
```