---
license: mit
---

# ESM-2 QLoRA for Predicting Binding Sites

This model is the ESM-2 model [esm2_t12_35M_UR50D](https://huggingface.co/facebook/esm2_t12_35M_UR50D) finetuned with QLoRA on 
[this dataset](https://huggingface.co/datasets/AmelieSchreiber/2600K_binding_sites) of 2.6M protein sequences with binding and active 
site annotations from UniProt. The model and dataset size were scaled in a one-to-one way (following the Chinchilla paper) up from the smaller 
QLoRA adaptations of the `esm2_t6_8M_UR50D` models which were trained on 600K proteins. Since this model is 4.375 times larger, a dataset 
approximately 4.375 times larger is needed if Chinchilla scaling laws hold for QLoRA finetuning of protein language models. Determining if 
such scaling laws also hold is part of this project, so checking for improvements in performance metrics over a period of 3 epochs, as well 
as checking for signs of overfitting for each epoch are underway. 


## QLoRA Info

```
trainable params: 71046 || all params: 17246053 || trainable%: 0.41195512967517844
```

```python
'eval_loss': 0.6011912822723389,
'eval_accuracy': 0.9297529150299436,
'eval_precision': 0.22835223718675476,
'eval_recall': 0.697386656717114,
'eval_f1': 0.3440490710592986,
'eval_auc': 0.8167222019799886,
'eval_mcc': 0.3730152153022164
```