|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
base_model: |
|
- SIRIS-Lab/affilgood-affilroberta |
|
tags: |
|
- affiliations |
|
- ner |
|
- science |
|
--- |
|
|
|
# AffilGood-SPAN |
|
|
|
## Overview |
|
|
|
<details> |
|
<summary>Click to expand</summary> |
|
|
|
- **Model type:** Language Model |
|
- **Architecture:** RoBERTa-base |
|
- **Language:** English |
|
- **License:** Apache 2.0 |
|
- **Task:** Named Entity Recognition |
|
- **Data:** AffilGood-SPAN |
|
- **Additional Resources:** |
|
- [Paper](https://https://aclanthology.org/2024.sdp-1.13/) |
|
- [GitHub](https://github.com/sirisacademic/affilgood) |
|
</details> |
|
|
|
## Model description |
|
|
|
The English version of **affilgood-span** is a Named Entity Recognition (NER) model for identifying span of single affiliation strings in raw affiliation strings from scientific papers and projects, fine-tuned from the [AffilRoberta](https://huggingface.co/SIRIS-Lab/affilgood-affilroberta) model, a [RoBERTa](https://arxiv.org/abs/1907.11692) base model futher pre-trained for MLM task on a medium-size corpus of raw affiliation stirngs collected from OpenAlex. |
|
|
|
Raw affiliation span identification task is aimed at extracting and cleaning affiliation strings when there is noise and/or when there are multiple affiliation strings in the same signature. |
|
Typically, multiple institutions have been considered to be separated by semicolons. However, other punctuation marks, spaces or *and* connectors are frequently used to separate affiliations. |
|
|
|
We have annotated a dataset containing 2,072 raw affiliation strings obtained from OpenAlex to identify spans containing relevant affiliation data within them. |
|
The annotated instances were selected by a stratified random sampling by country, focusing on ensuring diversity in affiliation languages and origins. |
|
Additional manually-chosen instances with noisy sequences were included in the annotated data so we could train our model to filter out non-affiliation strings. |
|
It is frequent that affiliation data automatically extracted from PDF files contain texts that should have been discarded (e.g. email, acknowledgements or part of the contents of the publication). These data can introduce errors in the subsequent steps of the pipeline. |
|
|
|
We have fine-tuned the models for 25 epochs, using 80% of the dataset for training, 10% for validation and 10% for testing. Best performing model is available here. |
|
|
|
## Intended Usage |
|
|
|
This model is intended to be used for raw affiliation strings in English, because this model is pre-trained on English RoBERTa, however SPAN and large further pre-training corpora are both multilingual. |
|
|
|
## How to use |
|
|
|
```python |
|
from transformers import pipeline |
|
affilgood_span_pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple") |
|
sentence = "Deanery of Biomedical Sciences, University of Edinburgh, and TW2Informatics Ltd, Gothenburg, 42166, Sweden / Received in revised form February 22, 2016; ENaC, epithelial sodium channel" |
|
output = affilgood_span_pipeline(sentence) |
|
print(output) |
|
``` |
|
|
|
|
|
## Limitations and bias |
|
|
|
No measures have been taken to estimate the bias and toxicity embedded in the model. |
|
|
|
We have annotated a dataset containing 2,072 raw affiliation strings obtained from OpenAlex to identify spans containing relevant affiliation data within them. |
|
The annotated instances were selected by a stratified random sampling by country, focusing on ensuring diversity in affiliation languages and origins. |
|
Additional manually-chosen instances with noisy sequences were included in the annotated data so we could train our model to filter out non-affiliation strings. |
|
It is frequent that affiliation data automatically extracted from PDF files contain texts that should have been discarded (e.g. email, acknowledgements or part of the contents of the publication). These data can introduce errors in the subsequent steps of the pipeline. |
|
|
|
## Training |
|
|
|
We used the [AffilGood-SPAN dataset](link) for training and evaluation. |
|
|
|
We fine-tuned the adapted and base models for token classification with the IOB annotation schema. |
|
We trained the models for 25 epochs, using 80% of the dataset for training, 10% for validation and 10% for testing. |
|
|
|
Hyperparameters used for training are described here: |
|
- Learning Rate: 2e-5 |
|
- Learning Rate Decay: Linear |
|
- Weight Decay: 0.01 |
|
- Warmup Portion: 0.06 |
|
- Batch Size: 128 |
|
- Number of Steps: 25k steps |
|
- Adam ε: 1e-6 |
|
- Adam β<sub>1</sub>: 0.9 |
|
- Adam β<sub>2</sub>: 0.999 |
|
|
|
The **best performing epoch (considering macro-averaged F1 with *strict* matching criteria) was used to select the model**. |
|
|
|
### Evaluation |
|
|
|
The model's performance was evaluated on a 10% of the dataset. |
|
|
|
| **Model** | **Exact F1** | **Partial F1** | |
|
|--------------------------|--------------|----------------| |
|
| Semicolon split (baseline) | .793 | .907 | |
|
| RoBERTa | .929 | .981 | |
|
| XLM | .931 | .978 | |
|
| AffilRoBERTa | **.938** | **.981** | |
|
| AffilXLM | .927 | .979 | |
|
|
|
All the numbers reported above represent F1-score with *strict* and *partial* match, when both the boundaries and types of the entities match. |
|
|
|
## Additional information |
|
|
|
### Authors |
|
|
|
- SIRIS Lab, Research Division of SIRIS Academic, Barcelona, Spain |
|
- LaSTUS Lab, TALN Group, Universitat Pompeu Fabra, Barcelona, Spain |
|
- Institute of Computer Science, Polish Academy of Sciences, Warsaw, Poland |
|
|
|
### Contact |
|
|
|
For further information, send an email to either <[email protected]> or <[email protected]>. |
|
|
|
### License |
|
|
|
This work is distributed under a [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). |
|
|
|
### Funding |
|
|
|
This work was partially funded and supporter by: |
|
- Industrial Doctorates Plan of the Department of Research and Universities of the Generalitat de Catalunya, by Departament de Recerca i Universitats de la Generalitat de Catalunya (ajuts SGR-Cat 2021), |
|
- Maria de Maeztu Units of Excellence Programme CEX2021-001195-M, funded by MCIN/AEI /10.13039/501100011033 |
|
- EU HORIZON SciLake (Grant Agreement 101058573) |
|
- EU HORIZON ERINIA (Grant Agreement 101060930) |
|
|
|
### Citation |
|
|
|
```bibtex |
|
@inproceedings{duran-silva-etal-2024-affilgood, |
|
title = "{A}ffil{G}ood: Building reliable institution name disambiguation tools to improve scientific literature analysis", |
|
author = "Duran-Silva, Nicolau and |
|
Accuosto, Pablo and |
|
Przyby{\l}a, Piotr and |
|
Saggion, Horacio", |
|
editor = "Ghosal, Tirthankar and |
|
Singh, Amanpreet and |
|
Waard, Anita and |
|
Mayr, Philipp and |
|
Naik, Aakanksha and |
|
Weller, Orion and |
|
Lee, Yoonjoo and |
|
Shen, Shannon and |
|
Qin, Yanxia", |
|
booktitle = "Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024)", |
|
month = aug, |
|
year = "2024", |
|
address = "Bangkok, Thailand", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2024.sdp-1.13", |
|
pages = "135--144", |
|
} |
|
``` |
|
|
|
### Disclaimer |
|
|
|
<details> |
|
<summary>Click to expand</summary> |
|
|
|
The model published in this repository is intended for a generalist purpose |
|
and is made available to third parties under a Apache v2.0 License. |
|
|
|
Please keep in mind that the model may have bias and/or any other undesirable distortions. |
|
When third parties deploy or provide systems and/or services to other parties using this model |
|
(or a system based on it) or become users of the model itself, they should note that it is under |
|
their responsibility to mitigate the risks arising from its use and, in any event, to comply with |
|
applicable regulations, including regulations regarding the use of Artificial Intelligence. |
|
|
|
In no event shall the owners and creators of the model be liable for any results arising from the use made by third parties. |
|
</details> |
|
|
