Update README.md
Browse files
README.md
CHANGED
@@ -4,3 +4,74 @@ language: multilingual
|
|
4 |
library_name: transformers
|
5 |
license: cc-by-nc-sa-4.0
|
6 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
4 |
library_name: transformers
|
5 |
license: cc-by-nc-sa-4.0
|
6 |
---
|
7 |
+
|
8 |
+
<div align="center">
|
9 |
+
|
10 |
+
<h1 style="font-family: 'Arial', sans-serif; font-size: 28px; font-weight: bold; color: black;">
|
11 |
+
🛡️ Guardians of the Machine Translation Meta-Evaluation:<br>
|
12 |
+
Sentinel Metrics Fall In!
|
13 |
+
</h1>
|
14 |
+
|
15 |
+
</div>
|
16 |
+
|
17 |
+
<div style="display:flex; justify-content: center; align-items: center; flex-direction: row;">
|
18 |
+
<a href="https://2024.aclweb.org/"><img src="http://img.shields.io/badge/ACL-2024-4b44ce.svg"></a>
|
19 |
+
<a href="https://aclanthology.org/"><img src="http://img.shields.io/badge/paper-ACL--anthology-B31B1B.svg"></a>
|
20 |
+
<a href="https://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg"></a>
|
21 |
+
</div>
|
22 |
+
<div style="display:flex; justify-content: center; align-items: center; flex-direction: row;">
|
23 |
+
<a href="https://huggingface.co/collections/sapienzanlp/mt-sentinel-metrics-66ab643b32aab06f3157e5c1"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-FCD21D"></a>
|
24 |
+
<a href="https://github.com/SapienzaNLP/guardians-mt-eval"><img src="https://img.shields.io/badge/GitHub-Repo-121013?logo=github&logoColor=white"></a>
|
25 |
+
</div>
|
26 |
+
|
27 |
+
This repository contains the **SENTINEL<sub>REF</sub>** metric model trained on Direct Assessments (DA) annotations. For details on how to use our sentinel metric models, check our [GitHub repository](https://github.com/SapienzaNLP/guardians-mt-eval).
|
28 |
+
|
29 |
+
## Usage
|
30 |
+
|
31 |
+
After having installed our repository package, you can use this model within Python in the following way:
|
32 |
+
|
33 |
+
```python
|
34 |
+
from sentinel_metric import download_model, load_from_checkpoint
|
35 |
+
|
36 |
+
model_path = download_model("sapienzanlp/sentinel-ref-da")
|
37 |
+
model = load_from_checkpoint(model_path)
|
38 |
+
|
39 |
+
data = [
|
40 |
+
{"ref": "There's no place like home."},
|
41 |
+
{"ref": "Toto, I've a feeling we're not in Kansas anymore."}
|
42 |
+
]
|
43 |
+
|
44 |
+
output = model.predict(data, batch_size=8, gpus=1)
|
45 |
+
```
|
46 |
+
|
47 |
+
Output:
|
48 |
+
```python
|
49 |
+
# Segment scores
|
50 |
+
>>> output.scores
|
51 |
+
[0.26086926460266113, 0.17275342345237732]
|
52 |
+
|
53 |
+
# System score
|
54 |
+
>>> output.system_score
|
55 |
+
0.21681134402751923
|
56 |
+
```
|
57 |
+
|
58 |
+
## Cite this work
|
59 |
+
This work has been published at [ACL 2024 (main conference)](https://2024.aclweb.org/program/main_conference_papers/). If you use any part, please consider citing our paper as follows:
|
60 |
+
|
61 |
+
```bibtex
|
62 |
+
@inproceedings{perrella-etal-2024-guardians,
|
63 |
+
title = "Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!",
|
64 |
+
author = "Perrella, Stefano and
|
65 |
+
Proietti, Lorenzo and
|
66 |
+
Scirè, Alessandro and
|
67 |
+
Barba, Edoardo and
|
68 |
+
Navigli, Roberto",
|
69 |
+
booktitle = "Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2024)",
|
70 |
+
year = "2024",
|
71 |
+
address = "Bangkok, Thailand",
|
72 |
+
publisher = "Association for Computational Linguistics",
|
73 |
+
}
|
74 |
+
```
|
75 |
+
|
76 |
+
## License
|
77 |
+
This work is licensed under [Creative Commons Attribution-ShareAlike-NonCommercial 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).
|