Prosho commited on
Commit
d448c08
1 Parent(s): c9b3a04

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +71 -0
README.md CHANGED
@@ -4,3 +4,74 @@ language: multilingual
4
  library_name: transformers
5
  license: cc-by-nc-sa-4.0
6
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  library_name: transformers
5
  license: cc-by-nc-sa-4.0
6
  ---
7
+
8
+ <div align="center">
9
+
10
+ <h1 style="font-family: 'Arial', sans-serif; font-size: 28px; font-weight: bold; color: black;">
11
+ 🛡️ Guardians of the Machine Translation Meta-Evaluation:<br>
12
+ Sentinel Metrics Fall In!
13
+ </h1>
14
+
15
+ </div>
16
+
17
+ <div style="display:flex; justify-content: center; align-items: center; flex-direction: row;">
18
+ <a href="https://2024.aclweb.org/"><img src="http://img.shields.io/badge/ACL-2024-4b44ce.svg"></a> &nbsp; &nbsp;
19
+ <a href="https://aclanthology.org/"><img src="http://img.shields.io/badge/paper-ACL--anthology-B31B1B.svg"></a> &nbsp; &nbsp;
20
+ <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg"></a>
21
+ </div>
22
+ <div style="display:flex; justify-content: center; align-items: center; flex-direction: row;">
23
+ <a href="https://huggingface.co/collections/sapienzanlp/mt-sentinel-metrics-66ab643b32aab06f3157e5c1"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-FCD21D"></a> &nbsp; &nbsp;
24
+ <a href="https://github.com/SapienzaNLP/guardians-mt-eval"><img src="https://img.shields.io/badge/GitHub-Repo-121013?logo=github&logoColor=white"></a> &nbsp; &nbsp;
25
+ </div>
26
+
27
+ This repository contains the **SENTINEL<sub>REF</sub>** metric model trained on Direct Assessments (DA) annotations. For details on how to use our sentinel metric models, check our [GitHub repository](https://github.com/SapienzaNLP/guardians-mt-eval).
28
+
29
+ ## Usage
30
+
31
+ After having installed our repository package, you can use this model within Python in the following way:
32
+
33
+ ```python
34
+ from sentinel_metric import download_model, load_from_checkpoint
35
+
36
+ model_path = download_model("sapienzanlp/sentinel-ref-da")
37
+ model = load_from_checkpoint(model_path)
38
+
39
+ data = [
40
+ {"ref": "There's no place like home."},
41
+ {"ref": "Toto, I've a feeling we're not in Kansas anymore."}
42
+ ]
43
+
44
+ output = model.predict(data, batch_size=8, gpus=1)
45
+ ```
46
+
47
+ Output:
48
+ ```python
49
+ # Segment scores
50
+ >>> output.scores
51
+ [0.26086926460266113, 0.17275342345237732]
52
+
53
+ # System score
54
+ >>> output.system_score
55
+ 0.21681134402751923
56
+ ```
57
+
58
+ ## Cite this work
59
+ This work has been published at [ACL 2024 (main conference)](https://2024.aclweb.org/program/main_conference_papers/). If you use any part, please consider citing our paper as follows:
60
+
61
+ ```bibtex
62
+ @inproceedings{perrella-etal-2024-guardians,
63
+ title = "Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!",
64
+ author = "Perrella, Stefano and
65
+ Proietti, Lorenzo and
66
+ Scirè, Alessandro and
67
+ Barba, Edoardo and
68
+ Navigli, Roberto",
69
+ booktitle = "Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2024)",
70
+ year = "2024",
71
+ address = "Bangkok, Thailand",
72
+ publisher = "Association for Computational Linguistics",
73
+ }
74
+ ```
75
+
76
+ ## License
77
+ This work is licensed under [Creative Commons Attribution-ShareAlike-NonCommercial 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/).