Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
{'login': 'lmvasque', 'id': 6901031, 'node_id': 'MDQ6VXNlcjY5MDEwMzE=', 'avatar_url': 'https://avatars.githubusercontent.com/u/6901031?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/lmvasque', 'html_url': 'https://github.com/lmvasque', 'followers_url': 'https://api.github.com/users/lmvasque/followers', 'following_url': 'https://api.github.com/users/lmvasque/following{/other_user}', 'gists_url': 'https://api.github.com/users/lmvasque/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/lmvasque/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/lmvasque/subscriptions', 'organizations_url': 'https://api.github.com/users/lmvasque/orgs', 'repos_url': 'https://api.github.com/users/lmvasque/repos', 'events_url': 'https://api.github.com/users/lmvasque/events{/privacy}', 'received_events_url': 'https://api.github.com/users/lmvasque/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
['Hi, thanks for reporting! We can add a `sources: datasets.Value("string")` feature to the `Features` dict in the `SARI` script to fix this. Would you be interested in submitting a PR?'
 'Hi Mario,\r\n\r\nThanks for your message. I did try to add `sources` into the `Features` dict using a script for the metric:\r\n```\r\n            features=datasets.Features(\r\n                {\r\n                    "sources": datasets.Value("string", id="sequence"),\r\n                    "predictions": datasets.Value("string", id="sequence"),\r\n                    "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),\r\n                }\r\n            ),\r\n```\r\n\r\nBut that only avoids a failure in `encode_batch` in the `add_batch` method:\r\n```\r\n        batch = {"predictions": predictions, "references": references}\r\n        batch = self.info.features.encode_batch(batch)\r\n```\r\n\r\nThe real problem is that `add_batch()`, `add()` and `compute()` does not receive a `sources` param:\r\n```\r\ndef add_batch(self, *, predictions=None, references=None):\r\ndef add(self, *, prediction=None, reference=None):\r\ndef compute(self, *, predictions=None, references=None, **kwargs)\r\n```\r\n\r\nAnd then, it fails:\r\n`TypeError: add_batch() got an unexpected keyword argument sources`\r\n\r\nI need this for adding any metric based on SARI or alike, not only for sari.py :)\r\n\r\nLet me know if I understood correctly the proposed solution.\r\n'
 "The `Metric` class has been modified recently to support this use-case, but the `add_batch` + `compute` pattern still doesn't work correctly. I'll open a PR."]
**Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input. 

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."], 
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```
  
**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch). 

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/3818/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}