🐛 [Metrics] ROUGE is non-deterministic
{'login': 'astariul', 'id': 43774355, 'node_id': 'MDQ6VXNlcjQzNzc0MzU1', 'avatar_url': 'https://avatars.githubusercontent.com/u/43774355?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/astariul', 'html_url': 'https://github.com/astariul', 'followers_url': 'https://api.github.com/users/astariul/followers', 'following_url': 'https://api.github.com/users/astariul/following{/other_user}', 'gists_url': 'https://api.github.com/users/astariul/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/astariul/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/astariul/subscriptions', 'organizations_url': 'https://api.github.com/users/astariul/orgs', 'repos_url': 'https://api.github.com/users/astariul/repos', 'events_url': 'https://api.github.com/users/astariul/events{/privacy}', 'received_events_url': 'https://api.github.com/users/astariul/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi, can you give a full self-contained example to reproduce this behavior?'
 '> Hi, can you give a full self-contained example to reproduce this behavior?\r\n\r\nThere is a notebook in the post ;)'
 '> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.\r\n> \r\n> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.\r\n> \r\n> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :\r\n> \r\n> > [\'0.3350\', \'0.1470\', \'0.2329\']\r\n> > [\'0.3358\', \'0.1451\', \'0.2332\']\r\n> \r\n> Why ROUGE is not deterministic ?\r\n\r\nThis is because of rouge\'s `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).\r\nYou can get deterministic scores per sentence pair by using\r\n```python\r\nscore = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_agregator=False)\r\n```\r\nOr you can set numpy\'s random seed if you still want to use the aggregator.'
 'Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?'
 'We should probably indeed!'
 "Now if you re-run the notebook, the two printed results are the same @colanim\r\n```\r\n['0.3356', '0.1466', '0.2318']\r\n['0.3356', '0.1466', '0.2318']\r\n```\r\nHowever across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:\r\n```python\r\nrouge = nlp.load_metric('rouge', seed=42)\r\n```"]
If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/361/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}