Allow for incremental cumulative metric updates in a distributed setup
{'login': 'eladsegal', 'id': 13485709, 'node_id': 'MDQ6VXNlcjEzNDg1NzA5', 'avatar_url': 'https://avatars.githubusercontent.com/u/13485709?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/eladsegal', 'html_url': 'https://github.com/eladsegal', 'followers_url': 'https://api.github.com/users/eladsegal/followers', 'following_url': 'https://api.github.com/users/eladsegal/following{/other_user}', 'gists_url': 'https://api.github.com/users/eladsegal/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/eladsegal/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/eladsegal/subscriptions', 'organizations_url': 'https://api.github.com/users/eladsegal/orgs', 'repos_url': 'https://api.github.com/users/eladsegal/repos', 'events_url': 'https://api.github.com/users/eladsegal/events{/privacy}', 'received_events_url': 'https://api.github.com/users/eladsegal/received_events', 'type': 'User', 'site_admin': False}
[{'id': 1935892871, 'node_id': 'MDU6TGFiZWwxOTM1ODkyODcx', 'url': 'https://api.github.com/repos/huggingface/datasets/labels/enhancement', 'name': 'enhancement', 'color': 'a2eeef', 'default': True, 'description': 'New feature or request'}]
[]
Currently, using a metric allows for one of the following:
- Per example/batch metrics
- Cumulative metrics over the whole data

What I'd like is to have an efficient way to get cumulative metrics over the examples/batches added so far, in order to display it as part of the progress bar during training/evaluation.

Since most metrics are just an average of per-example metrics (which aren't?), an efficient calculation can be done as follows:
`((score_cumulative * n_cumulative) + (score_new * n_new)) / (n_cumulative+ n_new)`
where `n` and `score`  refer to number of examples and metric score, `cumulative` refers to the cumulative metric and `new` refers to the addition of new examples.

If you don't want to add this capability in the library, a simple solution exists so users can do it themselves:
It is easy to implement for a single process setup, but in a distributed one there is no way to get the correct `n_new`.
The solution for this is to return the number of examples that was used to compute the metrics in `.compute()` by adding the following line here:
https://github.com/huggingface/datasets/blob/5a3221785311d0ce86c2785b765e86bd6997d516/src/datasets/metric.py#L402-L403
```
output["number_of_examples"] = len(predictions)
```
and also remove the log message here so it won't spam:
https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/src/datasets/metric.py#L411

If this change is ok with you, I'll open a pull request.

{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2550/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}