How to Add New Metrics Guide
Hi ! sorry for the late response 

It would be fantastic to have a guide for adding metrics as well ! Currently we only have this template here:
https://github.com/huggingface/datasets/blob/master/templates/new_metric_script.py

We can also include test utilities for metrics in the guide.

We have a pytest suite with commands that you can use to make sure your metric works as expected.
It has two useful commands:

1. This commands tests the code in the `Examples:` desction of the docstring of the metric:
```
pytest tests/test_metric_common.py::LocalMetricTest::test_load_metric_<metric_name>
```
This will run this code for example:

https://github.com/huggingface/datasets/blob/e0787aa2a781cc15a80f7597f56d1f12e23df4c9/metrics/accuracy/accuracy.py#L40-L45

Moreover this test is meant to be fast so users are free to add patches to the metric to avoid intensive computations.
And example of intensive call patch can be found here:

https://github.com/huggingface/datasets/blob/e0787aa2a781cc15a80f7597f56d1f12e23df4c9/tests/test_metric_common.py#L138-L151

2. This test runs the same thing as 1. except that it doesn't use patches (the real metric is used):
```
RUN_SLOW=1 pytest tests/test_metric_common.py::LocalMetricTest::test_load_metric_<metric_name>
```

Finally additional metric-specific tests can be added to `test_metric_common.py`.

Voila :) Feel free to ping me if you have any question or if I can help

**Is your feature request related to a problem? Please describe.**
Currently there is an absolutely fantastic guide for how to contribute a new dataset to the library. However, there isn't one for adding new metrics.

**Describe the solution you'd like**
I'd like for a guide in a similar style to the dataset guide for adding metrics. I believe many of the content in the dataset guide such as setup can be easily copied over with minimal changes. Also, from what I've seen with existing metrics, it shouldn't be as complicated, especially in documentation of the metric, mainly just citation and usage. The most complicated part I see would be in automated tests that run the new metrics, but y'all's test suite seem pretty comprehensive, so it might not be that hard.

**Describe alternatives you've considered**
One alternative would be just not having the metrics be community generated and so would not need a step by step guide. New metrics would just be proposed as issues and the internal team would take care of them. However, I think it makes more sense to have a step by step guide for contributors to follow.

**Additional context**
I'd be happy to help with creating this guide as I am very interested in adding software engineering metrics to the library :nerd_face:, the part I would need guidance on would be testing.

P.S. Love the library and community y'all have built! :hugs: 

How to Add New Metrics Guide 
 **Is your feature request related to a problem? Please describe.**
Currently there is an absolutely fantastic guide for how to contribute a new dataset to the library. However, there isn't one for adding new metrics.

**Describe the solution you'd like**
I'd like for a guide in a similar style to the dataset guide for adding metrics. I believe many of the content in the dataset guide such as setup can be easily copied over with minimal changes. Also, from what I've seen with existing metrics, it shouldn't be as complicated, especially in documentation of the metric, mainly just citation and usage. The most complicated part I see would be in automated tests that run the new metrics, but y'all's test suite seem pretty comprehensive, so it might not be that hard.

**Describe alternatives you've considered**
One alternative would be just not having the metrics be community generated and so would not need a step by step guide. New metrics would just be proposed as issues and the internal team would take care of them. However, I think it makes more sense to have a step by step guide for contributors to follow.

**Additional context**
I'd be happy to help with creating this guide as I am very interested in adding software engineering metrics to the library :nerd_face:, the part I would need guidance on would be testing.

P.S. Love the library and community y'all have built! :hugs: 
 
 Hi ! sorry for the late response 

It would be fantastic to have a guide for adding metrics as well ! Currently we only have this template here:
https://github.com/huggingface/datasets/blob/master/templates/new_metric_script.py

We can also include test utilities for metrics in the guide.

We have a pytest suite with commands that you can use to make sure your metric works as expected.
It has two useful commands:

1. This commands tests the code in the `Examples:` desction of the docstring of the metric:
```
pytest tests/test_metric_common.py::LocalMetricTest::test_load_metric_<metric_name>
```
This will run this code for example:

https://github.com/huggingface/datasets/blob/e0787aa2a781cc15a80f7597f56d1f12e23df4c9/metrics/accuracy/accuracy.py#L40-L45

Moreover this test is meant to be fast so users are free to add patches to the metric to avoid intensive computations.
And example of intensive call patch can be found here:

https://github.com/huggingface/datasets/blob/e0787aa2a781cc15a80f7597f56d1f12e23df4c9/tests/test_metric_common.py#L138-L151

2. This test runs the same thing as 1. except that it doesn't use patches (the real metric is used):
```
RUN_SLOW=1 pytest tests/test_metric_common.py::LocalMetricTest::test_load_metric_<metric_name>
```

Finally additional metric-specific tests can be added to `test_metric_common.py`.

Voila :) Feel free to ping me if you have any question or if I can help
