added metrics for CUAD
{'login': 'bhavitvyamalik', 'id': 19718818, 'node_id': 'MDQ6VXNlcjE5NzE4ODE4', 'avatar_url': 'https://avatars.githubusercontent.com/u/19718818?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/bhavitvyamalik', 'html_url': 'https://github.com/bhavitvyamalik', 'followers_url': 'https://api.github.com/users/bhavitvyamalik/followers', 'following_url': 'https://api.github.com/users/bhavitvyamalik/following{/other_user}', 'gists_url': 'https://api.github.com/users/bhavitvyamalik/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/bhavitvyamalik/subscriptions', 'organizations_url': 'https://api.github.com/users/bhavitvyamalik/orgs', 'repos_url': 'https://api.github.com/users/bhavitvyamalik/repos', 'events_url': 'https://api.github.com/users/bhavitvyamalik/events{/privacy}', 'received_events_url': 'https://api.github.com/users/bhavitvyamalik/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["> For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here\r\n\r\n@bhavitvyamalik  I guess the mentioned metrics are enough but it would be better if exact match is also added since the standard SQUAD dataset also has it."
 "I would like to quote it from the website that I am following to learn\nthese things.\nExact Match:\nThis metric is as simple as it sounds. For each question+answer pair, if\nthe characters of the model's prediction exactly match the characters of\n*(one\nof) the True Answer(s)*, EM = 1, otherwise EM = 0. This is a strict\nall-or-nothing metric; being off by a single character results in a score\nof 0. When assessing against a negative example, if the model predicts any\ntext at all, it automatically receives a 0 for that example.\n\nSo, I guess you need to ensure at least 1 predicted answer matches for EM\nto be 1.\nSource:\nhttps://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html\n\nYou can go to their homepage and read the other links. They have detailed\nexplanations on evaluation metrics. You can also have a look at the\nsquad_v2 metric file for further clarification.\n\nRegards,\nMohammed Rakib\n\nOn Sun, 25 Apr 2021 at 15:20, Bhavitvya Malik ***@***.***>\nwrote:\n\n> I'm a little confused when it comes to 2 ground truths which can be a\n> possible answer. Like here for eg.\n>\n> predictions = [{'prediction_text': ['The seller:', 'The buyer/End-User:\n> Shenzhen LOHAS Supply Chain Management Co., Ltd.'], 'id':\n> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply\n> Agreement__Parties'}]\n>\n> references = [{'answers': {'answer_start': [143, 49], 'text': ['The\n> seller:', 'The buyer/End-User: Shenzhen LOHAS Supply Chain Management Co.,\n> Ltd.']}, 'id':\n> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply\n> Agreement__Parties'}]\n>\n> Should I ensure at least 1 predicted answer matches or both predicted\n> answers should match (like in this case) for EM to be 1?\n>\n> —\n> You are receiving this because you commented.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/huggingface/datasets/pull/2257#issuecomment-826289753>,\n> or unsubscribe\n> <https://github.com/notifications/unsubscribe-auth/AHMYZAZSAEZNFWEMVAPK6M3TKPNHLANCNFSM43QFZVPQ>\n> .\n>\n"
 "Updated the same @MohammedRakib! Even if a single answer matches I'm returning 1 in that case for EM (not traversing all predictions once we have one `exact_match` from prediction)"]
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/2257', 'html_url': 'https://github.com/huggingface/datasets/pull/2257', 'diff_url': 'https://github.com/huggingface/datasets/pull/2257.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/2257.patch', 'merged_at': None}
For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2257/reactions', 'total_count': 1, '+1': 1, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}