Task casting for text classification & question answering
{'login': 'SBrandeis', 'id': 33657802, 'node_id': 'MDQ6VXNlcjMzNjU3ODAy', 'avatar_url': 'https://avatars.githubusercontent.com/u/33657802?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/SBrandeis', 'html_url': 'https://github.com/SBrandeis', 'followers_url': 'https://api.github.com/users/SBrandeis/followers', 'following_url': 'https://api.github.com/users/SBrandeis/following{/other_user}', 'gists_url': 'https://api.github.com/users/SBrandeis/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/SBrandeis/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/SBrandeis/subscriptions', 'organizations_url': 'https://api.github.com/users/SBrandeis/orgs', 'repos_url': 'https://api.github.com/users/SBrandeis/repos', 'events_url': 'https://api.github.com/users/SBrandeis/events{/privacy}', 'received_events_url': 'https://api.github.com/users/SBrandeis/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['cc @abhi1thakur '
 "Looks really nice so far, thanks !\r\nMaybe if a dataset doesn't have a template for a specific task we could try the default template of this task ?"
 'hey @SBrandeis @lhoestq,\r\n\r\ni now have a better idea about what you guys are trying to achieve with the task templates and have a few follow-up questions:\r\n\r\n1. how did you envision using `DatasetInfo` for running evaluation? my understanding is that all `dataset_infos.json` files are stored in the `datasets` repo (unlike `transformers` where each model\'s weights etc are stored in a dedicated repo). \r\nthis suggests the following workflow:\r\n\r\n```\r\n- git clone datasets\r\n- load target dataset to evaluate\r\n- load `dataset_infos.json` for target dataset\r\n- run eval for each task template in `task_templates`\r\n- store metrics as evaluation cards (similar to what is done in `autonlp`)\r\n```\r\n2. assuming the above workflow, i see that the current `TaskTemplate` attributes of `task`, `input_schema`, and `label_schema` still require some wrangling from `dataset_infos.json` to reproduce additional mappings like `label2id` that we\'d need for e.g. text classification. an alternative would be to instantiate the task template class directly from the JSON with something like\r\n```python\r\nfrom datasets.tasks import TextClassification\r\nfrom transformers import AutoModelForSequenceClassification, AutoConfig\r\n\r\ntc = TextClassification.from_json("path/to/dataset_infos.json")\r\n# load a model with the desired config\r\nmodel_ckpt = ...\r\nconfig = AutoConfig.from_pretrained(model_ckpt, label2id=tc.label2id, id2label=tc.id2label)\r\nmodel = AutoModelForSequenceClassification.from_pretrained(model_ckpt, config=config)\r\n# run eval ...\r\n```\r\nperhaps this is what @SBrandeis had in mind with the `TaskTemplate.from_dict` method?\r\n\r\n3. i personally prefer using `task_templates` over `supervised_keys` because it encourages the contributor to think in terms of 1 or more tasks. my question here is do we currently use `supervised_keys` for anything important in the `datasets` library?'
 '1. How do you envision using DatasetInfo for running evaluation?\r\n\r\nThe initial idea was to be able to do something like this:\r\n```python\r\nfrom datasets import load_dataset\r\ndset = load_dataset("name", task="binary_classification")\r\n# OR\r\ndset = load_dataset("name")\r\ndset = dset.prepare_for_task("binary_classification")\r\n```\r\n\r\n2. I don\'t think that\'s needed if we proceed as mentioned above\r\n\r\n3. `supervised_keys` are mostly a legacy compatibility thing with TF datasets, not sure it\'s used for anything right now. I\'ll let @lhoestq give more details on that\r\n\r\n[Edit 1] Typo'
 '> The initial idea was to be able to do something like this:\r\n> \r\n> ```python\r\n> from datasets import load_dataset\r\n> dset = load_dataset("name", task="binary_classification")\r\n> # OR\r\n> dset = load_dataset("name")\r\n> dset = dset.prepare_for_task("binary_classification")\r\n> ```\r\n\r\nah that\'s very elegant! just so i\'ve completely understood, the result would be that the relevant column names of `dset` would be mapped to e.g. `text` and `label` and thus we\'d have a uniform schema for the evaluation of all `binary_classification` tasks?'
 'That\'s correct! Also, the features need to be appropriately casted\r\nFor a classification task for example, we would need to cast the datasets features to something like this:\r\n```python\r\ndatasets.Features({\r\n    "text": datasets.Value("string"),\r\n    "label": datasets.ClassLabel(names=[...]),\r\n})\r\n```\r\n'
 "3. We can ignore `supervised_keys` (it came from TFDS and we're not using it) and use `task_templates`"
 "great, thanks a lot for your answers! now it's much clearer what i need to do next 😃 "
 'hey @lhoestq @SBrandeis, \r\n\r\ni\'ve made some small tweaks to @SBrandeis\'s code so that `Dataset.prepare_for_task` is called in `DatasetBuilder`. using the `emotion` dataset as a test case, the following now works:\r\n\r\n ```python\r\n# DatasetDict with default columns\r\nds = load_dataset("./datasets/emotion/")\r\n# DatasetDict({\r\n#     train: Dataset({\r\n#         features: [\'tweet\', \'emotion\'],\r\n#         num_rows: 16000\r\n#     })\r\n#     validation: Dataset({\r\n#         features: [\'tweet\', \'emotion\'],\r\n#         num_rows: 2000\r\n#     })\r\n#     test: Dataset({\r\n#         features: [\'tweet\', \'emotion\'],\r\n#         num_rows: 2000\r\n#     })\r\n# })\r\n\r\n# DatasetDict with remapped columns\r\nds = load_dataset("./datasets/emotion/", task="text_classification")\r\nDatasetDict({\r\n#     train: Dataset({\r\n#         features: [\'text\', \'label\'],\r\n#         num_rows: 16000\r\n#     })\r\n#     validation: Dataset({\r\n#         features: [\'text\', \'label\'],\r\n#         num_rows: 2000\r\n#     })\r\n#     test: Dataset({\r\n#         features: [\'text\', \'label\'],\r\n#         num_rows: 2000\r\n#     })\r\n# })\r\n\r\n# Dataset with default columns\r\nds = load_dataset("./datasets/emotion/", split="train")\r\n# Map/cast features\r\nds = ds.prepare_for_task("text_classification")\r\n# Dataset({\r\n#     features: [\'text\', \'label\'],\r\n#     num_rows: 16000\r\n# })\r\n```\r\n\r\ni have a few follow-up questions / remarks:\r\n\r\n1. i\'m working under the assumption that contributors / users only provide a unique set of task types. in particular, the current implementation does not support something like:\r\n```python\r\ntask_templates=[TextClassification(labels=class_names, text_column="tweet", label_column="emotion"), TextClassification(labels=class_names, text_column="some_other_column", label_column="some_other_column")]\r\n```\r\nsince we use `TaskTemplate.task` and the filter for compatible templates in `Dataset.prepare_for_task`. should we support these scenarios? my hunch is that this is rare in practice, but please correct me if i\'m wrong.\r\n\r\n2. when we eventually run evaluation for `transformers` models, i expect we\'ll be using the `Trainer` for which we can pass the standard label names to `TrainingArguments.label_names`. if that\'s the case, it might be prudent to heed the warning from the [docs](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#trainer) and use `labels` instead of `label` in the schema:\r\n> your model can accept multiple label arguments (use the label_names in your TrainingArguments to indicate their name to the Trainer) but none of them should be named "label".\r\n\r\n3. i plan to forge ahead on the rest of the pipeline taxonomy. please let me know if you\'d prefer smaller, self-contained pull requests (e.g. one per task)'
 'hey @lhoestq @SBrandeis, i think this is ready for another review 😃 \r\n\r\nin addition to a few comments / questions i\'ve left in the pr, here\'s a few remarks:\r\n\r\n1. after some experimentation, i decided against allowing the user to specify nested column names for question-answering. i couldn\'t find a simple solution with the current api and suspect that i\'d have to touch many areas of `datasets` to "unflatten" columns in a generic fashion.\r\n2. in the current implementation, the user can specify the outer column name for question-answering, but is expected to follow the inner schema for e.g. `answers.text` and `answers.answer_start`. we can decide later how much flexibility we want to give users\r\n3. i added a few unit tests\r\n4. as discussed, let\'s keep this pr focused on text classification / question answering and i\'ll add the other tasks in separate prs\r\n5. i renamed the tasks e.g. `text_classification` -> `text-classification` for consistency with the `Trainer` model cards [here](https://github.com/huggingface/transformers/pull/11599#pullrequestreview-656371007).'
 "i'm not sure why the benchmarks are getting cancelled - is this expected?"
 "> i'm not sure why the benchmarks are getting cancelled - is this expected?\r\n\r\nHmm I don't know. It's certainly unrelated to this PR though. Maybe github has some issues"
 'Something is happening with actions: https://www.githubstatus.com/'
 "hey @lhoestq and @SBrandeis, i've: \r\n\r\n* extended the `prepare_for_task` API along the lines that @lhoestq suggested. i wasn't entirely sure what the `datasets` convention is for docstrings with mixed types, so please see if my proposal makes sense\r\n* added a few new tests to check that we trigger the value errors on incorrect input\r\n\r\ni think this is ready for another review :)"
 '> Looks all good thank you :)\r\n> \r\n> Can you also add `prepare_for_task` in the `main_classes.rst` file of the documentation ?\r\n\r\nDone! I also remembered that I needed to do the same for `DatasetDict`, so included this as well :)']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/2255', 'html_url': 'https://github.com/huggingface/datasets/pull/2255', 'diff_url': 'https://github.com/huggingface/datasets/pull/2255.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/2255.patch', 'merged_at': datetime.datetime(2021, 5, 18, 13, 31, 35)}
This PR implements task preparation for a given task, in the continuation of #2143 

Task taxonomy follows 🤗 Transformers's pipelines taxonomy: https://github.com/huggingface/transformers/tree/master/src/transformers/pipelines

Edit by @lewtun:

This PR implements support for the following tasks:

* `text-classification`
* `question-answering`

The intended usage is as follows:

```python
# Load a dataset with default column names / features
ds = load_dataset("dataset_name")
# Cast column names / features to schema. Casting is defined in the dataset's `DatasetInfo`
ds = ds.prepare_for_task(task="text-classification")
# Casting can also be realised during load
ds = load_dataset("dataset_name", task="text-classification")
# We can also combine shared tasks across dataset concatenation
ds1 = load_dataset("dataset_name_1", task="text-classification")
ds2 = load_dataset("dataset_name_2", task="text-classification")
# If the tasks have the same schema, so will `ds_concat`
ds_concat = concatenate_datasets([ds1, ds2])
```

Note that the current implementation assumes that `DatasetInfo.task_templates` has been pre-defined by the user / contributor when overriding the `MyDataset(GeneratorBasedBuilder)._info` function.

As pointed out by @SBrandeis, for evaluation we'll need a way to detect which datasets are already have a compatible schema so we don't have to edit hundreds of dataset scripts. One possibility is to check if the schema features are a subset of the dataset ones, e.g.

```python
squad = load_dataset("./datasets/squad", split="train")
qa = QuestionAnswering()
schema = Features({**qa.input_schema, **qa.label_schema})
assert all(item in squad.features.items() for item in schema.items())
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/2255/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}