How to reuse functionality of a (generic) dataset?
{'login': 'ArneBinder', 'id': 3375489, 'node_id': 'MDQ6VXNlcjMzNzU0ODk=', 'avatar_url': 'https://avatars.githubusercontent.com/u/3375489?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/ArneBinder', 'html_url': 'https://github.com/ArneBinder', 'followers_url': 'https://api.github.com/users/ArneBinder/followers', 'following_url': 'https://api.github.com/users/ArneBinder/following{/other_user}', 'gists_url': 'https://api.github.com/users/ArneBinder/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/ArneBinder/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/ArneBinder/subscriptions', 'organizations_url': 'https://api.github.com/users/ArneBinder/orgs', 'repos_url': 'https://api.github.com/users/ArneBinder/repos', 'events_url': 'https://api.github.com/users/ArneBinder/events{/privacy}', 'received_events_url': 'https://api.github.com/users/ArneBinder/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi @ArneBinder, we have a few "generic" datasets which are intended to load data files with a predefined format:\r\n- csv: https://github.com/huggingface/nlp/tree/master/datasets/csv\r\n- json: https://github.com/huggingface/nlp/tree/master/datasets/json\r\n- text: https://github.com/huggingface/nlp/tree/master/datasets/text\r\n\r\nYou can find more details about this way to load datasets here in the documentation: https://huggingface.co/nlp/loading_datasets.html#from-local-files\r\n\r\nMaybe your brat loading script could be shared in a similar fashion?'
 '> Maybe your brat loading script could be shared in a similar fashion?\r\n\r\n@thomwolf that was also my first idea and I think I will tackle that in the next days. I separated the code and created a real abstract class `AbstractBrat` to allow to inherit from that (I\'ve just seen that the dataset_loader loads the first non abstract class), now `Brat` is very similar in its functionality to https://github.com/huggingface/nlp/tree/master/datasets/text but inherits from `AbstractBrat`.\r\n\r\nHowever, it is still not clear to me how to add a specific dataset (as explained in https://huggingface.co/nlp/add_dataset.html) to your repo that uses this format/abstract class, i.e. re-using the `features` entry of the  `DatasetInfo` object and `_generate_examples()`. Again, by doing so, the only remaining entries/functions to define would be `_DESCRIPTION`, `_CITATION`, `homepage` and `_URL` (which is all copy-paste stuff) and `_split_generators()`.\r\n \r\nIn a lack of better ideas, I tried sth like below, but of course it does not work outside `nlp` (`AbstractBrat` is currently defined in [datasets/brat.py](https://github.com/ArneBinder/nlp/blob/5e81fb8710546ee7be3353a7f02a3045e9a8351e/datasets/brat/brat.py)):\r\n```python\r\nfrom __future__ import absolute_import, division, print_function\r\n\r\nimport os\r\n\r\nimport nlp\r\n\r\nfrom datasets.brat.brat import AbstractBrat\r\n\r\n_CITATION = """\r\n@inproceedings{lauscher2018b,\r\n  title = {An argument-annotated corpus of scientific publications},\r\n  booktitle = {Proceedings of the 5th Workshop on Mining Argumentation},\r\n  publisher = {Association for Computational Linguistics},\r\n  author = {Lauscher, Anne and Glava\\v{s}, Goran and Ponzetto, Simone Paolo},\r\n  address = {Brussels, Belgium},\r\n  year = {2018},\r\n  pages = {40–46}\r\n}\r\n"""\r\n\r\n_DESCRIPTION = """\\\r\nThis dataset is an extension of the Dr. Inventor corpus (Fisas et al., 2015, 2016) with an annotation layer containing \r\nfine-grained argumentative components and relations. It is the first argument-annotated corpus of scientific \r\npublications (in English), which allows for joint analyses of argumentation and other rhetorical dimensions of \r\nscientific writing.\r\n"""\r\n\r\n_URL = "http://data.dws.informatik.uni-mannheim.de/sci-arg/compiled_corpus.zip"\r\n\r\n\r\nclass Sciarg(AbstractBrat):\r\n\r\n    VERSION = nlp.Version("1.0.0")\r\n\r\n    def _info(self):\r\n\r\n        brat_features = super()._info().features\r\n        return nlp.DatasetInfo(\r\n            # This is the description that will appear on the datasets page.\r\n            description=_DESCRIPTION,\r\n            # nlp.features.FeatureConnectors\r\n            features=brat_features,\r\n            # If there\'s a common (input, target) tuple from the features,\r\n            # specify them here. They\'ll be used if as_supervised=True in\r\n            # builder.as_dataset.\r\n            #supervised_keys=None,\r\n            # Homepage of the dataset for documentation\r\n            homepage="https://github.com/anlausch/ArguminSci",\r\n            citation=_CITATION,\r\n        )\r\n\r\n    def _split_generators(self, dl_manager):\r\n        """Returns SplitGenerators."""\r\n        # TODO: Downloads the data and defines the splits\r\n        # dl_manager is a nlp.download.DownloadManager that can be used to\r\n        # download and extract URLs\r\n        dl_dir = dl_manager.download_and_extract(_URL)\r\n        data_dir = os.path.join(dl_dir, "compiled_corpus")\r\n        print(f\'data_dir: {data_dir}\')\r\n        return [\r\n            nlp.SplitGenerator(\r\n                name=nlp.Split.TRAIN,\r\n                # These kwargs will be passed to _generate_examples\r\n                gen_kwargs={\r\n                    "directory": data_dir,\r\n                },\r\n            ),\r\n        ]\r\n```  \r\n\r\nNevertheless, many thanks for tackling the dataset accessibility problem with this great library!'
 "As temporary fix I've created [ArneBinder/nlp-formats](https://github.com/ArneBinder/nlp-formats) (contributions welcome)."]
I have written a generic dataset for corpora created with the Brat annotation tool ([specification](https://brat.nlplab.org/standoff.html), [dataset code](https://github.com/ArneBinder/nlp/blob/brat/datasets/brat/brat.py)). Now I wonder how to use that to create specific dataset instances. What's the recommended way to reuse formats and loading functionality for datasets with a common format?

In my case, it took a bit of time to create the Brat dataset and I think others would appreciate to not have to think about that again. Also, I assume there are other formats (e.g. conll) that are widely used, so having this would really ease dataset onboarding and adoption of the library.
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/433/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}