Fork dataset
{'login': 'timothyjlaurent', 'id': 2000204, 'node_id': 'MDQ6VXNlcjIwMDAyMDQ=', 'avatar_url': 'https://avatars.githubusercontent.com/u/2000204?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/timothyjlaurent', 'html_url': 'https://github.com/timothyjlaurent', 'followers_url': 'https://api.github.com/users/timothyjlaurent/followers', 'following_url': 'https://api.github.com/users/timothyjlaurent/following{/other_user}', 'gists_url': 'https://api.github.com/users/timothyjlaurent/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/timothyjlaurent/subscriptions', 'organizations_url': 'https://api.github.com/users/timothyjlaurent/orgs', 'repos_url': 'https://api.github.com/users/timothyjlaurent/repos', 'events_url': 'https://api.github.com/users/timothyjlaurent/events{/privacy}', 'received_events_url': 'https://api.github.com/users/timothyjlaurent/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['To be able to generate the Arrow dataset you need to either use our csv or json utilities `load_dataset("json", data_files=my_json_files)` OR write your own custom dataset script (you can find some inspiration from the [squad](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py) script for example). Custom dataset scripts can be called locally with `nlp.load_dataset(path_to_my_script_directory)`.\r\n\r\nThis should help you get what you call "Dataset1".\r\n\r\nThen using some dataset transforms like `.map` for example you can get to "DatasetNER" and "DatasetREL".\r\n'
 "Thanks for the helpful advice, @lhoestq  -- I wasn't quite able to get the json recipe working - \r\n\r\n```\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.py in __init__(self, source)\r\n     60 \r\n     61     def __init__(self, source):\r\n---> 62         self._open(source)\r\n     63 \r\n     64 \r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()\r\nArrowInvalid: Tried reading schema message, was null or length 0\r\n```\r\n\r\nBut I'm going to give the generator_dataset_builder a try.\r\n\r\n1 more quick question -- can .map be used to output different length mappings -- could I skip one, or yield 2, can you map_batch  "
 'You can use `.map(my_func, batched=True)` and return less examples, or more examples if you want'
 'Thanks this answers my question. I think the issue I was having using the json loader were due to using gzipped jsonl files.\r\n\r\nThe error I get now is :\r\n\r\n```\r\n\r\nUsing custom data configuration test\r\n---------------------------------------------------------------------------\r\n\r\nValueError                                Traceback (most recent call last)\r\n\r\n<ipython-input-38-29082a31e5b2> in <module>\r\n      5 print(ner_datafiles)\r\n      6 \r\n----> 7 ds = nlp.load_dataset("json", "test",  data_files=ner_datafiles[0])\r\n      8 \r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)\r\n    522         download_mode=download_mode,\r\n    523         ignore_verifications=ignore_verifications,\r\n--> 524         save_infos=save_infos,\r\n    525     )\r\n    526 \r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)\r\n    430                 verify_infos = not save_infos and not ignore_verifications\r\n    431                 self._download_and_prepare(\r\n--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs\r\n    433                 )\r\n    434                 # Sync info\r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)\r\n    481             try:\r\n    482                 # Prepare split will record examples associated to the split\r\n--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)\r\n    484             except OSError:\r\n    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))\r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _prepare_split(self, split_generator)\r\n    736                     schema_dict[field.name] = Value(str(field.type))\r\n    737 \r\n--> 738         parse_schema(writer.schema, features)\r\n    739         self.info.features = Features(features)\r\n    740 \r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in parse_schema(schema, schema_dict)\r\n    734                     parse_schema(field.type.value_type, schema_dict[field.name])\r\n    735                 else:\r\n--> 736                     schema_dict[field.name] = Value(str(field.type))\r\n    737 \r\n    738         parse_schema(writer.schema, features)\r\n\r\n<string> in __init__(self, dtype, id, _type)\r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in __post_init__(self)\r\n     55 \r\n     56     def __post_init__(self):\r\n---> 57         self.pa_type = string_to_arrow(self.dtype)\r\n     58 \r\n     59     def __call__(self):\r\n\r\n~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in string_to_arrow(type_str)\r\n     32         if str(type_str + "_") not in pa.__dict__:\r\n     33             raise ValueError(\r\n---> 34                 f"Neither {type_str} nor {type_str + \'_\'} seems to be a pyarrow data type. "\r\n     35                 f"Please make sure to use a correct data type, see: "\r\n     36                 f"https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions"\r\n\r\nValueError: Neither list<item: int64> nor list<item: int64>_ seems to be a pyarrow data type. Please make sure to use a correct data type, see: https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions.\r\n```\r\n\r\nIf I just create a pa- table manually like is done in the jsonloader -- it seems to work fine. Ths JSON I\'m trying to load isn\'t overly complex -  1 integer field, the rest text fields with a nested list of objects with text fields .'
 "I'll close this -- It's still unclear how to go about troubleshooting the json example as I mentioned above. If I decide it's worth the trouble, I'll create another issue, or wait for a better support for using nlp for making custom data-loaders."]
We have a multi-task learning model training I'm trying to convert to using the Arrow-based nlp dataset. 

We're currently training a custom TensorFlow model but the nlp paradigm should be a bridge for us to be able to use the wealth of pre-trained models in Transformers.

Our preprocessing flow parses raw text and json with Entity and Relations annotations and creates 2 datasets for training a NER and Relations prediction heads.

Is there some good way to "fork" dataset-

EG

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 -> DatasetREL

or 

1. text + json -> Dataset1
1. Dataset1 -> DatasetNER
1. Dataset1 + DatasetNER -> DatasetREL


{'url': 'https://api.github.com/repos/huggingface/datasets/issues/328/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}