ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures
{'login': 'timothyjlaurent', 'id': 2000204, 'node_id': 'MDQ6VXNlcjIwMDAyMDQ=', 'avatar_url': 'https://avatars.githubusercontent.com/u/2000204?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/timothyjlaurent', 'html_url': 'https://github.com/timothyjlaurent', 'followers_url': 'https://api.github.com/users/timothyjlaurent/followers', 'following_url': 'https://api.github.com/users/timothyjlaurent/following{/other_user}', 'gists_url': 'https://api.github.com/users/timothyjlaurent/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/timothyjlaurent/subscriptions', 'organizations_url': 'https://api.github.com/users/timothyjlaurent/orgs', 'repos_url': 'https://api.github.com/users/timothyjlaurent/repos', 'events_url': 'https://api.github.com/users/timothyjlaurent/events{/privacy}', 'received_events_url': 'https://api.github.com/users/timothyjlaurent/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi, it depends on what it is in your `dataset_builder.py` file. Can you share it?\r\n\r\nIf you are just loading `json` files, you can also directly use the `json` script (which will find the schema/features from your JSON structure):\r\n\r\n```python\r\nfrom nlp import load_dataset\r\nds = load_dataset("json", data_files=rel_datafiles)\r\n```'
 'The behavior I\'m seeing is from the `json` script. \r\nI hacked this together to overcome the error with the `JSON` dataloader\r\n\r\n```\r\nclass DatasetBuilder(hf_nlp.ArrowBasedBuilder):\r\n    BUILDER_CONFIG_CLASS = BuilderConfig\r\n\r\n    def _info(self):\r\n        return DatasetInfo()\r\n\r\n    def _split_generators(self, dl_manager):\r\n        """ We handle string, list and dicts in datafiles\r\n        """\r\n        if isinstance(self.config.data_files, (str, list, tuple)):\r\n            files = self.config.data_files\r\n            if isinstance(files, str):\r\n                files = [files]\r\n            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]\r\n        splits = []\r\n        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:\r\n            if split_name in self.config.data_files:\r\n                files = self.config.data_files[split_name]\r\n                if isinstance(files, str):\r\n                    files = [files]\r\n                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))\r\n        return splits\r\n\r\n    def _prepare_split(self, split_generator):\r\n        fname = "{}-{}.arrow".format(self.name, split_generator.name)\r\n        fpath = os.path.join(self._cache_dir, fname)\r\n\r\n        writer = ArrowWriter(path=fpath)\r\n\r\n        generator = self._generate_tables(**split_generator.gen_kwargs)\r\n        for key, table in utils.tqdm(generator, unit=" tables", leave=False):\r\n            writer.write_table(table)\r\n        num_examples, num_bytes = writer.finalize()\r\n\r\n        split_generator.split_info.num_examples = num_examples\r\n        split_generator.split_info.num_bytes = num_bytes\r\n        # this is where the error is coming from\r\n        # def parse_schema(schema, schema_dict):\r\n        #     for field in schema:\r\n        #         if pa.types.is_struct(field.type):\r\n        #             schema_dict[field.name] = {}\r\n        #             parse_schema(field.type, schema_dict[field.name])\r\n        #         elif pa.types.is_list(field.type) and pa.types.is_struct(field.type.value_type):\r\n        #             schema_dict[field.name] = {}\r\n        #             parse_schema(field.type.value_type, schema_dict[field.name])\r\n        #         else:\r\n        #             schema_dict[field.name] = Value(str(field.type))\r\n        # \r\n        # parse_schema(writer.schema, features)\r\n        # self.info.features = Features(features)\r\n\r\n    def _generate_tables(self, files):\r\n        for i, file in enumerate(files):\r\n            pa_table = paj.read_json(\r\n                file\r\n            )\r\n            yield i, pa_table\r\n```\r\n\r\nSo I basically just don\'t populate the `self.info.features` though this doesn\'t seem to cause any problems in my downstream applications. \r\n\r\nThe other workaround I was doing was to just use pyarrow.json to build a table and then to create the Dataset with its constructor or from_table methods. `load_dataset` has nice split logic, so I\'d prefer to use that.\r\n\r\n'
 "Also noticed that if you for example in a loader script\r\n\r\n```\r\nfrom nlp import ArrowBasedBuilder\r\n\r\nclass MyBuilder(ArrowBasedBuilder):\r\n...\r\n\r\n```\r\nand use that in the subclass, it will be on the module's __dict__ and will be selected before the `MyBuilder` subclass, and it will raise `NotImplementedError` on its `_generate_examples` method... In the code it check for abstract classes but Builder and ArrowBasedBuilder aren't abstract classes, they're regular classes with `@abstract_methods`."
 'Indeed this is part of a more general limitation which is the fact that we should generate and update the `features` from the auto-inferred Arrow schema when they are not provided (also happen when a user change the schema using `map()`, the features should be auto-generated and guessed as much as possible to keep the `features` synced with the underlying Arrow table schema).\r\n\r\nWe will try to solve this soon.']
I tried using the Json dataloader to load some JSON lines files. but get an exception in the parse_schema function.

```
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-23-9aecfbee53bd> in <module>
     55 from nlp import load_dataset
     56 
---> 57 ds = load_dataset("../text2struct/model/dataset_builder.py", data_files=rel_datafiles)
     58 
     59 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    736                     schema_dict[field.name] = Value(str(field.type))
    737 
--> 738         parse_schema(writer.schema, features)
    739         self.info.features = Features(features)
    740 

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in parse_schema(schema, schema_dict)
    734                     parse_schema(field.type.value_type, schema_dict[field.name])
    735                 else:
--> 736                     schema_dict[field.name] = Value(str(field.type))
    737 
    738         parse_schema(writer.schema, features)

<string> in __init__(self, dtype, id, _type)

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in __post_init__(self)
     55 
     56     def __post_init__(self):
---> 57         self.pa_type = string_to_arrow(self.dtype)
     58 
     59     def __call__(self):

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in string_to_arrow(type_str)
     32         if str(type_str + "_") not in pa.__dict__:
     33             raise ValueError(
---> 34                 f"Neither {type_str} nor {type_str + '_'} seems to be a pyarrow data type. "
     35                 f"Please make sure to use a correct data type, see: "
     36                 f"https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions"

ValueError: Neither list<item: string> nor list<item: string>_ seems to be a pyarrow data type. Please make sure to use a correct data type, see: https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions
```

If I create the dataset imperatively, using a pyarrow table, the dataset is created correctly. If I override the `_prepare_split` method to avoid calling the validate schema, the dataset can load as well. 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/359/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}