Added method to remove excess nesting in a DatasetDict
{'login': 'CakeCrusher', 'id': 37946988, 'node_id': 'MDQ6VXNlcjM3OTQ2OTg4', 'avatar_url': 'https://avatars.githubusercontent.com/u/37946988?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/CakeCrusher', 'html_url': 'https://github.com/CakeCrusher', 'followers_url': 'https://api.github.com/users/CakeCrusher/followers', 'following_url': 'https://api.github.com/users/CakeCrusher/following{/other_user}', 'gists_url': 'https://api.github.com/users/CakeCrusher/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/CakeCrusher/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/CakeCrusher/subscriptions', 'organizations_url': 'https://api.github.com/users/CakeCrusher/orgs', 'repos_url': 'https://api.github.com/users/CakeCrusher/repos', 'events_url': 'https://api.github.com/users/CakeCrusher/events{/privacy}', 'received_events_url': 'https://api.github.com/users/CakeCrusher/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Hi ! I think the issue you linked is closed and suggests to use `remove_columns`.\r\n\r\nMoreover if you end up with a dataset with an unnecessarily nested data, please modify your processing functions to not output nested data, or use `map(..., batched=True)` if you function take batches as input'
 "Hi @lhoestq , you are right about the issues this pull has steered beyond that issue. I created this [colab notebook](https://colab.research.google.com/drive/16aLu6QrDSV_aUYRdpufl5E4iS08qkUGj?usp=sharing) to present the error. I tried using batch and that won't resolve it either. I'm looking into that error right now."
 'I think you just need to pass one example at a time to your tokenizer, this way you don\'t end up with nested data:\r\n```python\r\n\r\ndef preprocessFunction(row):\r\n  collatedContext = tokenizer.eos_token.join([row["context"+str(i+1)] for i in range(int(AMT_OF_CONTEXT))])\r\n  response = row["response"]\r\n  tokenizedContext = tokenizer(\r\n    collatedContext, max_length=max_context_length, truncation=True  # don\'t pass as a list here\r\n  )\r\n  with tokenizer.as_target_tokenizer():\r\n    tokenized_response = tokenizer(\r\n      response, max_length=max_response_length, truncation=True  # don\'t pass a a list here\r\n    )\r\n  tokenizedContext["labels"] = tokenized_response["input_ids"]\r\n  return tokenizedContext\r\n```'
 'Yes that is correct, the purpose of this pull is to advise of a more general solution like with `def remove_excess_nesting(self)` or maybe automate the solution (stas00 advised not to automate it as it could "not be backwards compatible").'
 "I'm not sure I understand how having `remove_excess_nesting` would make more sense than just fixing the preprocessFunction to simply not return nested samples, can you elaborate ?"
 'Figuring out the issue can be a bit difficult to figure out. Only until I added batch does it make a little more sense with the error\r\n\r\n> sequence item 0: expected str instance, list found\r\n\r\nbut batch was never intended.\r\n\r\nWhen you run the colab you will notice that only until collating do you learn there is this error. So i figured it would be better to address it during at the `DatasetDict` level.\r\nI think it would be ideal if the user could be notified at the preprocess function.'
 "I'm not arguing that `remove_excess_nesting` is the right solution but what I aim to address is dealing with unnecessary nesting as early as possible."
 "> When you run the colab you will notice that only until collating do you learn there is this error.\r\n\r\nI think users can just check the `dataset.features` and they would notice that the data are nested\r\n```python\r\n{\r\n   'input_ids': Sequence(Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), length=-1, id=None)\r\n   ...\r\n}\r\n```\r\n\r\nSometime nested data are intentional, so you can't know in advance if it's a user's mistake or something planned."
 'Yes, I understand, it could be intentional and only the collator has problems with it. So, it is not worth handling it any differently in any other non-erroneous data. \r\n\r\nThat being said do you think there is any use for the `remove_excess_nesting` method? Or maybe it should be applied in a different way? If not feel free to close this PR. '
 "I think users can write it and use `map` themselves if needed, it is pretty straightforward to implement.\r\n\r\nI'm closing this PR if you don't mind, and thank you for the discussion :)"
 'No problem @lhoestq , thanks for walking me through it.']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/4679', 'html_url': 'https://github.com/huggingface/datasets/pull/4679', 'diff_url': 'https://github.com/huggingface/datasets/pull/4679.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/4679.patch', 'merged_at': None}
Added the ability for a DatasetDict to remove additional nested layers within its features to avoid conflicts when collating. It is meant to accompany [this PR](https://github.com/huggingface/transformers/pull/18119) to resolve the same issue [#15505](https://github.com/huggingface/transformers/issues/15505).

@stas00 @lhoestq 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/4679/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}