Concatenate datasets
{'login': 'jarednielsen', 'id': 4564897, 'node_id': 'MDQ6VXNlcjQ1NjQ4OTc=', 'avatar_url': 'https://avatars.githubusercontent.com/u/4564897?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/jarednielsen', 'html_url': 'https://github.com/jarednielsen', 'followers_url': 'https://api.github.com/users/jarednielsen/followers', 'following_url': 'https://api.github.com/users/jarednielsen/following{/other_user}', 'gists_url': 'https://api.github.com/users/jarednielsen/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/jarednielsen/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/jarednielsen/subscriptions', 'organizations_url': 'https://api.github.com/users/jarednielsen/orgs', 'repos_url': 'https://api.github.com/users/jarednielsen/repos', 'events_url': 'https://api.github.com/users/jarednielsen/events{/privacy}', 'received_events_url': 'https://api.github.com/users/jarednielsen/received_events', 'type': 'User', 'site_admin': False}
[]
[]
['Looks cool :)\r\n\r\nI feel like \r\n```python\r\nconcatenated_dataset = dataset1.concatenate(dataset2)\r\n```\r\ncould be more natural. What do you think ?\r\n\r\nAlso could you also concatenate the `nlp.Dataset._data_files` ?\r\n```python\r\nreturn cls(table, info=info, split=split, data_files=self._data_files + other_dataset._data_files)\r\n```'
 'I feel like "WikiBooks" would be a multi task dataset that could fit in the #217 discussion.\r\nNot sure concatenate should be the solution for a multi task dataset.'
 'Thanks for the suggestion! `dset1.concatenate(dset2)` does feel more natural. Although this seems to be a different "class" of transformation function than map() or filter(), acting on two datasets rather than on one. I would prefer the function signature treat both datasets symmetrically.\r\n\r\nPython lists have `list1 + list2` or `list1.extend(list2)`.\r\nNumPy has `np.concatenate((arr1, arr2))`.\r\nPandas has `pd.join((df1, df2))`.\r\nPyTorch has `ConcatDataset((dset1, dset2))`.\r\n\r\nGiven the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?'
 "The multi-task discussion is interesting, thanks for pointing me to that! I'll be focusing on T5 in a few weeks, so I'm sure I'll have many opinions then :). For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text."
 '> Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?\r\n\r\nYep I like this idea. Maybe `nlp.concatenate_datasets()` ?\r\n\r\n> For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.\r\n\r\nI agree :)'
 'Great, just updated!']
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/390', 'html_url': 'https://github.com/huggingface/datasets/pull/390', 'diff_url': 'https://github.com/huggingface/datasets/pull/390.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/390.patch', 'merged_at': datetime.datetime(2020, 7, 22, 9, 49, 58)}
I'm constructing the "WikiBooks" dataset, which is a concatenation of Wikipedia & BookCorpus. So I implemented the `Dataset.from_concat()` method, which concatenates two datasets with the same schema.

This would also be useful if someone wants to pretrain on a large generic dataset + their own custom dataset. Not in love with the method name, so would love to hear suggestions.

Usage:
```python
from nlp import Dataset, load_dataset

data1, data2 = {"id": [0, 1, 2]}, {"id": [3, 4, 5]}
dset1, dset2 = Dataset.from_dict(data1), Dataset.from_dict(data2)
dset_concat = Dataset.from_concat([dset1, dset2])
print(dset_concat)
# Dataset(schema: {'id': 'int64'}, num_rows: 6)
```
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/390/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}