[Question] Combine 2 datasets which have the same columns
{'login': 'airKlizz', 'id': 25703835, 'node_id': 'MDQ6VXNlcjI1NzAzODM1', 'avatar_url': 'https://avatars.githubusercontent.com/u/25703835?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/airKlizz', 'html_url': 'https://github.com/airKlizz', 'followers_url': 'https://api.github.com/users/airKlizz/followers', 'following_url': 'https://api.github.com/users/airKlizz/following{/other_user}', 'gists_url': 'https://api.github.com/users/airKlizz/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/airKlizz/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/airKlizz/subscriptions', 'organizations_url': 'https://api.github.com/users/airKlizz/orgs', 'repos_url': 'https://api.github.com/users/airKlizz/repos', 'events_url': 'https://api.github.com/users/airKlizz/events{/privacy}', 'received_events_url': 'https://api.github.com/users/airKlizz/received_events', 'type': 'User', 'site_admin': False}
[]
[]
We are thinking about ways to combine datasets for T5 in #217, feel free to share your thoughts about this. Ok great! I will look at it. Thanks
Hi,

I am using ``nlp`` to load personal datasets. I created summarization datasets in multi-languages based on wikinews. I have one dataset for english and one for german (french is getting to be ready as well). I want to keep these datasets independent because they need different pre-processing (add different task-specific prefixes for T5 : *summarize:* for english and *zusammenfassen:* for german)

My issue is that I want to train T5 on the combined english and german datasets to see if it improves results. So I would like to combine 2 datasets (which have the same columns) to make one and train T5 on it. I was wondering if there is a proper way to do it? I assume that it can be done by combining all examples of each dataset but maybe you have a better solution.

Hoping this is clear enough,

Thanks a lot 😊
Best
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/206/reactions', 'total_count': 0, '+1': 0, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 0, 'eyes': 0}