Add open book corpus
{'login': 'vblagoje', 'id': 458335, 'node_id': 'MDQ6VXNlcjQ1ODMzNQ==', 'avatar_url': 'https://avatars.githubusercontent.com/u/458335?v=4', 'gravatar_id': '', 'url': 'https://api.github.com/users/vblagoje', 'html_url': 'https://github.com/vblagoje', 'followers_url': 'https://api.github.com/users/vblagoje/followers', 'following_url': 'https://api.github.com/users/vblagoje/following{/other_user}', 'gists_url': 'https://api.github.com/users/vblagoje/gists{/gist_id}', 'starred_url': 'https://api.github.com/users/vblagoje/starred{/owner}{/repo}', 'subscriptions_url': 'https://api.github.com/users/vblagoje/subscriptions', 'organizations_url': 'https://api.github.com/users/vblagoje/orgs', 'repos_url': 'https://api.github.com/users/vblagoje/repos', 'events_url': 'https://api.github.com/users/vblagoje/events{/privacy}', 'received_events_url': 'https://api.github.com/users/vblagoje/received_events', 'type': 'User', 'site_admin': False}
[]
[]
["@lhoestq I fixed issues except for the dummy_data zip file. But I think I know why is it happening. So when unzipping dummy_data.zip it gets save in /tmp directory where glob doesn't pick it up. For regular downloads, the archive gets unzipped in ~/.cache/huggingface. Could that be a reason?"
 'Nice thanks :)\r\n\r\nWhen testing with the dummy data, the `download_manager.download_and_extract()` call returns the path to the unzipped dummy_data.zip archive. Therefore glob should be able to find your dummy .epub.txt file'
 '@lhoestq I understand but for some reason, it is not happening. I added logs to see where dummy_data.zip gets unzipped in /tmp but I suppose when the test process finishes that tmp is gone. I also tried to glob anything in _generate_examples from that directory using /* instead of **/*.epub.txt and nothing is being returned. Always an empty array.  '
 'Ok weird ! I can take a look tomorrow if you want'
 'Please do, I will take a fresh look as well. '
 'In _generate_examples_ I wrote the following:\r\n```\r\nglob_target = os.path.join(directory, "**/*.epub.txt")\r\nprint(f"Glob target {glob_target    }")\r\n```\r\n\r\nAnd here is the test failure:\r\n\r\n\r\n========================================================================================== FAILURES ===========================================================================================\r\n________________________________________________________________ LocalDatasetTest.test_load_dataset_all_configs_bookcorpusopen ________________________________________________________________\r\n\r\nself = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_bookcorpusopen>, dataset_name = \'bookcorpusopen\'\r\n\r\n    @slow\r\n    def test_load_dataset_all_configs(self, dataset_name):\r\n        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)\r\n>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)\r\n\r\ntests/test_dataset_common.py:232: \r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\ntests/test_dataset_common.py:193: in check_load_dataset\r\n    self.parent.assertTrue(len(dataset[split]) > 0)\r\nE   AssertionError: False is not true\r\n------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------\r\nDownloading and preparing dataset book_corpus_open/plain_text (download: 1.00 MiB, generated: 1.00 MiB, post-processed: Unknown size, total: 2.00 MiB) to /var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpmuu0_ln2/book_corpus_open/plain_text/1.0.0...\r\nGlob target /var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpm6tpvb3f/extracted/d953b414cceb4fe3985eeaf68aec2f4435f166b2edf66863d805e3825b7d336b/dummy_data/**/*.epub.txt\r\nDataset book_corpus_open downloaded and prepared to /var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpmuu0_ln2/book_corpus_open/plain_text/1.0.0. Subsequent calls will reuse this data.\r\n------------------------------------------------------------------------------------ Captured stderr call -------------------------------------------------------------------------------------\r\n                                \r\n'
 'And when I do os.listdir on the given directory I get:\r\n\r\n        glob_target = os.path.join(directory, "**/*.epub.txt")\r\n        print(f"Glob target {glob_target    }")\r\n>       print(os.listdir(path=directory))\r\nE       FileNotFoundError: [Errno 2] No such file or directory: \'/var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpbu_aom5q/extracted/d953b414cceb4fe3985eeaf68aec2f4435f166b2edf66863d805e3825b7d336b/dummy_data\'\r\n'
 "Thanks for the info, I'm looking at it right now"
 'Ok found the issue !\r\n\r\nThe dummy_data.zip file must be an archive of a folder named dummy_data. Currently the dummy_data.zip is an archive of a folder named book1. In order to have a valid dummy_data.zip file you must first take the dummy book1 folder, place it inside a folder named dummy_data and then compress the dummy_data folder to get dummy_data.zip'
 'Excellent, I am on it @lhoestq '
 "> Awesome thank you so much for adding it :)\r\n\r\nYou're welcome, ok all tests are green now! I needed it asap as well. Thanks for your help @lhoestq ."
 "I just wanted to say thank you to everyone involved in making this happen! I was certain that I would have to add bookcorpusnew myself, but then @vblagoje came along and did it, and @lhoestq gave some great support in a timely fashion.\r\n\r\nBy the way @vblagoje, are you on Twitter? I'm https://twitter.com/theshawwn if you'd like to DM and say hello. Once again, thanks for doing this!\r\n\r\nI'll mention over at https://github.com/soskek/bookcorpus/issues/27 that this was merged."
 'Thank you Shawn. You did all the heavy lifting ;-)'
 "@vblagoje Would you be interested in adding books3 as well? https://twitter.com/theshawwn/status/1320282149329784833\r\n\r\nHuggingface is interested and asked me to add it, but I had a bit of trouble during setup (https://github.com/huggingface/datasets/issues/790) and never got around to it. At this point you have much more experience than I do with the datasets lib.\r\n\r\nIt *seems* like it might simply be a matter of copy-pasting this PR, changing books1 to books3, and possibly trimming off the leading paths -- each book is at e.g. the-eye/Books/Bibliotok/J/Jurassic Park.epub.txt, which is rather lengthy compared to just the filename -- but the full path is probably fine, so feel free to do the least amount of work that gets the job done. Otherwise I suppose I'll get around to it eventually; thanks again!"
 "@shawwn I'll take a look as soon as I clear my work queue. TBH, I would likely work on making sure HF datasets has all the datasets used to train https://github.com/alexa/bort/ and these are: Wikipedia, Wiktionary, OpenWebText (Gokaslan and Cohen, 2019), UrbanDictionary, Onel Billion Words (Chelba et al., 2014), the news subset of Common Crawl (Nagel, 2016)10, and Bookcorpus. cc @lhoestq "]
{'url': 'https://api.github.com/repos/huggingface/datasets/pulls/856', 'html_url': 'https://github.com/huggingface/datasets/pull/856', 'diff_url': 'https://github.com/huggingface/datasets/pull/856.diff', 'patch_url': 'https://github.com/huggingface/datasets/pull/856.patch', 'merged_at': datetime.datetime(2020, 11, 17, 15, 22, 17)}
Adds book corpus based on Shawn Presser's [work](https://github.com/soskek/bookcorpus/issues/27)  @richarddwang, the author of the original BookCorpus dataset, suggested it should be named [OpenBookCorpus](https://github.com/huggingface/datasets/issues/486). I named it BookCorpusOpen to be easily located alphabetically. But, of course, we can rename it if needed. 

It contains 17868 dataset items; each item contains two fields: title and text. The title is the name of the book (just the file name) while the text contains unprocessed book text. Note that bookcorpus is pre-segmented into a sentence while this bookcorpus is not. This is intentional (see https://github.com/huggingface/datasets/issues/486) as some users might want to further process the text themselves. 

@lhoestq and others please review this PR thoroughly. cc @shawwn 
{'url': 'https://api.github.com/repos/huggingface/datasets/issues/856/reactions', 'total_count': 5, '+1': 2, '-1': 0, 'laugh': 0, 'hooray': 0, 'confused': 0, 'heart': 0, 'rocket': 3, 'eyes': 0}